Camera Image Anomaly Detection Method and System

Abstract

This application discloses a method and system for anomaly detection in camera images. By introducing a spatiotemporal reconstruction path and a spatiotemporal prediction path, image frame data is processed in parallel to obtain reconstruction error and prediction error, respectively. These two errors characterize the spatiotemporal fidelity and causality of the current image in the time series, providing the system with more comprehensive anomaly criteria. Furthermore, the error vector composed of these two errors is input into a trained small decision model for comprehensive judgment, enabling the system to more accurately distinguish between normal changes in image content and genuine imaging system anomalies, effectively avoiding frequent false alarms in high-dynamic scenes. In this way, the accuracy and robustness of anomaly detection are significantly improved, thus meeting the stringent requirements of low false alarm rate and high detection rate in practical applications, and greatly enhancing the practicality and reliability of camera monitoring systems.

Description

Technical Field

[0001] This application relates to the field of intelligent detection, and more specifically, to a method and system for detecting anomalies in camera images. Background Technology

[0002] With the rapid development of intelligent monitoring, smart cities, industrial automation, and autonomous driving, cameras have become core sensors for acquiring environmental information and ensuring the normal operation of systems. In these applications, cameras need to operate stably for extended periods and provide high-quality image data. However, in practical applications, the image data captured by cameras often faces various anomalies, such as image quality degradation or interruption caused by equipment failure, environmental interference, or human sabotage. Therefore, timely and accurate detection of these anomalies is crucial for ensuring system reliability, reducing human intervention, and improving the effectiveness of data analysis.

[0003] Existing technologies have proposed various solutions for anomaly detection in camera images. For example, some methods based on simple inter-frame difference or motion detection determine whether changes exist by calculating pixel differences between consecutive frames. While these methods have low computational cost, they struggle to adapt to complex dynamic backgrounds. Any target movement, lighting fluctuations, or even swaying leaves can be incorrectly labeled as anomalies, leading to a high false alarm rate. More advanced methods, such as reconstruction models based on autoencoders or generative adversarial networks, learn the image feature distribution in a normal pattern and then use the reconstruction error between the input image and the reconstructed image as the anomaly criterion. Theoretically, inputs deviating from the normal pattern will lead to high reconstruction errors and thus be identified as anomalies. However, in real-world high-dynamic scenes, this single error metric also has inherent limitations. When changes occur in legitimate scene content, such as a sudden increase in pedestrian traffic, drastic changes in ambient brightness, or the entry of new legitimate objects into the field of view, the complexity and information content of the image increase, leading to a significant increase in reconstruction error. This high reconstruction error may be very close in numerical value to, or even higher than, the reconstruction error caused by genuine image quality anomalies such as lens occlusion, image freezing, or signal loss. In other words, existing models often struggle to deeply understand the intrinsic nature of "change" and cannot effectively determine whether high errors stem from "legitimate evolution of scene content" or physical malfunctions of the "imaging system." This confusion means that anomaly detection in dynamic contexts still faces bottlenecks in accuracy and robustness, making it difficult to meet the practical application requirements for low false alarm rates and high detection rates.

[0004] Therefore, an optimized method for detecting anomalies in camera images is needed. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a camera image anomaly detection method and system. By introducing a spatiotemporal reconstruction path and a spatiotemporal prediction path, image frame data is processed in parallel to obtain reconstruction error and prediction error, respectively. These two errors characterize the spatiotemporal fidelity and time-series causality of the current image, providing the system with more comprehensive anomaly criteria. Furthermore, the error vector composed of these two errors is input into a trained small decision model for comprehensive judgment, enabling the system to more precisely distinguish between normal changes in image content and genuine imaging system anomalies, effectively avoiding frequent false alarms in high-dynamic scenes. In this way, the accuracy and robustness of anomaly detection are significantly improved, thus meeting the stringent requirements of low false alarm rate and high detection rate in practical applications, and greatly enhancing the practicality and reliability of camera monitoring systems.

[0006] According to one aspect of this application, a method for detecting anomalies in camera images is provided, comprising:

[0007] Obtain the timing window of image frames captured by the camera;

[0008] The image frame temporal window is input into the parallel spatiotemporal feature processing engine to obtain the image frame reconstruction temporal window and the predicted image frame. The parallel spatiotemporal feature processing engine includes a spatiotemporal reconstruction path and a spatiotemporal prediction path.

[0009] The reconstruction error is calculated based on the image frame temporal window and the image frame reconstruction temporal window.

[0010] The prediction error is calculated based on the current image frame and the predicted image frame in the image frame time window.

[0011] An error vector consisting of reconstruction error and prediction error is input into a trained small decision model to obtain diagnostic results.

[0012] According to another aspect of this application, a camera image anomaly detection system is provided, comprising:

[0013] The image frame timing window acquisition module is used to acquire the timing window of image frames captured by the camera;

[0014] The parallel spatiotemporal feature processing module is used to input the image frame temporal window into the parallel spatiotemporal feature processing engine to obtain the image frame reconstruction temporal window and the predicted image frame. The parallel spatiotemporal feature processing engine includes a spatiotemporal reconstruction path and a spatiotemporal prediction path.

[0015] The reconstruction error calculation module is used to calculate the reconstruction error based on the image frame time window and the image frame reconstruction time window.

[0016] The prediction error calculation module is used to calculate the prediction error based on the current image frame and the predicted image frame in the image frame time window.

[0017] The diagnostic module is used to input an error vector consisting of reconstruction error and prediction error into a trained small decision model to obtain diagnostic results.

[0018] Compared with existing technologies, this application provides a camera image anomaly detection method and system that introduces a spatiotemporal reconstruction path and a spatiotemporal prediction path to process image frame data in parallel, thereby obtaining reconstruction error and prediction error respectively. These two errors characterize the spatiotemporal fidelity and time-series causality of the current image, respectively, providing the system with more comprehensive anomaly criteria. Furthermore, the error vector composed of these two errors is input into a trained small decision model for comprehensive judgment, enabling the system to more precisely distinguish between normal changes in image content and genuine imaging system anomalies, effectively avoiding frequent false alarms in high-dynamic scenes. In this way, the accuracy and robustness of anomaly detection are significantly improved, thus meeting the stringent requirements of low false alarm rate and high detection rate in practical applications, and greatly enhancing the practicality and reliability of camera monitoring systems. Attached Figure Description

[0019] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 This is a flowchart of a camera image anomaly detection method according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of the data flow in the camera image anomaly detection method according to an embodiment of this application;

[0022] Figure 3 This is a block diagram of a camera image anomaly detection system according to an embodiment of this application. Detailed Implementation

[0023] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0024] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0025] While this application makes various references to certain modules of the systems according to embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0026] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0027] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0028] The technical solution of this application proposes a method for detecting anomalies in camera images. Figure 1 This is a flowchart of a camera image anomaly detection method according to an embodiment of this application. Figure 2 This is a system architecture diagram of a camera image anomaly detection method according to an embodiment of this application. Figure 1 and Figure 2 As shown, the camera image anomaly detection method according to an embodiment of this application includes the following steps: S1, acquiring a temporal window of image frames captured by the camera; S2, inputting the image frame temporal window into a parallel spatiotemporal feature processing engine to obtain an image frame reconstruction temporal window and a predicted image frame, wherein the parallel spatiotemporal feature processing engine includes a spatiotemporal reconstruction path and a spatiotemporal prediction path; S3, calculating a reconstruction error based on the image frame temporal window and the image frame reconstruction temporal window; S4, calculating a prediction error based on the current image frame and the predicted image frame in the image frame temporal window; S5, inputting an error vector composed of the reconstruction error and the prediction error into a trained small decision model to obtain a diagnostic result.

[0029] Specifically, in S1, a temporal window of image frames acquired by the camera is obtained. The raw image data acquired by the camera is instantaneous, but the judgment of image anomalies often requires contextual information based on the image change trend or image sequence over a period of time. A single image frame cannot effectively capture temporal anomalies such as image freezing, stuttering, flickering, slow blurring, or sudden image loss, nor can it provide sufficient information to determine the degree and persistence of spatial anomalies such as partial occlusion or lens deviation. Therefore, in the technical solution of this application, a sequence containing multiple consecutive image frames is constructed to provide the necessary spatiotemporal contextual information for the subsequent parallel spatiotemporal feature processing engine, thereby enabling a comprehensive analysis of the spatiotemporal characteristics of the image and laying a data foundation for accurately identifying various anomalies.

[0030] In practice, the following steps can be taken to obtain the time-series window of image frames captured by the camera: First, acquire real-time image frames. During this process, the system continuously receives raw image data output from the camera sensor by establishing a connection with the camera. This typically involves decoding the video stream, parsing the continuous video stream into a series of independent image frames. For example, an IP network camera might send a video stream via network protocols (such as RTSP or HTTP). The system receives this data in real time and uses a corresponding decoder (such as the FFmpeg library) to restore the video encoding format into a sequence of image frames, each representing a visual snapshot at a certain moment. Next, the real-time image frames are scaled and normalized to obtain processed frames. Scaled-up refers to adjusting the original image frames to a preset uniform resolution. For example, if the original image resolution output by the camera is 1920x1080 pixels, but the subsequent deep learning model has a fixed input size of 256x256 pixels for computational efficiency or model architecture requirements, then the original image needs to be downsampled or scaled. This can be achieved through various image interpolation algorithms, such as bilinear interpolation or bicubic interpolation; standardization refers to normalizing the pixel values ​​of an image. Image pixel values ​​are typically in the integer range of 0-255, while deep learning models usually have certain requirements for the numerical range and distribution of the input data, for example, normalizing the pixel values ​​to the interval [0, 1] or [-1, 1]. This can be achieved by dividing the pixel value by 255 (for 8-bit images) or by performing mean-variance normalization (i.e., subtracting the pixel mean and then dividing by the pixel standard deviation). After scaling and standardization, the original real-time image frame is transformed into a processed frame suitable for further processing, with a uniform size and numerical range; then, the processed frames are filled into a first-in-first-out (FIFO) frame buffer to obtain the image frame timing window. The frame buffer is a memory area used to temporarily store a certain number of image frames. The FIFO queue is a data structure that ensures that the data that enters the buffer first is retrieved first. In this process, each time a processed frame is generated, it is added to the end of the queue. If the number of image frames stored in the queue reaches the preset timing window size (for example, the preset window contains 16 images), then the oldest (first to enter) image frame in the queue will be removed (dequeued) to make room for newly entering image frames. Through this mechanism, the frame buffer always maintains a fixed-length (i.e., timing window size) sequence of the latest image frames, i.e., the image frame timing window. This window continuously slides as new image frames arrive, reflecting the visual information from the camera over a recent period in real time.

[0031] Specifically, in S2, the image frame temporal window is input into a parallel spatiotemporal feature processing engine to obtain an image frame reconstruction temporal window and a predicted image frame. The parallel spatiotemporal feature processing engine includes a spatiotemporal reconstruction path and a spatiotemporal prediction path. It should be understood that single-dimensional anomaly detection (focusing only on image reconstruction or only on image prediction) often has limitations. For example, a frozen image may be spatially normal, but its temporal dynamics may be abnormal, making it difficult to detect through reconstruction errors alone. Furthermore, static spatial anomalies such as lens blur or occlusion may not significantly affect the prediction of the next frame in some cases, potentially leading to insufficient prediction errors for effective judgment. Therefore, in the technical solution of this application, by constructing the spatiotemporal reconstruction path and the spatiotemporal prediction path in parallel, the system can simultaneously analyze from both the structural reconstructability and behavioral predictability dimensions of the image. This ensures that both spatial structural anomalies (such as image blur, occlusion, color distortion, etc.) and temporal dynamic anomalies (such as image freezing, stuttering, severe shaking, flickering, etc.) can be effectively modeled and detected, thus providing a comprehensive and accurate data foundation for subsequent error calculation and integrated diagnosis.

[0032] In practice, this parallel processing engine includes the following two main paths:

[0033] For the spatiotemporal reconstruction path, its feature encoding process includes:

[0034] First, the temporal window of the image frame is input into a spatiotemporal autoencoder to obtain the spatiotemporal feature vector of the image frame. It should be understood that traditional single-frame processing methods cannot effectively capture the temporal dynamic information in video data, while the spatiotemporal autoencoder, by jointly modeling the spatial and temporal dimensions, can learn more discriminative motion patterns and appearance change features, which is crucial for accurately detecting image anomalies. Here, the spatiotemporal autoencoder includes an encoder composed of 3D convolutional layers and a decoder composed of 3D deconvolutional layers. Unlike traditional 2D convolution, which operates only in the image plane, 3D convolution can extract features simultaneously in the spatial dimension (height, width) and the temporal dimension (frame sequence). Through layer-by-layer 3D convolution, pooling, and other operations, the encoder can learn and condense the inherent, most representative spatiotemporal feature vector of the image frame from the input temporal window.

[0035] In this process, the encoder receives a temporal window of image frames as input. This temporal window is a high-dimensional tensor, typically represented by (time step T, image height HJ, image width W, and number of image channels C). Then, 3D convolutional layers can simultaneously slide and compute convolutions along the temporal, spatial height, and spatial width dimensions. That is, each convolution operation considers not only spatial information within a local region of the image but also captures temporal correlations and motion patterns across consecutive frames. For example, a 3x3x3 convolutional kernel can simultaneously observe changes in a 3x3 pixel region across three frames. As data flows through 3D convolutional layers of varying depths in the encoder, the network gradually extracts higher-level, more abstract spatiotemporal features. Lower-level 3D convolutional layers may capture edges, textures, and simple motion trajectories in an image sequence; while deeper 3D convolutional layers can identify more complex spatiotemporal patterns, such as object shapes, motion directions, and changes in scene activity. After each 3D convolutional layer, a non-linear activation function (such as ReLU, Rectified LinearUnit) is typically applied to introduce the model's non-linearity, enabling it to learn and represent complex, non-linear feature relationships. Finally, to effectively compress the high-dimensional temporal window of image frames into a low-dimensional spatiotemporal feature vector of image frames, the encoder employs dimensionality reduction strategies at each layer, such as using 3D pooling layers. Through these operations, the size of the feature map (including time, height, and width) is reduced layer by layer, while the number of feature channels (determined by the number of convolutional kernels) typically increases gradually, allowing each pixel to contain richer semantic information.

[0036] Furthermore, the spatiotemporal feature vectors of the image frames are decoded and regressed to obtain the temporal window for image frame reconstruction. This decoder consists of a series of 3D deconvolutional layers. The decoder's task is to perform the opposite operation to the encoder, namely, to progressively upsample and recover the image frame reconstruction temporal window from the spatiotemporal feature vectors of the image frames, reproducing the same size and number of frames as the original input image frame temporal window. Here, if the input temporal window is normal and the autoencoder is sufficiently trained, the reconstructed temporal window should be highly similar to the original input at the pixel level; any significant differences will be considered potential signals of anomalies.

[0037] For spatiotemporal prediction paths, the feature encoding process includes:

[0038] First, all image frames between the current image frame and the current image frame are extracted from the image frame time sequence window to obtain the image frame time sequence history window. In other words, the historical portion is filtered out from the complete image frame time sequence window, i.e., the current (last) image frame is removed, resulting in the image frame time sequence history window. This history window contains all consecutive image frames up to the current moment, serving as the basis for predicting future frames.

[0039] Furthermore, the temporal history window of the image frames is input into the spatiotemporal prediction network to obtain the predicted image frames. It should be understood that video sequences in normal scenes typically exhibit temporal continuity and motion regularity, while anomalous events often disrupt this regularity. Constructing an accurate spatiotemporal prediction model can effectively capture these anomalous patterns. Compared to traditional single-frame analysis methods, this temporal prediction-based method can better detect motion-related anomalies and progressive anomalies.

[0040] In the technical solution of this application, the specific process of inputting the temporal history window of the image frame into the spatiotemporal prediction network includes:

[0041] First, each image frame in the temporal history window is input into a convolutional encoder to obtain a sequence of semantic feature vectors for each image frame. It should be understood that the original image frames contain a large amount of redundant information and high-dimensional data. Direct temporal prediction from the raw pixel level faces problems such as high computational complexity and sensitivity to noise. Furthermore, minute changes at the pixel level may not be significant for anomaly detection, while overall semantic changes in the image content (e.g., whether new objects have been added to the scene, whether object positions have moved, whether lighting conditions have changed abruptly, etc.) more directly reflect potential anomalies. In the technical solution of this application, feature extraction is performed on each image frame using a convolutional encoder, which not only effectively extracts key visual information from each image frame but also significantly reduces data dimensionality.

[0042] A convolutional encoder is a deep learning neural network architecture typically composed of multiple convolutional layers, activation functions, pooling layers, and possibly batch normalization layers. The convolutional layer is its core component. Each convolutional layer extracts local features by sliding a learned convolutional kernel across a small region of the image and calculating a weighted sum of pixels within that region. Different levels of convolutional kernels learn features of varying complexity, from basic edges and textures to more advanced shapes and patterns. Activation functions are applied after the convolution operation, introducing non-linearity and enabling the network to learn more complex mappings. Pooling layers primarily reduce the spatial dimensions (length and width) of the feature map while preserving key features and increasing robustness to small positional changes (i.e., translation invariance). For example, a max-pooling layer selects the maximum value within a region as its output.

[0043] Next, the sequence of semantic feature vectors of image frames is input into the image semantic temporal propagation recursive encoder to obtain the image frame semantic propagation latent encoding vector. It should be understood that the complex environment faced by modern surveillance systems means that anomaly detection needs to focus not only on the visual integrity of a single frame image, but also on grasping the dynamic evolution patterns inherent in the video sequence. Traditional methods often employ a uniform decay memory mechanism when processing temporal information, failing to fully consider the differences in the importance of image features at different times. This is inadequate when facing sudden anomalies (such as sudden screen obstruction) or gradual anomalies (such as gradual camera shift). Therefore, in the technical solution of this application, by introducing a message propagation mechanism based on temporal decay, the system can dynamically adjust the influence weight of historical frames on the current prediction, allowing frames with significant semantic changes (such as key frames showing abnormal objects) to maintain a more lasting influence during temporal propagation, while relatively stable ordinary frames gradually fade according to natural decay patterns. This temporal coding method significantly improves the system's ability to capture various anomaly patterns.

[0044] Specifically, firstly, the maximum eigenvalue of each image frame's semantic feature vector in the sequence of image frame semantic feature vectors is extracted as a temporal saliency identifier for the image frame's semantic features. It should be understood that in actual monitoring environments, abnormal events are often accompanied by significant abrupt changes in image features. Traditional temporal processing methods apply equal weight to historical frames, making it difficult to distinguish between slowly changing normal states and abnormal signs requiring focused attention. Therefore, in the technical solution of this application, the maximum eigenvalue of each image frame's semantic feature vector is extracted from the sequence of image frame semantic feature vectors to capture the most active dimension of the image frame's semantic feature vectors and quantify the semantic saliency of each frame. This saliency identifier based on the maximum eigenvalue provides a content-aware foundation for subsequent temporal modeling.

[0045] Specifically, in a specific example of this application, the temporal salient identifier of the semantic features of an image frame is extracted using the following formula:

[0046]

[0047] in, For timestamps, For the first semantic feature vectors of each image frame This indicates taking the maximum eigenvalue across all dimensions of the semantic feature vector of the image frame. It is a temporal significant identifier for the semantic features of an image frame.

[0048] Next, based on the timestamps of the semantic feature vectors of each image frame in the sequence of image frame semantic feature vectors, the time span of the semantic feature vectors of each image frame is calculated. It should be understood that in real surveillance environments, events captured by cameras do not occur uniformly. Traditional time-series models assume that the contribution of all historical frames to the current prediction is only monotonically decreasing with time intervals, ignoring the semantic significance differences of the events themselves. Therefore, in the technical solution of this application, by calculating the time span of the semantic feature vectors of each image frame (i.e., the physical time difference between the current prediction time and the actual capture time of that historical frame), the accurate timestamp information attached to each image frame is used to calculate the actual time difference between adjacent frames. These time differences reflect the true rhythm of scene changes. This time-series modeling based on real time spans significantly improves the robustness of the system in complex scenarios.

[0049] Specifically, in a specific example of this application, the time span of the semantic feature vector of each image frame is calculated using the following formula:

[0050]

[0051] in, The timestamp of the semantic feature vector of the image frame. This represents the time span of the semantic feature vectors for each image frame.

[0052] Furthermore, based on the time span of each image frame semantic feature vector in the sequence of image frame semantic feature vectors and the temporal saliency identifier of the image frame semantic features, a temporal decay propagation is performed on the sequence of image frame semantic feature vectors to obtain the implicit encoding vector for image frame semantic transmission. It should be understood that traditional temporal processing methods often employ fixed-pattern memory mechanisms, which cannot adapt to the complex and ever-changing temporal dynamics and differences in content importance in monitoring scenarios. This is particularly evident when dealing with sudden abnormal events or long-term monitoring tasks. Therefore, in the technical solution of this application, by organically combining the two key dimensions of physical time span and semantic saliency, intelligent filtering and weighted aggregation of historical information are achieved. This intelligent temporal propagation mechanism significantly improves the discrimination capability of the anomaly detection system in complex scenarios.

[0053] Specifically, in the technical solution of this application, the specific process of performing temporal attenuation weighted transfer on the sequence of semantic feature vectors of image frames includes:

[0054] First, the temporal decay weights of the semantic feature vectors of each image frame are calculated, and the process is expressed by the formula:

[0055] in, For the first Temporal decay weights of semantic feature vectors of each image frame It is a learnable attenuation coefficient, and The rate of time decay is controlled and optimized through backpropagation. It is a Gaussian decay kernel. Let i be the set of neighbors of the target node i. Enhance the contributions of important neighbors;

[0056] Here, by using the largest eigenvalue As a modulation factor for attenuation amplitude, salient features retain higher weights even over time; furthermore, through learnable parameters... Dynamically adjust the shape of the decay curve to adapt to the time-sensitive requirements of different scenarios;

[0057] Next, based on the temporal attenuation weights of the semantic feature vectors of each image frame, feature attenuation modulation is performed on the sequence of semantic feature vectors of the image frames to obtain the sequence of semantic attenuation feature encoding vectors of the image frames. The process is expressed by the following formula:

[0058]

[0059]

[0060] in, for The corresponding image frame semantic decay feature encoding vector, These are the first, second, and third elements in the sequence of semantic decay feature encoding vectors for image frames. The semantic decay feature encoding vector of each image frame A sequence of encoding vectors for semantic decay features of an image frame;

[0061] Ultimately, through based on The model's temporal encoder performs temporal encoding on the sequence of semantic attenuation feature encoding vectors of image frames to obtain the semantic transitive latent encoding vectors of image frames. The process is expressed by the following formula:

[0062]

[0063] in, Represents timing coding. The implicit encoding vector is passed to the semantics of the image frame.

[0064] Furthermore, the semantic information of the image frame is passed to the implicit encoded vector and input into the image feature decoder to obtain the predicted image frame. The image feature decoder is typically composed of a series of deconvolutional layers, upsampling layers (such as bilinear interpolation or nearest neighbor interpolation), convolutional layers, and activation functions stacked together.

[0065] In specific implementation, image feature decoding of the semantic transitive latent encoding vector of the image frame can be performed through the following steps: First, the image feature decoder receives the semantic transitive latent encoding vector of the image frame from the image semantic temporal transitive recursive encoder as input, and converts it into a small feature map with multiple channels (e.g., a 4x4x512 feature map) through a fully connected layer or an initial deconvolution layer; then, the decoder gradually expands the spatial size of the feature map through multiple deconvolution operations, with each deconvolution layer using a 3×3 convolution kernel and a ReLU activation function. During the deconvolution process, the system upsamples by a factor of 2, and simultaneously fuses low-level features with high-level features through a skip connection mechanism; finally, a 1×1 convolution is used in the output layer to adjust the number of channels to 3 (corresponding to RGB three channels), and the pixel values ​​are normalized to the [0,1] range through a Sigmoid activation function, thereby generating a predicted image frame with the same size as the input image.

[0066] Specifically, S3 calculates the reconstruction error based on the image frame temporal window and the image frame reconstruction temporal window. It should be understood that during the training phase, spatiotemporal autoencoders typically learn on a large number of normal image frame temporal windows. Their goal is to learn how to effectively compress the spatiotemporal features of normal images and accurately reconstruct them. Therefore, for a normal image frame temporal window, the autoencoder should be able to generate an image frame reconstruction temporal window that is highly similar to the original window, at which point the reconstruction error will be small. However, when the input image frame temporal window is abnormal, such as blurred images, partial occlusion, lens shift, watermarks, image damage, or pixel errors, these abnormal features do not match the patterns learned by the model on normal data. When faced with these unseen or non-compliant abnormal inputs, the autoencoder often cannot effectively compress and accurately reconstruct them, resulting in a significant difference between the reconstructed image frame reconstruction temporal window and the original abnormal input. This significant difference, i.e., a large reconstruction error, can serve as an indication of anomalies in the image's spatial structure. Therefore, in the technical solution of this application, the reconstruction error is calculated and analyzed to quantify the degree of difference between the original image and the reconstructed image, providing an important basis for subsequent anomaly diagnosis. Specifically, the calculation of the reconstruction error directly reflects the learning ability of the spatiotemporal autoencoder to normal image patterns. When an anomaly occurs in the input image, the reconstruction error will increase significantly, thus becoming an effective indicator for anomaly detection. In this way, the system can effectively detect structural defects or interferences that deviate from the normal image appearance pattern.

[0067] Among them, reconstruction error refers to the mean square error between the current image frame and its corresponding currently reconstructed image frame, which is used to measure whether the image's own structure or content deviates from the normal pattern.

[0068] In specific implementation, the reconstruction error can be calculated through the following steps: First, extract the current image frame and the current reconstructed image frame from the image frame timing window and the image frame reconstruction timing window. During this process, the system first extracts the current image frame from the image frame timing window, and simultaneously extracts the corresponding current reconstructed image frame from the image frame reconstruction timing window. Then, the system converts these two image frames into the same data format, ensuring that they have the same size and pixel value range. Next, the mean square error (MSE) between the current image frame and the current reconstructed image frame is calculated as the reconstruction error. That is, the mean square error (MSE) between the two image frames is calculated using a pixel-by-pixel comparison method, and its calculation process is expressed by the formula:

[0069] Wherein, the image resolution is Pixels These are the original pixel values. To reconstruct pixel values, This is the normalized scalar value of the reconstruction error.

[0070] Taking the scheme of this application as an example, assume that the image resolution H×W=4×4 pixels (for simplified demonstration, the actual resolution is 256×256).

[0071] Current original frame (The lower right area is blurry due to lens damage):

[0072] [ [100, 100, 100, 100], / / Pixel value range 0~255 (after normalization)

[0073] [100, 100, 100, 100],

[0074] [100, 100,50,50], / / Stained area (pixel value decreased)

[0075] [100, 100, 50, 50]

[0076] Current reconstructed frame (Normal reconstruction of model output):

[0077] [ [100, 100, 100, 100],

[0078] [100, 100, 100, 100],

[0079] [100, 100, 100, 100], / / The model has not learned the features of soiling, and outputs a clear image.

[0080] [100, 100, 100, 100]

[0081] The reconstruction error is then calculated using the following steps:

[0082] First, calculate the squared error for each pixel. The calculation process is as follows:

[0083] coordinate :

[0084] coordinate :

[0085] coordinate : / / Stained area

[0086] coordinate :

[0087] coordinate :

[0088] coordinate :

[0089] The error for the remaining 12 pixels is 0.

[0090] Then, the squared errors of all pixels are summed, and the calculation process is as follows:

[0091] Finally, the mean squared error is calculated using normalization. The calculation process is as follows:

[0092] The reconstruction error is then calculated to be 625.

[0093] Specifically, in step S4, the prediction error is calculated based on the current image frame and the predicted image frame within the image frame temporal window. It should be understood that image sequences captured by a camera typically follow certain temporal patterns and motion patterns (e.g., a stable background, slow object movement, gradual lighting changes, etc.). The spatiotemporal prediction network, trained on a large number of normal image sequences, can learn and model these complex temporal dependencies, thereby accurately predicting the current frame based on historical frames. When there is a significant difference between the actually received current image frame and the predicted image frame predicted by the model based on historical frames, this indicates an anomaly in the temporal evolution of the image sequence that deviates from the normal pattern. Such anomalies include, but are not limited to: image freezing (sudden cessation of image change), image stuttering or frame skipping (discontinuous images), severe shaking, signal loss leading to black / distorted screens, and sudden scene changes. By calculating the prediction error, the system can effectively detect these unexpected, sudden, or long-term visual events that violate temporal patterns, providing crucial information for comprehensive anomaly identification.

[0094] Prediction error is a quantified measure of the difference between the currently observed image frame and the predicted image frame generated by the model based on historical data. High prediction error typically indicates unexpected temporal changes in the image sequence.

[0095] In a specific implementation, the prediction error can be calculated through the following steps: First, extract the current image frame from the image frame time window and extract the prediction image frame; then, calculate the mean square error between the current image frame and the prediction image frame as the prediction error.

[0096] Specifically, in step S5, an error vector consisting of reconstruction error and prediction error is input into a trained small decision model to obtain a diagnostic result. It should be understood that, specifically, reconstruction error (spatial dimension anomalies, such as blurry or occluded images) and prediction error (temporal dimension anomalies, such as frozen or jittery images) can only detect anomalies in a single dimension, while in real-world scenarios, anomalies are often spatiotemporally coupled (e.g., lens smudges accompanied by video stuttering). By combining the two errors into a single vector and inputting it into the decision model, the system can fuse spatial and temporal information to generate a unified anomaly diagnostic result, avoiding misjudgments or missed detections caused by a single error source, thereby improving detection robustness and decision accuracy in complex environments.

[0097] The trained small-scale decision model is a logistic regression classifier. In this process, the logistic regression classifier can weigh the importance of different errors and make more robust and accurate diagnoses under complex combinations of errors, ultimately outputting a clear diagnostic result (e.g., normal or abnormal).

[0098] In practice, the error vector, composed of reconstruction error and prediction error, can be input into the trained small decision model through the following steps: First, the reconstruction error and prediction error are concatenated to obtain the error vector, which can be expressed by the formula:

[0099]

[0100] in, To reconstruct the error scalar value, For the prediction error scalar value, This is the error vector;

[0101] Subsequently, the error vector is input into a trained small decision model to obtain the diagnostic result. In the technical solution of this application, the logistic regression classifier is a binary classification model, the core of which is the sigmoid function, which outputs the anomaly probability. The calculation process is expressed by the formula:

[0102]

[0103] in, and Here is the weight matrix, and b is the bias term. This represents the probability of an anomaly.

[0104] It's worth noting that the model was trained on a historical dataset containing error vectors and labels (0 = normal, 1 = abnormal) of both normal and abnormal samples. The weights were optimized using maximum likelihood estimation. and And bias b, make the predicted probability Approaching real-world labels;

[0105] Finally, the output probability based on the logistic regression classifier This process generates the final diagnostic result. During this process, a decision threshold (usually 0.5) can be set; if... The diagnosis was "abnormal"; if The diagnosis was "normal".

[0106] In summary, the camera image anomaly detection method according to the embodiments of this application is explained. It introduces a spatiotemporal reconstruction path and a spatiotemporal prediction path to process image frame data in parallel, thereby obtaining reconstruction error and prediction error, respectively. These two errors characterize the spatiotemporal fidelity and causality of the current image in the time series, providing the system with more comprehensive anomaly criteria. Furthermore, the error vector composed of these two errors is input into a trained small decision model for comprehensive judgment, enabling the system to more precisely distinguish between normal changes in image content and genuine imaging system anomalies, effectively avoiding frequent false alarms in high-dynamic scenes. In this way, the accuracy and robustness of anomaly detection are significantly improved, thus meeting the stringent requirements of low false alarm rate and high detection rate in practical applications, and greatly improving the practicality and reliability of the camera monitoring system.

[0107] Furthermore, a camera image anomaly detection system is also provided.

[0108] Figure 3 This is a block diagram of a camera image anomaly detection system according to an embodiment of this application. Figure 3As shown, the camera image anomaly detection system 300 according to an embodiment of this application includes: an image frame temporal window acquisition module 310, used to acquire an image frame temporal window captured by a camera; a parallel spatiotemporal feature processing module 320, used to input the image frame temporal window into a parallel spatiotemporal feature processing engine to obtain an image frame reconstruction temporal window and a predicted image frame, wherein the parallel spatiotemporal feature processing engine includes a spatiotemporal reconstruction path and a spatiotemporal prediction path; a reconstruction error calculation module 330, used to calculate a reconstruction error based on the image frame temporal window and the image frame reconstruction temporal window; a prediction error calculation module 340, used to calculate a prediction error based on the current image frame and the predicted image frame in the image frame temporal window; and a diagnosis module 350, used to input an error vector composed of the reconstruction error and the prediction error into a trained small decision model to obtain a diagnosis result.

[0109] As described above, the camera image anomaly detection system 300 according to the embodiments of this application can be implemented in various wireless terminals, such as servers with camera image anomaly detection algorithms. In one possible implementation, the camera image anomaly detection system 300 according to the embodiments of this application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the camera image anomaly detection system 300 can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the camera image anomaly detection system 300 can also be one of many hardware modules of the wireless terminal.

[0110] Alternatively, in another example, the camera image anomaly detection system 300 and the wireless terminal can also be separate devices, and the camera image anomaly detection system 300 can be connected to the wireless terminal via wired and / or wireless networks, and transmit interactive information in accordance with an agreed data format.

[0111] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for detecting anomalies in camera images, characterized in that, include: Obtain the timing window of image frames captured by the camera; The image frame temporal window is input into the parallel spatiotemporal feature processing engine to obtain the image frame reconstruction temporal window and the predicted image frame. The parallel spatiotemporal feature processing engine includes a spatiotemporal reconstruction path and a spatiotemporal prediction path. The reconstruction error is calculated based on the image frame temporal window and the image frame reconstruction temporal window. The prediction error is calculated based on the current image frame and the predicted image frame in the image frame time window. The error vector, consisting of reconstruction error and prediction error, is input into a trained small decision model to obtain diagnostic results. The image frame temporal window is input into the parallel spatiotemporal feature processing engine to obtain the image frame reconstruction temporal window and the predicted image frame, including: Extract all image frames between the current image frame from the image frame timing window to obtain the image frame timing history window; Each image frame in the temporal history window is input into a convolutional encoder to obtain a sequence of semantic feature vectors for the image frames; The maximum eigenvalue of each semantic feature vector in the sequence of image frame semantic feature vectors is extracted as the temporal saliency identifier of the image frame semantic features; The time span of each semantic feature vector in the sequence of image frames is calculated based on the timestamp of each semantic feature vector in the sequence of image frames. Based on the time span of each semantic feature vector in the sequence of image frame semantic feature vectors and the temporal significant identifier of the semantic feature vectors of the image frame, the sequence of semantic feature vectors of the image frame is subjected to temporal decay propagation to obtain the semantic transfer implicit coding vector of the image frame. The semantic pass-through hidden encoded vector of the image frame is input into the image feature decoder to obtain the predicted image frame.

2. The camera image anomaly detection method according to claim 1, characterized in that, Acquire a time-series window of image frames captured by the camera, including: Acquire real-time image frames; Real-time image frames are scaled and normalized to obtain processed frames; The processed frames are filled into the first-in-first-out queue frame buffer to obtain the image frame timing window.

3. The camera image anomaly detection method according to claim 2, characterized in that, The image frame temporal window is input into the parallel spatiotemporal feature processing engine to obtain the image frame reconstruction temporal window and the predicted image frame, including: The temporal window of the image frame is input into the spatiotemporal autoencoder to obtain the spatiotemporal feature vector of the image frame; The spatiotemporal feature vectors of image frames are decoded and regressed to obtain the temporal window for image frame reconstruction.

4. The camera image anomaly detection method according to claim 3, characterized in that, The spatiotemporal autoencoder includes an encoder composed of 3D convolutional layers and a decoder composed of 3D deconvolutional layers.

5. The camera image anomaly detection method according to claim 1, characterized in that, Based on the image frame temporal window and the image frame reconstruction temporal window, the reconstruction error is calculated, including: Extract the current image frame and the current reconstructed image frame from the image frame timing window and the image frame reconstruction timing window; The mean square error between the current image frame and the current reconstructed image frame is calculated as the reconstruction error.

6. The camera image anomaly detection method according to claim 1, characterized in that, The trained small-scale decision model is a logistic regression classifier.

7. A camera image anomaly detection system, used to perform the method as described in any one of claims 1 to 6, characterized in that, include: The image frame timing window acquisition module is used to acquire the timing window of image frames captured by the camera; The parallel spatiotemporal feature processing module is used to input the image frame temporal window into the parallel spatiotemporal feature processing engine to obtain the image frame reconstruction temporal window and the predicted image frame. The parallel spatiotemporal feature processing engine includes a spatiotemporal reconstruction path and a spatiotemporal prediction path. The reconstruction error calculation module is used to calculate the reconstruction error based on the image frame time window and the image frame reconstruction time window. The prediction error calculation module is used to calculate the prediction error based on the current image frame and the predicted image frame in the image frame time window. The diagnostic module is used to input an error vector consisting of reconstruction error and prediction error into a trained small decision model to obtain diagnostic results.

Images