Video anomaly detection method, system and device based on edge device and medium

Abstract

The invention discloses a video anomaly detection method, system, equipment and medium based on edge equipment, which are in one-to-one correspondence schemes, in the scheme, an anomaly detection technology which is supported by a cloud and can adapt and learn on the edge equipment in real time is provided, and a knowledge graph is combined, so that the video anomaly detection efficiency is improved. End-to-end adaptive anomaly detection is realized through a graph neural network and a deep reinforcement learning (DRL) model; a processing mode based on the edge device is adopted, cloud dependence is avoided, consumption and response delay of computing resources are greatly reduced, the knowledge graph can be adaptively updated on the edge device due to the fact that additional cloud computing resources are not needed, and therefore the accuracy and robustness of video anomaly detection are continuously optimized.

Description

Technical Field

[0001] The present invention relates to the field of video anomaly detection technology, and in particular to a video anomaly detection method, system, device and medium based on edge devices. Background Art

[0002] With the increasing demand for applications such as intelligent surveillance, public safety, evidence analysis, and violence detection, video anomaly detection (VAD) technology has become crucial in modern security systems. Designed to automatically identify anomalous behaviors that deviate from typical patterns in video data, VAD is widely used in real-time monitoring and event detection. Traditional video anomaly detection methods rely on deep learning models, particularly architectures such as convolutional neural networks (CNNs), long short-term memory networks (LSTMs), and transformers, to identify anomalous behaviors by extracting complex spatiotemporal features.

[0003] However, as application scenarios continue to expand, existing VAD methods face numerous challenges in adapting to dynamic environments. In particular, as behavioral patterns evolve in the environment, video anomaly detection models often require frequent updates to address new anomalous patterns or changing data trends. Existing models often rely on large-scale pre-trained models and knowledge graphs, which are typically generated based on large language models (LLMs) such as GPT-4 (the fourth-generation generative pre-trained transformer model). However, this approach faces significant computational and memory overhead when real-time updates are required, especially when running on edge computing devices. The reliance on cloud computing also leads to inevitable latency and bandwidth bottlenecks.

[0004] Existing models, such as Mission GNN (hierarchical multimodal graph neural network model), propose an approach based on graph neural networks and knowledge graphs, which achieves efficient anomaly detection by converting task-specific knowledge graphs into graph structures. Despite significant progress in computational and memory efficiency, Mission GNN's application in dynamic environments remains limited. Specifically, Mission GNN requires regular updates to the knowledge graph via cloud computing to respond to new anomaly patterns or environmental changes, which is not feasible for resource-constrained edge computing devices.

[0005] How to achieve continuous adaptation of knowledge graphs on edge devices and respond to changes in anomaly patterns in real time to ensure detection performance has become an important research direction in video anomaly detection technology. Existing technologies have failed to effectively address this problem, especially in terms of the ability to perform continuous learning and knowledge graph updates on edge devices.

[0006] In view of this, the present invention is proposed. Summary of the Invention

[0007] The purpose of the present invention is to provide a video anomaly detection method, system, device and medium based on edge devices, which can realize real-time and adaptive video anomaly detection on edge computing devices, and can automatically adjust the knowledge graph in a dynamic environment to quickly adapt to new anomaly patterns.

[0008] The purpose of the present invention is achieved through the following technical solutions: A video anomaly detection method based on an edge device, comprising: Combined with the specified tasks, build the corresponding knowledge graph; In combination with the video surveillance data and the knowledge graph, a decision model is trained, the decision model comprising: a video frame encoding module, a graph neural network, a short-term time module, and an anomaly detection decision module; during the training process, each video frame of the video surveillance data is encoded into an embedding vector by the video frame encoding module and attached to the knowledge graph; each embedding vector is aggregated with information in the knowledge graph by the graph neural network to obtain an inference embedding vector corresponding to each video frame; the inference embedding vectors of several consecutive video frames are comprehensively processed by the short-term time module to obtain a temporal embedding feature vector; the temporal embedding feature vector is subjected to anomaly detection by the anomaly detection decision module, and a detection result of each video frame in the several consecutive video frames is output; the decision model is trained based on the detection results; The trained decision model is deployed in the edge device, and the input video surveillance data to be detected is subjected to real-time anomaly detection in combination with the knowledge graph; and the knowledge graph is adaptively updated in combination with the real-time anomaly detection results.

[0009] A video anomaly detection system based on edge devices, comprising: The knowledge graph construction unit is used to build the corresponding knowledge graph in combination with the specified task; A decision model training unit is used to train a decision model by combining video surveillance data, historical anomaly data, and the knowledge graph. The decision model includes: a video frame encoding module, a graph neural network, a short-term time module, and an anomaly detection decision module. During the training process, each video frame of the video surveillance data is encoded into an embedding vector by the video frame encoding module and attached to the knowledge graph. The graph neural network aggregates the information of each embedding vector and the knowledge graph to obtain an inference embedding vector corresponding to each video frame. The short-term time module performs comprehensive processing on the inference embedding vectors of several consecutive video frames to obtain a temporal embedding feature vector. The anomaly detection decision module performs anomaly detection on the temporal embedding feature vector and outputs the detection result of each video frame in the several consecutive video frames. The decision model is trained based on the detection results. The anomaly detection and knowledge graph update unit is used to deploy the trained decision model in the edge device, and perform real-time anomaly detection on the input video surveillance data to be detected in combination with the knowledge graph; and, based on the real-time anomaly detection results, adaptively update the knowledge graph.

[0010] A processing device comprising: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0011] A readable storage medium stores a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0012] It can be seen from the technical solution provided by the above-mentioned present invention that a cloud-supported anomaly detection solution that can adapt and learn in real time on edge devices is provided. It combines knowledge graphs and implements end-to-end adaptive anomaly detection through graph neural networks and deep reinforcement learning (DRL) models (that is, trained decision models); it adopts an edge device-based processing method to avoid cloud dependence, greatly reducing computing resource consumption and response delays. Since no additional cloud computing resources are required, the knowledge graph can be adaptively updated on the edge device, thereby continuously optimizing the accuracy and robustness of video anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 A flowchart of a method for detecting video anomalies based on edge devices provided by an embodiment of the present invention; Figure 2 A schematic diagram of the knowledge graph generation process provided by an embodiment of the present invention; Figure 3 A schematic diagram of adaptive learning of knowledge graphs provided by an embodiment of the present invention; Figure 4 A schematic diagram of a video anomaly detection system based on edge devices provided by an embodiment of the present invention; Figure 5 A schematic diagram of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0016] First, the following terms may be used in this article: The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles)" should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0017] The term "consisting of" excludes any technical features not explicitly listed. If used in a claim, this term renders the claim closed, excluding any technical features other than those explicitly listed, except for conventional impurities associated with them. If this term appears only in a clause of a claim, it limits only the elements explicitly listed in that clause; elements listed in other clauses are not excluded from the claim as a whole.

[0018] The following describes in detail the edge device-based video anomaly detection method, system, device, and medium provided by the present invention. Any information not described in detail in the embodiments of the present invention is prior art known to those skilled in the art. For any conditions not specified in the embodiments of the present invention, the conditions are conventional in the art or those recommended by the manufacturer. Instruments used in the embodiments of the present invention, for which the manufacturer is not specified, are all commercially available conventional products.

[0019] Example 1 like Figure 1 As shown in FIG, a video anomaly detection method based on edge devices mainly includes the following steps: Step 1: Combine the specified tasks and build the corresponding knowledge graph.

[0020] Preferably, a knowledge graph can be generated based on a large language model in combination with a specified task and a preset prompt format; the knowledge graph is a hierarchical directed acyclic graph, each node represents a specified concept expressed in a text, and is assigned to a specified level, each edge connects two nodes of adjacent layers, and points from the lower layer to the higher layer; the lower layer and the higher layer here are corresponding concepts, for example, adjacent layers are recorded as the i-th layer and the i+1-th layer, then the i-th layer is the lower layer, and the i+1-th layer is the higher layer.

[0021] Step 2: Combine the video surveillance data and the knowledge graph to train a decision model.

[0022] In this embodiment of the present invention, the decision model includes: a video frame encoding module, a graph neural network, a short-term time module, and an anomaly detection decision module. The training process is as follows: (1) Each video frame of the video surveillance data is encoded into an embedding vector through a video frame encoding module and attached to the knowledge graph.

[0023] Preferably, each video frame of the video surveillance data is encoded into an embedding vector through a video frame encoding module, and each video frame is attached to the knowledge graph as a sensor node as the top layer of the knowledge graph, and the embedding vector of each video frame is used as the representation of the corresponding sensor node; an embedding node is created and attached to the knowledge graph as the bottom layer of the knowledge graph, then the k-layer knowledge graph becomes k+2 layers after the sensor nodes and the embedding nodes are attached.

[0024] (2) Through the graph neural network, the information of each embedding vector and the knowledge graph is aggregated to obtain the inference embedding vector corresponding to each video frame.

[0025] Preferably, the graph neural network has a k+2 layer structure, which is responsible for propagating the embedding vector corresponding to each sensor node layer by layer until it reaches the embedding node, and obtains the inference embedding vector of the aggregated embedding vector and the information in the knowledge graph.

[0026] (3) The inference embedding vectors of several consecutive video frames are comprehensively processed through the short-term time module to obtain the temporal embedding feature vector.

[0027] Preferably, the video surveillance data is divided into multiple groups of sub-data, each group of sub-data contains several consecutive frames, and the inference embedding vectors of the several consecutive video frames contained in each group are feature extracted and fused through a short-term time module to obtain a temporal embedding feature vector.

[0028] (4) Anomaly detection is performed on the temporal embedded feature vector by an anomaly detection decision module, and a detection result of each video frame in the plurality of continuous video frames is output.

[0029] Preferably, the anomaly detection decision module includes a linear layer and a softmax function layer, inputs a time series embedded feature vector, and after being processed in sequence by the linear layer and the softmax function layer, outputs the detection results of each video frame in each group of sub-data, and the detection results include: the probability that the video frame belongs to the normal type, and the probability of belonging to each abnormal type; the softmax function is a normalized exponential function.

[0030] Finally, the decision model can be trained in combination with the detection results. The specific training process involved can be implemented with reference to conventional technologies and will not be described in detail in the present invention.

[0031] Step 3: Video anomaly detection based on edge devices and adaptive update of knowledge graph.

[0032] In an embodiment of the present invention, the trained decision model is deployed in an edge device, and real-time anomaly detection is performed on the input video surveillance data to be detected in combination with the knowledge graph; and, in combination with the real-time anomaly detection results, the knowledge graph is adaptively updated.

[0033] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail below with reference to specific embodiments.

[0034] 1. Overall overview of the plan.

[0035] In the field of video anomaly detection, traditional methods rely on frequent cloud-based updates and retraining, which presents significant challenges in resource-constrained edge computing environments that require low-latency responses. In particular, most existing anomaly detection systems lack the flexibility to adapt to dynamically changing anomaly patterns and are unable to effectively learn and update in real time without cloud support. To address these issues, embodiments of the present invention provide an edge-based video anomaly detection method that combines large language models (LLMs), graph neural networks (GNNs), and knowledge graphs (KGs) for real-time monitoring and adaptive detection of abnormal behavior in dynamic environments. By performing adaptive knowledge graph updates on edge devices, this method achieves real-time anomaly detection and learning capabilities without cloud support, making it particularly suitable for resource-constrained scenarios. In this system, a large language model (such as GPT-4) is used to generate task-specific knowledge graphs and guide the dynamic adaptation of anomaly detection, significantly improving the detection capabilities of new and complex anomaly patterns.

[0036] 2. Detailed introduction of the plan.

[0037] 1. Generate a knowledge graph (KG) for a specified task.

[0038] In this embodiment of the present invention, a large language model is used to generate a knowledge graph based on a defined task and a preset prompt format. This graph is then used for reasoning to complete the specified task. This invention defines the knowledge graph used for reasoning as a hierarchical directed acyclic graph (DAG), where each node represents a unique concept expressed through a short text and is assigned to a specific level. Edges in the KG only connect nodes at level i with nodes at level i+1, enforcing a strict hierarchical structure.

[0039] like Figure 2 The figure shows a schematic diagram of the knowledge graph generation process, which mainly includes: creating initial nodes to form the first layer of the knowledge graph. Generally, the first inference node generated based on the input data and the given task can be used as the initial node; thereafter, nodes of the i+1 layer are generated each time based on the nodes of the i-th layer. The following description uses the i-th layer as the current layer. When iterating based on the nodes generated by the current layer, each iteration performs node generation, edge generation, and error detection and correction steps. In the node generation step, a preset prompt format is used to guide the large language model to generate nodes of the next layer that can be derived from the nodes of the current layer. In the edge generation step, a preset prompt format is used to guide the large language model to establish connecting edges between the nodes of the current layer and the next layer. In the error detection and correction step, whether there are errors is detected. If so, the large language model is used to correct them; if not, the next iteration is entered. Specifically, error detection focuses on two types of errors: duplicate concepts (i.e., nodes representing concepts that have already appeared in the previous layer) and invalid edges (i.e., connections that violate the rule that edges only connect nodes of the i-th layer and the i+1-th layer). If no errors are detected, the next iteration is entered, meaning the next layer is generated. If errors are found, an error correction loop is entered, during which the large language model is prompted to correct the detected issues using predefined correction prompts. Given that the large language model may introduce new errors during the correction process, this loop is repeated until no errors are detected or the maximum number of iterations is reached. If all errors are not resolved after the maximum number of iterations is reached, the problematic nodes or edges are pruned.

[0040] Continue to iterate until the set number of layers is reached to obtain a knowledge graph in the form of a layered directed acyclic graph.

[0041] Those skilled in the art will understand that the training set contains a large number of training videos, and different training videos construct corresponding knowledge graphs. The subsequent knowledge graphs will be adaptively updated, and different abnormal videos will be detected to achieve iterative updates of detection.

[0042] 2. Train a lightweight GNN decision model.

[0043] (1) Video frame encoding module.

[0044] Video frame encoding module Each video frame of the video surveillance data are encoded into embedding vectors respectively.

[0045] In an embodiment of the present invention, in order to promote efficient reasoning for a given task and provide a solid foundation for subsequent tasks, each video frame is attached to the knowledge graph as a sensor node as the top layer of the knowledge graph, and the embedding vector of each video frame is used as the representation of the corresponding sensor node; an embedding node is created and attached to the knowledge graph as the bottom layer of the knowledge graph, then the k-layer knowledge graph becomes k+2 layers after the sensor nodes and embedding nodes are attached.

[0046] (2) Graph neural networks.

[0047] The graph neural network has a k+2 layer structure, which is responsible for propagating the embedding vector corresponding to each sensor node layer by layer until it reaches the embedding node, and obtains the inference embedding vector of the aggregated embedding vector and the information in the knowledge graph.

[0048] In this embodiment of the present invention, each layer of the graph neural network (GNN layer) includes five sublayers: a dense layer, a hierarchical message passing layer, a hierarchical aggregate layer, a batch normalization layer, and an activation layer.

[0049] (2.1) Dense layer ( ): This layer aims to optimize the embedding space so that it can more accurately represent the relationship between knowledge embeddings. Its calculation formula is as follows: ; in, is a dense layer, subscript 、 Represents the sequence number of the corresponding layer, is the symbol of the real number set, is the number of elements in the node set V in the knowledge graph (i.e. the total number of nodes), and Respectively represent Layer (previous layer) and the dimensions of the layer (the current layer), is a trainable parameter matrix, is a trainable bias term, Indicates the The node embedding of the layer, where the node embedding is an embedding matrix, which maps each node to a vector space and is passed as input to the current layer.

[0050] (2.2) Information transmission layer ): This layer calculates the message of each node, where Is connected Layer and The edge set of the layer node, E is the edge set of the knowledge graph. The calculation formula for information transmission is: ; in, Represents the information transmission layer, s and d represent the source node and target node respectively. The source node and target node here are the node descriptions of each layer. The source node is the sender of the message transmission, and the target node is the receiver. Indicates in In the layer, there is an edge from the source node s to the target node d (that is, there is a connection), represents node embedding, 、 Represents the embedding of source node s and target node d.

[0051] (2.3) Hierarchical aggregation layer ( ): This layer merges the messages of the source nodes into a unified vector while maintaining the embeddings of the nodes that have not received any messages. Its calculation formula is: ; in, Represents a hierarchical aggregation layer, represents node embedding, Represents a message from the aggregation layer (i.e. After pooling processing), is the polymerization layer ( layer), is the indicator function, Indicates whether node d belongs to the node set in the current aggregation layer , its role is to ensure that only Only the nodes in the message aggregation update will be performed.

[0052] (2.4) Batch normalization layer and activation layer. The final output of the hierarchical GNN is: ; In this process, the embedding of the current layer It is based on the node embedding of the previous layer Derived; ELU is the exponential linear unit, which is used as the activation function.

[0053] (2.5) Final reasoning embedding vector extraction: The embedding vector of the final knowledge graph node is embedded from the last layer of the knowledge graph Extract the inference embedding vector It is obtained by concatenating the embedding vectors of each knowledge graph node: ; in, is the embedding vector of each node in the knowledge graph, m is the number of nodes in the knowledge graph, and the symbol Used for concatenation of embedding vectors, D is the total dimension of the concatenated embedding vectors of all nodes, , It is the output embedding dimension of the k+2th layer, that is, the output dimension of the last layer of the graph neural network, that is, the dimension of the embedding vector of each node.

[0054] Those skilled in the art will understand that the output of the previous layer is the input of the next layer. Considering that there is pooling between each sub-layer to prevent overfitting, the information involved in the above sub-layer introduction is the information before pooling. Therefore, the symbols are not described uniformly. Taking the dense layer and the information transfer layer as an example, Finally, pooling is required to obtain the node embeddings of the source and target nodes, and use them as the input of the information transfer layer. The same is true for the subsequent steps, so I will not go into details.

[0055] (3) Short-term time module.

[0056] In order to extract richer video information, the present invention uses a temporal model based on Transformer As a short-term module The video surveillance data is divided into multiple groups of sub-data, each of which contains an embedding sequence consisting of the inference embedding vectors corresponding to several consecutive frames (for example, from the t-(T-1)th frame to the tth frame) After feature extraction and fusion of short-term time modules, the final frame is output , corresponding to the input embedding of the last frame.

[0057] Of course, the sub-data of other groups are processed in the same way and will not be listed here.

[0058] Those skilled in the art will understand that in a Transformer-based model, the input vector is passed to multiple layers of the model, where it is processed and a new embedding representation is generated. Typically, each layer of the Transformer can extract features at different levels, and the output of each layer contains information about different aspects of the input. The embedding of the last layer of the Transformer is usually used as the input for model decision-making. Because the Transformer uses a self-attention mechanism, the output of the last layer contains the richest information after multiple layers of self-attention and feature aggregation, and can most accurately represent the characteristics of the input data.

[0059] (4) Anomaly detection decision module.

[0060] Output of the short-term time module Sent to the anomaly detection decision module , the anomaly detection model consists of a simple linear layer and a softmax function: ; in, and is a trainable parameter whose output The predicted probability for each frame is provided.

[0061] Specifically: video frame The probability that it is normal is ,in, express Item 1 of each exception type The probability of ,in, express The i+1th item in n is the number of exception types, Represents a video frame The probability of abnormality, Represents a video frame The conditional probability of anomaly type i when is an anomaly.

[0062] The decision model is trained by constructing a loss function in combination with the predicted probability, and the parameters in the model are updated until the stopping condition is met (for example, the loss function converges or the set number of training times is reached). The loss function calculation process and related training process involved in this part can be implemented with reference to conventional technology, and will not be elaborated in this invention.

[0063] 3. Adaptive learning of knowledge graphs.

[0064] In an embodiment of the present invention, the trained decision model is deployed in the edge device to perform real-time anomaly detection on the input video surveillance data to be detected in combination with the knowledge graph. In addition, the real-time anomaly detection results are tracked, and the knowledge graph is adaptively updated according to the changes in the anomaly score distribution of each video frame. When a new anomaly type is determined to have occurred or the existing pattern has changed based on the changes in the anomaly score distribution, the knowledge graph is updated through back propagation.

[0065] like Figure 3 As shown, the method of updating the knowledge graph includes: tokenizing the knowledge graph, obtaining the token embedding of each node in the knowledge graph, and calculating the distance between each node and other nodes based on the token embedding; in an embodiment of the present invention, after the knowledge graph is initially generated and tokenized, the token embedding of each node is updated. Specifically, after the abnormal data point is detected and the loss function is calculated, these loss functions are back-propagated to the node embedding in the knowledge graph. The back-propagation process will cause the node embedding vector related to the abnormal data to be updated, and the token embedding is fine-tuned based on the difference between the current embedding and the new data. The process uses the gradient descent method to update the token embedding of the node, and optimizes the token embedding so that it can represent the new abnormal type. For the current node, after updating the node's token embedding, the distance between the token embedding before and after the update and its adjacent nodes is calculated, and a determination is made as to whether the distance has increased. If the distance has not increased, it is assumed that the current node is converging toward a certain concept, and no action is performed. The next node is then created or updated. If the distance has increased, it indicates a deviation and the current node and the corresponding edge need to be deleted. A new node is then randomly created in the same layer as the deleted node, edges with adjacent layers are randomly added, and token embeddings are randomly assigned. For example, for the current node A, after updating the token embedding, the distance between the token embedding before and after the update and the token embedding of its adjacent node B is calculated. If the distance has increased, the edge between the current node A and its adjacent node B needs to be deleted.

[0066] For example, the distance between token embeddings before and after the update can be measured using the L2 distance (Euclidean distance).

[0067] In an embodiment of the present invention, a change in the existing pattern means that a video frame that is currently considered non-abnormal is re-identified as abnormal after calculating the anomaly score. When a new anomaly type appears, the embedding of the existing node can be adjusted, or if the embedding value of a node in the knowledge graph changes significantly, indicating that it cannot well represent the current abnormal data or behavior, it will be considered that this may be due to the emergence of a new anomaly type. At this time, a new node needs to be added to represent this type. Although the initial embedding of the node is random, it provides a new semantic space for the subsequent learning process. In the initial stage, these newly generated nodes have no specific meaning, but through continuous learning and backpropagation, the system will compare the embedding vector of the new abnormal data with the random embedding of the new node. When the embedding vector of the new anomaly type becomes similar to the embedding vector of a new node, this node will learn and represent the anomaly type.

[0068] The adaptive learning mechanism provided by the embodiments of the present invention enables the knowledge graph to continuously adapt to new abnormal patterns without the need for cloud connection or retraining of the entire decision model. Even when relying solely on edge devices, this mechanism can ensure that the system can effectively detect anomalies in a dynamic environment.

[0069] 3. Example description

[0070] In this example, factory video surveillance data and historical anomaly data are used as input to construct a task-specific knowledge graph based on a large language model. This graph represents key elements of the production environment (such as equipment status, worker behavior, and production processes), and the generated knowledge graph is used to reason about abnormal behavior. This process is unsupervised, requiring only a large amount of historical data to train the model and build the knowledge graph. The task-specific knowledge graph is then used to train a lightweight anomaly detection decision model using a graph neural network. In this model, surveillance video frames are converted into embedding vectors and fused with node embeddings in the knowledge graph. Information is transferred and reasoned through a multi-layer graph neural network to determine whether each frame contains abnormal behavior. Once the model is trained, it can be deployed to edge devices for real-time anomaly detection, reducing reliance on cloud computing resources.

[0071] For scenarios where edge devices have limited computing resources and lack continuous access to the cloud, this invention introduces a continuous knowledge graph adaptive learning mechanism. This mechanism can monitor and update the knowledge graph in real time during actual operation after deployment. Furthermore, it continuously tracks the anomaly probability distribution of each video frame and dynamically selects new abnormal behavior patterns for learning based on changes in the anomaly probability distribution. When new anomaly types are detected or existing patterns change, the embedding vectors in the knowledge graph are updated, and the graph structure is adjusted to adapt to the new data trends. This process does not rely on cloud computing, ensuring that the device can continue to optimize even when offline.

[0072] After completing the above process, this invention can be deployed in a real-world smart manufacturing environment. Integrating with the factory's monitoring system, it provides an end-to-end anomaly detection service using real-time surveillance video data. The entire solution can run independently on edge devices, monitoring and identifying various abnormal behaviors that may occur during the production process in real time. It also automatically updates and optimizes the detection model, ensuring the system's continued effective operation in a dynamically changing environment.

[0073] Through the above description of the embodiments, those skilled in the art will clearly understand that the above embodiments can be implemented via software or by utilizing software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product. This software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or mobile hard drive) and includes a number of instructions for enabling a computer device (such as a personal computer, server, or network device) to execute the methods described in various embodiments of the present invention.

[0074] Example 2 The present invention also provides a video anomaly detection system based on edge devices, which is mainly used to implement the method provided in the above embodiment, such as Figure 4 As shown, the system mainly includes: The knowledge graph construction unit is used to build the corresponding knowledge graph in combination with the specified task; A decision model training unit is used to train a decision model by combining video surveillance data, historical anomaly data, and the knowledge graph. The decision model includes: a video frame encoding module, a graph neural network, a short-term time module, and an anomaly detection decision module. During the training process, each video frame of the video surveillance data is encoded into an embedding vector by the video frame encoding module and attached to the knowledge graph. The graph neural network aggregates the information of each embedding vector with the knowledge graph to obtain an inference embedding vector corresponding to each video frame. The short-term time module performs comprehensive processing on the inference embedding vectors of several consecutive video frames to obtain a temporal embedding feature vector. The anomaly detection decision module performs anomaly detection on the temporal embedding feature vector and outputs the detection result of each video frame in the several consecutive video frames. The decision model is trained based on the detection results. The anomaly detection and knowledge graph update unit is used to deploy the trained decision model in the edge device, and perform real-time anomaly detection on the input video surveillance data to be detected in combination with the knowledge graph; and, based on the real-time anomaly detection results, adaptively update the knowledge graph.

[0075] Considering that the main technical details involved in the system have been introduced in detail in the previous embodiments, they will not be repeated here.

[0076] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0077] Example 3 The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the aforementioned embodiment.

[0078] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0079] In the embodiment of the present invention, the specific types of the memory, input device, and output device are not limited; for example: The input device can be a touch screen, image acquisition device, physical button or mouse; The output device may be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0080] Example 4 The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0081] In the embodiments of the present invention, the computer-readable storage medium may be provided in the aforementioned processing device, for example, as a memory in the processing device. Alternatively, the computer-readable storage medium may be a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0082] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. A video anomaly detection method based on edge devices, characterized in that: include: Combined with the specified tasks, build the corresponding knowledge graph; In combination with the video surveillance data and the knowledge graph, a decision model is trained, the decision model comprising: a video frame encoding module, a graph neural network, a short-term time module, and an anomaly detection decision module; during the training process, each video frame of the video surveillance data is encoded into an embedding vector by the video frame encoding module and attached to the knowledge graph; each embedding vector is aggregated with information in the knowledge graph by the graph neural network to obtain an inference embedding vector corresponding to each video frame; the inference embedding vectors of several consecutive video frames are comprehensively processed by the short-term time module to obtain a temporal embedding feature vector; the temporal embedding feature vector is subjected to anomaly detection by the anomaly detection decision module, and a detection result of each video frame in the several consecutive video frames is output; the decision model is trained based on the detection results; The trained decision model is deployed in the edge device, and the input video surveillance data to be detected is subjected to real-time anomaly detection in combination with the knowledge graph; and the knowledge graph is adaptively updated in combination with the real-time anomaly detection results.

2. The method for detecting anomalies in a video based on an edge device according to claim 1, wherein: The construction of the corresponding knowledge graph in combination with the specified task includes: Combined with the specified task and the preset prompt format, a knowledge graph is generated based on the large language model; the knowledge graph is a hierarchical directed acyclic graph, where each node represents a specified concept expressed in a text and is assigned to a specified level. Each edge connects two nodes in adjacent levels and points from the i-th level to the i+1-th level.

3. The method for detecting anomalies in a video based on an edge device according to claim 2, wherein: Generating a knowledge graph based on a large language model includes: Initial nodes are created to form the first layer of the knowledge graph. Afterwards, each iteration is performed based on the nodes generated in the current layer, with each iteration performing node generation, edge generation, and error detection and correction steps. In the node generation step, a preset prompt format is used to guide the large language model to generate nodes in the next layer that can be derived from the nodes in the current layer. In the edge generation step, a preset prompt format is used to guide the large language model to establish connecting edges between the nodes in the current layer and the next layer. In the error detection and correction step, whether there are errors is detected. If so, the large language model is used to correct them. If not, the next iteration is entered. Continue to iterate until the set number of layers is reached to obtain a knowledge graph in the form of a layered directed acyclic graph.

4. The method for detecting anomalies in a video based on an edge device according to claim 1, wherein: The video frame encoding module encodes each video frame of the video surveillance data into an embedding vector and appends it to the knowledge graph; aggregating each embedding vector with information in the knowledge graph through a graph neural network to obtain an inference embedding vector corresponding to each video frame includes: Through the video frame encoding module, each video frame of the video surveillance data is encoded into an embedding vector, and each video frame is attached to the knowledge graph as a sensor node. As the top layer of the knowledge graph, the embedding vector of each video frame is used as the representation of the corresponding sensor node. An embedding node is created and attached to the knowledge graph as the bottom layer of the knowledge graph. After the k-layer knowledge graph is attached with sensor nodes and embedding nodes, it becomes k+2 layers. The graph neural network has a k+2 layer structure, which is responsible for propagating the embedding vector corresponding to each sensor node layer by layer until it reaches the embedding node, and obtains the inference embedding vector of the aggregated embedding vector and the information in the knowledge graph.

5. The method for detecting anomalies in a video based on an edge device according to claim 1, wherein: The inference embedding vectors of several consecutive video frames are comprehensively processed by the short-term time module to obtain a temporal embedding feature vector; Performing anomaly detection on the temporal embedded feature vector by an anomaly detection decision module and outputting a detection result of each video frame in the plurality of consecutive video frames includes: The video surveillance data is divided into multiple groups of sub-data, each group of sub-data contains several consecutive frames, and the inference embedding vectors of the several consecutive video frames contained in each group are subjected to feature extraction and fusion through a short-term time module to obtain a temporal embedding feature vector; The anomaly detection decision module includes a linear layer and a softmax function layer. The input time series embedding feature vector is processed by the linear layer and the softmax function layer in sequence, and the detection results of each video frame in each group of sub-data are output. The detection results include: the probability of the video frame belonging to the normal type and the probability of belonging to each abnormal type; wherein the softmax function is a normalized exponential function.

6. The method for detecting anomalies in a video based on an edge device according to claim 1, wherein: The adaptive updating of the knowledge graph based on the real-time anomaly detection results includes: Track real-time anomaly detection results and adaptively update the knowledge graph based on changes in the anomaly score distribution of each video frame. When changes in the anomaly score distribution indicate the emergence of new anomaly types or changes in existing patterns, the knowledge graph is updated.

7. A method for detecting anomalies in a video based on an edge device according to claim 1 or 6, characterized in that: The ways to update the knowledge graph include: Tokenize the knowledge graph to obtain the token embedding of each node in the knowledge graph, and calculate the distance between each node and other nodes based on the token embedding, where token represents a label; For the current node, after updating the node's token embedding, calculate the distance between the token embedding and its adjacent nodes before and after the update, and determine whether the distance has increased; If the distance has not increased, do nothing; If the distance increases, the current node and the corresponding edge are deleted, and a new node is randomly created at the same layer as the deleted node, edges with adjacent layers are randomly added, and token embeddings are randomly assigned.

8. A video anomaly detection system based on edge devices, characterized in that: include: The knowledge graph construction unit is used to build the corresponding knowledge graph in combination with the specified task; A decision model training unit is used to train a decision model by combining video surveillance data, historical anomaly data, and the knowledge graph. The decision model includes: a video frame encoding module, a graph neural network, a short-term time module, and an anomaly detection decision module. During the training process, each video frame of the video surveillance data is encoded into an embedding vector by the video frame encoding module and attached to the knowledge graph. The graph neural network aggregates the information of each embedding vector and the knowledge graph to obtain an inference embedding vector corresponding to each video frame. The short-term time module performs comprehensive processing on the inference embedding vectors of several consecutive video frames to obtain a temporal embedding feature vector. The anomaly detection decision module performs anomaly detection on the temporal embedding feature vector and outputs the detection result of each video frame in the several consecutive video frames. The decision model is trained based on the detection results. The anomaly detection and knowledge graph update unit is used to deploy the trained decision model in the edge device, and perform real-time anomaly detection on the input video surveillance data to be detected in combination with the knowledge graph; and, based on the real-time anomaly detection results, adaptively update the knowledge graph.

9. A processing device, characterized in that include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images