Abnormity detection method and system based on TOF and RGB deep fusion perception

By using a deep fusion perception method combining Time-of-Flight (TOF) and RGB sensors to construct an anomaly detection dataset, and employing a neural network model for training and real-time detection, the problem of insufficient detection accuracy and stability in complex environments is solved, achieving high-precision anomaly location positioning and timely alarm.

CN118967549APending Publication Date: 2024-11-15BEIJING CHINASOFT INT INFORMATION TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410869642.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-01
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing anomaly detection methods lack accuracy and stability in complex environments and are affected by factors such as changes in lighting, occlusion, and background complexity.

Method used

A TOF and RGB depth fusion sensing method is adopted. The TOF sensor collects depth data and the RGB sensor collects image data to build an anomaly detection dataset. A neural network model is used for training and testing to achieve real-time fusion sensing and generate alarm information.

Benefits of technology

It improves the accuracy and stability of anomaly detection, can accurately locate anomalies and issue timely alarms, and adapts to complex environmental changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967549A_ABST
    Figure CN118967549A_ABST
Patent Text Reader

Abstract

The invention discloses an anomaly detection method and system based on TOF and RGB deep fusion perception, and relates to the technical field of smart perception, and the method comprises the steps: simulating an anomaly condition based on a target detection scene, and obtaining a depth data set and an anomaly detection data set through TOF and RGB sensors; and modeling the depth data, and judging the depth change in an abnormal condition. And constructing a detection and classification neural network model, and performing training and testing by using the anomaly detection data set to obtain a fusion perception model. And deploying the fusion perception model to a hardware platform, connecting a TOF sensor and an RGB sensor, carrying out fusion perception on the real-time depth data and the image data, and obtaining an anomaly detection result. If the abnormal condition exists, alarm information is generated based on an abnormal detection result and transmitted to the cloud platform, and an alarm is given in an acousto-optic mode. The technical effect of improving the detection precision and the detection stability is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent perception technology, and in particular to an anomaly detection method and system based on TOF and RGB deep fusion perception. Background Art

[0002] Anomaly detection is an important technical means to ensure system stability and security. Existing anomaly detection methods mainly rely on single sensor data, such as temperature, smoke, gas and other sensor data. When facing a complex environment with various oil and gas pipelines, valves, valves, pistons and other key components, they are easily affected by factors such as lighting changes, occlusion and background complexity, and there are technical problems such as insufficient detection accuracy and stability. Summary of the invention

[0003] The purpose of this application is to provide an anomaly detection method and system based on TOF and RGB deep fusion perception, so as to solve the technical problems of insufficient detection accuracy and stability in the prior art.

[0004] In view of the above technical problems, the present application provides an anomaly detection method and system based on TOF and RGB deep fusion perception.

[0005] In a first aspect, the present application provides an anomaly detection method based on TOF and RGB deep fusion perception, wherein the method comprises:

[0006] Based on the target detection scenario, simulate abnormal situations, use the TOF sensor to collect original depth information, calculate the depth data from the object to the camera through the algorithm, continuously collect depth data at a preset frame rate, obtain a depth data set, collect image data through the RGB sensor, and build an abnormal detection data set; model the depth data to determine the depth change when an abnormal situation occurs; clean and annotate the abnormal detection data set, and divide it into a training set, a validation set, and a test set; use the detection and classification methods to build a neural network model respectively, and use the divided abnormal detection data set for training and testing to obtain a fusion perception model; deploy the fusion perception model to the hardware platform, connect the TOF sensor and the RGB sensor, fuse the depth data and image data obtained in real time, determine whether there is abnormal behavior, and accurately locate the abnormal position through coordinates to obtain the abnormal detection result, wherein the abnormal detection result includes the abnormal type and abnormal position; if there is an abnormal situation, generate an alarm information based on the abnormal detection result, transmit the alarm information to the cloud platform, and issue an abnormal alarm through sound and light.

[0007] In a second aspect, the present application also provides an anomaly detection system based on TOF and RGB deep fusion perception, wherein the system comprises:

[0008] The sample data module is used to simulate abnormal situations based on target detection scenarios, use the TOF sensor to collect original depth information, calculate the depth data from the object to the camera through an algorithm, continuously collect depth data at a preset frame rate, obtain a depth data set, collect image data through an RGB sensor, and build an abnormal detection data set.

[0009] A depth modeling module is used to model the depth data and determine the depth changes when an abnormal situation occurs.

[0010] The pre-processing module is used to clean and label the anomaly detection data set and divide it into a training set, a validation set and a test set.

[0011] The perception training module is used to construct a neural network model using detection and classification methods respectively, and use the divided anomaly detection data set for training and testing to obtain a fusion perception model.

[0012] The abnormal fusion perception module is used to deploy the fusion perception model to the hardware platform, and connect the TOF sensor and the RGB sensor to perform fusion perception on the depth data and image data acquired in real time, determine whether there is abnormal behavior, and accurately locate the abnormal position through coordinates to obtain the abnormal detection result, wherein the abnormal detection result includes the abnormal type and abnormal position.

[0013] The transmission alarm module is used to generate alarm information based on the abnormal detection result if an abnormal situation occurs, transmit the alarm information to the cloud platform, and issue an abnormal alarm through sound and light.

[0014] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0015] Based on the target detection scenario, simulate abnormal situations, use the TOF sensor to collect original depth information, calculate the depth data from the object to the camera, continuously collect depth data at a preset frame rate, and obtain a depth data set. Collect image data through the RGB sensor to build an abnormal detection data set. Model the depth data to determine the depth change in abnormal situations. Clean and annotate the abnormal detection data set, and divide it into training set, validation set, and test set. Build a detection and classification neural network model, and use the abnormal detection data set for training and testing to obtain a fusion perception model. Deploy the fusion perception model to the hardware platform, connect the TOF sensor and RGB sensor, fuse the real-time depth data and image data to determine whether there is abnormal behavior, and accurately locate the abnormal position through coordinates to obtain the abnormal detection results, including the abnormal type and location. If there is an abnormal situation, generate an alarm based on the abnormal detection result, transmit it to the cloud platform, and issue an alarm through sound and light. Then achieve the technical effect of improving detection accuracy and detection stability.

[0016] The above description is only an overview of the technical solution of the present application. In order to more clearly illustrate the technical means of the present application and to implement it in accordance with the contents of the specification, and to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The embodiments of the present invention and the following brief description are illustrated in conjunction with the accompanying drawings, which are described as follows:

[0018] Figure 1 This is a flow chart of the anomaly detection method based on TOF and RGB deep fusion perception in this application;

[0019] Figure 2 This is a structural diagram of the anomaly detection system based on TOF and RGB deep fusion perception in this application.

[0020] Explanation of the accompanying drawings: sample data module 11, deep modeling module 12, pre-processing module 13, perception training module 14, abnormal fusion perception module 15, transmission alarm module 16. DETAILED DESCRIPTION

[0021] The present application solves the technical problems of insufficient detection accuracy and stability faced by the prior art by providing an anomaly detection method and system based on TOF and RGB deep fusion perception.

[0022] The solution in the embodiment of the present technology, in order to solve the above problems, adopts the following overall ideas:

[0023] First, based on the target detection scenario, simulate the abnormal situation, use the TOF sensor to collect the original depth information, and calculate the depth data from the object to the camera through the algorithm, continuously collect the depth data of the preset frame rate, obtain the depth data set, collect the image data through the RGB sensor, and build the abnormal detection data set. Then, model the depth data to determine the depth change when the abnormal situation occurs. Then, clean and annotate the abnormal detection data set, and divide it into a training set, a validation set, and a test set. Then, use the detection and classification methods to build a neural network model respectively, and use the divided abnormal detection data set for training and testing to obtain a fusion perception model. Then, deploy the fusion perception model to the hardware platform, connect the TOF sensor and the RGB sensor, fuse the real-time acquired depth data and image data, determine whether there is abnormal behavior, and accurately locate the abnormal position through coordinates to obtain the abnormal detection result, wherein the abnormal detection result includes the abnormal type and abnormal position. Finally, if there is an abnormal situation, generate an alarm message based on the abnormal detection result, transmit the alarm message to the cloud platform, and issue an abnormal alarm through sound and light.

[0024] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the drawings and specific implementation methods of the specification. It should be noted that the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments of the present application. It should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. It should also be noted that for the sake of ease of description, only the parts related to the present invention are shown in the drawings, rather than all of them.

[0025] Embodiment 1

[0026] like Figure 1 As shown, the present application provides an anomaly detection method based on TOF and RGB deep fusion perception, the method comprising:

[0027] Based on the target detection scenario, simulate abnormal situations, use the TOF sensor to collect original depth information, and use the algorithm to calculate the depth data from the object to the camera, continuously collect depth data at a preset frame rate, obtain the depth data set, collect image data through the RGB sensor, and build an abnormal detection data set;

[0028] Optionally, abnormal scene simulation is performed. First, a target detection scene is constructed, that is, a detection scene containing various target objects is constructed. For example, for the environment in which a train internal combustion engine is located, there are various oil and gas pipelines, valves, valves, pistons and other key components inside. Then, in the target detection scene, abnormal situations (such as leakage, damage, occlusion, etc.) are manually or automatically created to ensure that the acquired abnormal detection data set contains multiple abnormalities.

[0029] Optionally, a TOF (Time-of-Flight) sensor is installed in the target detection scene to collect raw depth information. An RGB sensor (camera) is installed to collect image data of the target scene. Next, the TOF sensor is used to continuously collect depth data from the target object to the camera, ensuring that the preset frame rate (such as 30 frames per second) is used for collection to ensure the continuity and stability of the data. Then, the raw depth information collected by the TOF sensor is processed by an algorithm to calculate the accurate depth data from the target object to the camera. Specifically, the TOF sensor calculates the distance between the target object and the camera by measuring the time required for the light pulse to be emitted from the camera and reflected back to the sensor.

[0030] Optionally, use an RGB sensor (camera) to collect image data of the target detection scene, and perform time synchronization processing on the collected depth data and image data to ensure that the depth data and image data at the same time point correspond one to one. Then, combine the depth data set and the image data set to form a multimodal anomaly detection data set containing depth information and image information. By simulating abnormal situations and combining the data of the TOF sensor and the RGB sensor, a high-quality anomaly detection data set is constructed, which provides a data foundation for the construction of subsequent perception analysis.

[0031] Furthermore, based on the target detection scenario, abnormal situations are simulated, including:

[0032] Obtain historical environment information of target detection scenes and obtain multiple sets of detection environment parameters;

[0033] Based on the multiple sets of detection environment parameters, various abnormal conditions with different brightness and angles are simulated, and different depth data are collected using a TOF sensor, wherein the collection time length ranges from 10 seconds to 1 minute;

[0034] Based on multiple sets of detection environment parameters, various abnormal conditions with different brightness and angles are simulated, and different image data are collected using RGB sensors. The collection time ranges from 10 seconds to 1 minute.

[0035] Optionally, historical environmental information of the target detection scene is collected, including parameters such as temperature, humidity, light intensity, angle, etc. Then, multiple sets of detection environment parameters are extracted based on the historical environmental information to ensure that various typical scene conditions are covered.

[0036] Optionally, according to the light intensity in the detection environment parameters, the abnormal conditions under different brightness are simulated, and according to the angle information in the detection environment parameters, the abnormal conditions under different angles are simulated. And the TOF sensor and the RGB sensor are used respectively to collect depth data and image data of different time lengths (ranging from 10 seconds to 1 minute) under different simulated brightness and angle conditions. Specifically, the image data can be a dynamic image or a static image, and the image data is obtained based on the preset resolution and image quality parameters.

[0037] Model the depth data to determine the depth changes when abnormal situations occur;

[0038] Furthermore, the depth data is modeled to determine the depth changes when an abnormal situation occurs, including:

[0039] Preprocessing the collected depth data set, and marking different abnormal situations based on the preprocessing results;

[0040] Modeling is performed based on the labeled depth data set to extract depth change features for determining the occurrence of abnormal situations.

[0041] Optionally, the collected depth data set is preprocessed, including denoising, smoothing, and normalization steps to ensure data quality. Then, based on the preprocessing results, different abnormal situations are annotated to ensure that the abnormal situation corresponding to each data point is clear.

[0042] Optionally, the change features in the depth data are extracted, including the fluctuation and mutation features of the depth value, and a mapping model between the depth features and the abnormal situation is established based on the pattern recognition technology. Specifically, the modeling based on the depth data set can select a mathematical model or a decision tree model. Exemplarily, the depth change features include the first-order features of the depth change (gradient information describing the depth change, reflecting the rate of change of the depth value), the second-order features (curvature information describing the depth change, reflecting the change in the rate of change of the depth value) and the distribution features (describing the distribution of the depth value in the image, which is achieved through statistical methods such as histogram, mean and variance).

[0043] Optionally, the annotated depth dataset is used to train the established mapping model, optimize model parameters, and improve the accuracy of the model.

[0044] Clean and annotate the anomaly detection dataset and divide it into training set, validation set and test set;

[0045] Furthermore, the anomaly detection dataset is cleaned, labeled, and divided into training set, validation set, and test set including:

[0046] Cleaning the collected RGB image data and expanding the data set using data enhancement operations, wherein the expanded anomaly detection data set contains at least 10,000 RGB images;

[0047] Different abnormal situations are distinguished and marked by labelImg.exe software, and the area to be detected is drawn;

[0048] Using Python language, divide the annotated anomaly detection dataset into a training set, a test set, and a validation set in a ratio of 8:1:1, wherein the training set contains at least 8,000 images, the test set contains at least 1,000 images, and the validation set contains at least 1,000 images;

[0049] Randomly extract pictures from the annotated anomaly detection dataset, repeat the operation at least 25,000 times, and score each picture from 0 to 100 to construct an anomaly scoring dataset, wherein a larger score value indicates a more reliable anomaly, and divide the anomaly scoring dataset into a training set, a test set, and a validation set in a ratio of 8:1:1, wherein the training set contains at least 20,000 pictures, the test set contains at least 2,500 pictures, and the validation set contains at least 2,500 pictures.

[0050] Specifically, first, the collected RGB image data is cleaned to remove invalid images such as blur and noise to ensure the quality of the data set. Then, the data set is expanded through data enhancement operations, including rotation, cropping, flipping, color adjustment, etc., so that the expanded anomaly detection data set contains at least 10,000 RGB images.

[0051] Optionally, use labelImg.exe software to distinguish and mark different abnormal situations, draw the area to be detected, and save the marked information as a standard XML or JSON format file to facilitate subsequent processing.

[0052] Optionally, use Python to divide the labeled anomaly detection dataset into a training set, a test set, and a validation set in a ratio of 8:1:1, where the training set contains at least 8,000 images, the test set contains at least 1,000 images, and the validation set contains at least 1,000 images.

[0053] Furthermore, randomly extract pictures from the annotated anomaly detection dataset and repeat at least 25,000 times. Score each picture from 0 to 100, and generate a data set containing scoring information, i.e., anomaly scoring data set, based on the scoring results. Then, divide the anomaly scoring data set into a training set, a test set, and a validation set in a ratio of 8:1:1. The training set includes at least 20,000 pictures, the test set includes up to 2,500 pictures, and the validation set includes up to 2,500 pictures.

[0054] Optionally, the scoring of the image includes a scoring based on professional technicians or based on a target detection algorithm, and the specific scoring based on the target detection algorithm is determined based on the probability of the target detection anchor box annotation.

[0055] Use detection and classification methods to build neural network models respectively, and use the divided anomaly detection data set for training and testing to obtain a fusion perception model;

[0056] Optionally, based on the target detection and classification algorithms, multiple neural network models are constructed respectively. Exemplarily, the target detection algorithms include those based on yolo, RNN, Fast-RNN, etc., and the classification algorithms include KNN, MobileNe, etc.

[0057] Optionally, a detection neural network model and a classification neural network model are constructed for depth data and image data, respectively, to further improve the accuracy and robustness of detection and classification. Exemplarily, a neural network model suitable for three-dimensional point cloud or deep image processing, such as PointNet++, VoxelNet or SparseConvNet, is selected to construct a detection neural network model and a classification neural network model for depth data. A neural network model suitable for two-dimensional image detection, such as the Yolo series, MobileNe series, Faster R-CNN or RetinaNet, is selected to construct a detection neural network model and a classification neural network model for image data.

[0058] Through the above steps, the detection and classification neural network models for depth data and image data were constructed respectively, and then these models were fused to construct a multimodal fusion perception model. This model can comprehensively utilize the feature information of depth data and image data to improve the accuracy and robustness of detection and classification.

[0059] Furthermore, neural network models are constructed using detection and classification methods respectively, and the divided anomaly detection data sets are used for training and testing to obtain a fusion perception model, including:

[0060] Use the Pytorch framework and Python code, use the Yolov8 framework to build a detection network, and use the anomaly detection dataset for training and testing to obtain the detection network;

[0061] Use the Pytorch framework and Python code to build a classification network using the MobileNetV3 framework, and use the anomaly scoring dataset for training and testing to obtain a classification network;

[0062] The detection network and the classification network are connected to generate the fusion perception model.

[0063] Optionally, choose the Yolov8 framework to build a detection network. This architecture is suitable for efficient target detection tasks. Then, use the anomaly detection dataset for training and testing. The anomaly detection dataset includes cleaned, annotated, and expanded RGB image data and depth image data.

[0064] Optionally, use the MobileNetV3 framework to build a classification network, and perform training and testing. This architecture is suitable for efficient image classification tasks. Then, use the anomaly scoring dataset for training and testing. The anomaly scoring dataset includes annotated image data with larger scores indicating more reliable anomalies. The classification network acquires the ability to classify detected abnormal areas by learning the mapping relationship between image features and annotated categories in annotated images.

[0065] Optionally, the detection network is connected to the classification network to generate a fusion perception model. First, the corresponding detection network and classification network are cascaded, that is, the abnormal area image output by the detection network is input into the classification network for classification, and a depth perception channel and an image perception channel are obtained. Then, a dynamic information sharing channel is established between the depth perception channel and the image perception channel. The dynamic information sharing channel is used to share the interference factors obtained by the image perception channel to the depth perception channel, thereby eliminating the interference of external objects on the depth information.

[0066] Deploy the fusion perception model to the hardware platform, connect the TOF sensor and the RGB sensor, perform fusion perception on the depth data and image data acquired in real time, determine whether there is abnormal behavior, and accurately locate the abnormal position through coordinates to obtain the abnormal detection result, wherein the abnormal detection result includes the abnormal type and abnormal position;

[0067] Optionally, the hardware platform refers to a computing platform or hardware system used for anomaly detection in the target detection environment. Exemplarily, the hardware platform includes an embedded device, a GPU accelerator or other high-performance computing device to ensure the real-time computing capability of the model.

[0068] Optionally, a physical connection between the TOF sensor and the RGB sensor and the hardware platform is established through a physical data interface, and a transmission protocol is defined based on a virtual data channel to establish a data connection between the TOF sensor and the RGB sensor and the hardware platform to achieve the above connection.

[0069] Exemplarily, real-time data is fused and perceived to monitor abnormal situations. First, the depth data and image data acquired in real time are preprocessed, including denoising, normalization and other operations. Then, the preprocessed depth data and image data are input into the fusion perception model. The fusion perception model combines the feature information of the detection network and the classification network to determine whether there is abnormal behavior and determine the type of abnormality. Then, according to the coordinates of the abnormal area output by the detection network, the depth data is combined for spatial positioning to determine the abnormal position, wherein the Z-axis information of the depth data is used for precise positioning to obtain the three-dimensional coordinates of the abnormality. Finally, a detection report containing the abnormal type and abnormal position is generated, and the detection results are output through an interface or display device for user viewing and analysis.

[0070] The above method steps use the high-performance computing capabilities of the hardware platform, and the fusion perception model can process depth data and image data in real time to quickly determine whether there is abnormal behavior. Using the depth information of the TOF sensor, the abnormal position can be accurately located and accurate three-dimensional coordinates can be provided. Combining the feature information of the detection network and the classification network, different types of anomalies can be fully identified, improving the accuracy and robustness of detection.

[0071] Furthermore, the fusion perception model is deployed to the hardware platform, and the TOF sensor and the RGB sensor are connected to perform fusion perception on the depth data and image data acquired in real time, including:

[0072] According to the target detection scenario, the TOF sensor and RGB sensor are installed with the goal of minimizing the blind spot, and a communication connection with the hardware platform is established to transmit data;

[0073] Deploy the trained fusion perception model to the hardware platform and transplant the deep data modeling results to the hardware platform;

[0074] Activate the TOF sensor and RGB sensor, synchronously collect and acquire real-time depth data and real-time image data, and input them into the fusion perception model for parallel detection classification and data fusion to obtain abnormal detection results.

[0075] Specifically, first, according to the target detection scenario, select the best location to install the TOF sensor and RGB sensor to ensure that the blind area is minimized. For example, the TOF sensor and RGB sensor are installed with the minimum blind area by professional technicians or multi-objective optimization algorithms. Then, the sensor is connected to the hardware platform through a communication interface (such as USB, UART, Ethernet, etc.) to ensure the stability and real-time performance of data transmission.

[0076] Optionally, deploy the trained fusion perception model to the hardware platform. The deployment process may involve model format conversion (such as conversion from Pytorch model to ONNX or TensorRT model) to adapt to the computing environment of the hardware platform. By porting the deep data modeling results to the hardware platform, it ensures that the model can correctly process the deep data, and helps improve the adaptability and portability of the fusion perception model.

[0077] Optional. Synchronously activate the TOF sensor and the RGB sensor, and ensure that the two collect data synchronously. Specifically, synchronous collection can be achieved through hardware triggers or software synchronization mechanisms. Exemplary, the software synchronization mechanism to achieve synchronous collection includes: synchronizing the clock (ensuring that the two sensors use the same clock source, or synchronizing the system clock); controlling the acquisition order (using software to control the acquisition order to minimize the time difference); timestamp alignment (adding a timestamp to each frame of data, and aligning the timestamp during subsequent processing).

[0078] Optionally, the synchronously collected real-time depth data and real-time image data are input into the fusion perception model. Parallel detection classification and data fusion are performed. The fusion perception model combines the feature information of the detection network and the classification network to process the input data and determine whether there is abnormal behavior. Among them, parallel detection refers to the perception analysis of real-time depth data and real-time image data through the depth perception channel and image perception channel in the fusion perception model respectively. Data fusion is to fuse the detection results from the depth data and RGB data to obtain the final anomaly detection result. Specifically, the anomaly detection result includes the anomaly type and the anomaly location.

[0079] In some implementations, during synchronous acquisition, the RGB sensor tracks the pedestrian's walking trajectory and eliminates the impact of the pedestrian's walking on the TOF sensor.

[0080] Optionally, eliminating the influence of the depth change caused by pedestrians on the TOF sensor includes predicting the position of the pedestrian in the depth data based on the pedestrian's motion trajectory and correcting the changes in the depth data caused by the pedestrian's motion. Exemplary correction methods include eliminating the depth data corresponding to the pedestrian's motion trajectory and processing the interpolation points based on time interpolation and space interpolation.

[0081] If an abnormal situation exists, an alarm message is generated based on the abnormal detection result, the alarm message is transmitted to the cloud platform, and an abnormal alarm is issued through sound and light.

[0082] Furthermore, if an abnormal situation exists, an alarm message is generated based on the abnormal detection result, the alarm message is transmitted to the cloud platform, and an abnormal alarm is issued by means of sound and light, including:

[0083] Performing abnormality classification according to the abnormality detection result to obtain the abnormality classification result;

[0084] Perform transmission judgment based on the abnormal classification result, and if the abnormal classification result meets the transmission constraint, generate the alarm information and transmit it to the cloud platform;

[0085] A local alarm is determined based on the abnormal classification result. If the abnormal classification result satisfies the local alarm constraint, a sound alarm and a light alarm are issued according to the alarm information.

[0086] Specifically, first, based on the real-time anomaly detection results, the anomaly is processed and analyzed. Then, based on the predefined classification rules, the anomaly is classified according to the severity, type and location of the detection results to obtain the anomaly classification results.

[0087] Furthermore, according to the abnormal classification results, transmission judgment is performed. If the abnormal classification results meet the transmission constraints (such as high severity or specific types of abnormalities), an alarm information is generated and transmitted to the cloud platform for recording and processing. Among them, the alarm information includes detailed information such as abnormality type, location, time, etc.

[0088] Furthermore, based on the abnormality classification results, local alarm discrimination is performed. If the abnormality classification results meet the local alarm constraints (such as medium or high severity or a specific type of abnormality), the local alarm operation is performed. According to the alarm information, sound alarms (such as alarm sounds) and light alarms (such as flashing lights) are issued to alert on-site personnel.

[0089] The above method steps can be used to classify abnormalities according to different severity levels, improve the accuracy and timeliness of alarms, and combine cloud platforms and local alarms to achieve a multi-level response mechanism, which helps ensure that abnormal situations are handled in a timely manner. At the same time, sound and light alarms can be used to prompt on-site personnel in real time, allowing them to respond quickly and reduce the harm caused by abnormalities.

[0090] In summary, the anomaly detection method based on TOF and RGB deep fusion perception provided by the present invention has the following technical effects:

[0091] Based on the target detection scenario, simulate abnormal situations, use the TOF sensor to collect original depth information, calculate the depth data from the object to the camera, continuously collect depth data at a preset frame rate, and obtain a depth data set. Collect image data through the RGB sensor to build an abnormal detection data set. Model the depth data to determine the depth change in abnormal situations. Clean and annotate the abnormal detection data set, and divide it into training set, validation set, and test set. Build a detection and classification neural network model, and use the abnormal detection data set for training and testing to obtain a fusion perception model. Deploy the fusion perception model to the hardware platform, connect the TOF sensor and RGB sensor, fuse the real-time depth data and image data to determine whether there is abnormal behavior, and accurately locate the abnormal position through coordinates to obtain the abnormal detection results, including the abnormal type and location. If there is an abnormal situation, generate an alarm based on the abnormal detection result, transmit it to the cloud platform, and issue an alarm through sound and light. Then achieve the technical effect of improving detection accuracy and detection stability.

[0092] Embodiment 2

[0093] Based on the same concept as the anomaly detection method based on TOF and RGB depth fusion perception in the embodiment, Figure 2 As shown, the present application also provides an anomaly detection system based on TOF and RGB deep fusion perception, the system comprising:

[0094] The sample data module 11 is used to simulate abnormal situations based on the target detection scenario, collect original depth information using the TOF sensor, calculate the depth data from the object to the camera through the algorithm, continuously collect depth data at a preset frame rate, obtain a depth data set, collect image data through the RGB sensor, and construct an abnormal detection data set;

[0095] A depth modeling module 12 is used to model the depth data and determine the depth change when an abnormal situation occurs;

[0096] The pre-processing module 13 is used to clean and label the anomaly detection data set and divide it into a training set, a validation set and a test set;

[0097] A perception training module 14 is used to construct a neural network model using detection and classification methods respectively, and use the divided anomaly detection data set for training and testing to obtain a fusion perception model;

[0098] The abnormal fusion perception module 15 is used to deploy the fusion perception model to the hardware platform, connect the TOF sensor and the RGB sensor, perform fusion perception on the depth data and image data acquired in real time, determine whether there is abnormal behavior, and accurately locate the abnormal position through coordinates to obtain the abnormal detection result, wherein the abnormal detection result includes the abnormal type and abnormal position;

[0099] The transmission alarm module 16 is used to generate alarm information based on the abnormal detection result if an abnormal situation exists, transmit the alarm information to the cloud platform, and issue an abnormal alarm through sound and light.

[0100] Furthermore, the sample data module 11 also includes:

[0101] An environment analysis unit, used to obtain historical environment information of target detection scenes and obtain multiple sets of detection environment parameters;

[0102] A depth simulation acquisition unit, used to simulate various abnormal conditions of different brightness and different angles based on the multiple sets of detection environment parameters, and use a TOF sensor to collect different depth data, wherein the acquisition time length ranges from 10 seconds to 1 minute;

[0103] The image simulation acquisition unit is used to simulate various abnormal conditions with different brightness and angles based on multiple sets of detection environment parameters, and use the RGB sensor to collect different image data, where the acquisition time ranges from 10 seconds to 1 minute.

[0104] Furthermore, the depth modeling module 12 also includes:

[0105] A preprocessing and labeling unit, used to preprocess the collected depth data set and label different abnormal situations based on the preprocessing results;

[0106] The feature modeling unit is used to perform modeling based on the labeled depth data set and extract depth change features for determining the occurrence of abnormal situations.

[0107] Furthermore, the pre-processing module 13 also includes:

[0108] A cleaning and expansion unit, used to clean the collected RGB image data and expand the data set using a data enhancement operation, wherein the expanded anomaly detection data set contains at least 10,000 RGB images;

[0109] The classification and labeling unit is used to distinguish and label different abnormal situations through the labelImg.exe software and draw the area to be detected;

[0110] An overall division unit, used to divide the annotated anomaly detection data set into a training set, a test set, and a validation set in a ratio of 8:1:1 using Python language, wherein the training set contains at least 8,000 images, the test set contains at least 1,000 images, and the validation set contains at least 1,000 images;

[0111] The partition division unit is used to randomly intercept pictures from the annotated anomaly detection data set, repeat the operation at least 25,000 times, and score each picture from 0 to 100 to construct an anomaly scoring data set, wherein a larger score value indicates a more reliable anomaly, and divide the anomaly scoring data set into a training set, a test set, and a validation set in a ratio of 8:1:1, wherein the training set contains at least 20,000 pictures, the test set contains at least 2,500 pictures, and the validation set contains at least 2,500 pictures.

[0112] Furthermore, the perception training module 14 also includes:

[0113] Detection network configuration unit, which is used to build a detection network using the Yolov8 framework using the Pytorch framework and Python code, and to obtain the detection network using the anomaly detection dataset for training and testing;

[0114] The classification network configuration unit is used to build a classification network using the MobileNetV3 framework using the Pytorch framework and Python code, and to obtain a classification network using the anomaly scoring dataset for training and testing;

[0115] A connection building unit is used to connect the detection network and the classification network to generate the fusion perception model.

[0116] Furthermore, the abnormal fusion perception module 15 also includes:

[0117] The sensor deployment unit is used to install the TOF sensor and the RGB sensor with the minimum blind spot according to the target detection scenario, and establish a communication connection with the hardware platform to transmit data;

[0118] Model embedding unit, used to deploy the trained fusion perception model to the hardware platform and transplant the deep data modeling results to the hardware platform;

[0119] The real-time detection unit is used to activate the TOF sensor and the RGB sensor, synchronously collect and obtain real-time depth data and real-time image data, and input them into the fusion perception model for parallel detection classification and data fusion to obtain abnormal detection results.

[0120] In some implementations, the abnormal fusion perception module 15 also includes a trajectory correction unit, which is used for synchronous acquisition. The RGB sensor tracks the pedestrian's walking trajectory and eliminates the depth change effect of the pedestrian's walking on the TOF sensor according to the pedestrian's walking trajectory.

[0121] Furthermore, the transmission alarm module 16 also includes:

[0122] an abnormality grading unit, used to perform abnormality grading according to the abnormality detection result and obtain an abnormality grading result;

[0123] A transmission determination unit, configured to perform transmission determination based on the abnormal classification result, and if the abnormal classification result satisfies the transmission constraint, generate the alarm information and transmit it to the cloud platform;

[0124] The local alarm distinguishing unit is used to perform local alarm distinguishing based on the abnormal classification result, and if the abnormal classification result meets the local alarm constraint, sound alarm and light alarm are issued according to the alarm information.

[0125] It should be understood that the embodiments mentioned in this specification focus on their differences from other embodiments. The specific embodiments in the aforementioned embodiment one are also applicable to the anomaly detection system based on TOF and RGB deep fusion perception described in embodiment two. For the sake of brevity of the specification, no further elaboration is given here.

[0126] It should be understood that the embodiments disclosed in this application and the above description can enable those skilled in the art to use this application to implement this application. At the same time, this application is not limited to the above-mentioned embodiments, and obvious modifications, combinations and substitutions of the embodiments mentioned in this application also fall within the scope of protection of this application.

Claims

1. Anomaly detection method based on TOF and RGB deep fusion perception, characterized in that: The method comprises: Based on the target detection scenario, simulate abnormal situations, use the TOF sensor to collect original depth information, and use the algorithm to calculate the depth data from the object to the camera, continuously collect depth data at a preset frame rate, obtain the depth data set, collect image data through the RGB sensor, and build an abnormal detection data set; Model the depth data to determine the depth changes when abnormal situations occur; Clean and annotate the anomaly detection dataset and divide it into training set, validation set and test set; Use detection and classification methods to build neural network models respectively, and use the divided anomaly detection data set for training and testing to obtain a fusion perception model; Deploy the fusion perception model to the hardware platform, connect the TOF sensor and the RGB sensor, perform fusion perception on the depth data and image data acquired in real time, determine whether there is abnormal behavior, and accurately locate the abnormal position through coordinates to obtain the abnormal detection result, wherein the abnormal detection result includes the abnormal type and abnormal position; If an abnormal situation exists, an alarm message is generated based on the abnormal detection result, the alarm message is transmitted to the cloud platform, and an abnormal alarm is issued through sound and light.

2. The anomaly detection method based on TOF and RGB deep fusion perception as claimed in claim 1, characterized in that: Based on the target detection scenario, simulate abnormal situations, including: Obtain historical environment information of target detection scenes and obtain multiple sets of detection environment parameters; Based on the multiple sets of detection environment parameters, various abnormal conditions with different brightness and angles are simulated, and different depth data are collected using a TOF sensor, wherein the collection time length ranges from 10 seconds to 1 minute; Based on multiple sets of detection environment parameters, various abnormal conditions with different brightness and angles are simulated, and different image data are collected using RGB sensors. The collection time ranges from 10 seconds to 1 minute.

3. The anomaly detection method based on TOF and RGB deep fusion perception as claimed in claim 1, characterized in that: Model the depth data to determine the depth changes when abnormal situations occur, including: Preprocessing the collected depth data set, and marking different abnormal situations based on the preprocessing results; Modeling is performed based on the labeled depth data set to extract depth change features for determining the occurrence of abnormal situations.

4. The anomaly detection method based on TOF and RGB deep fusion perception as claimed in claim 1, characterized in that: The anomaly detection dataset is cleaned, labeled, and divided into training set, validation set, and test set, including: Cleaning the collected RGB image data and expanding the data set using data enhancement operations, wherein the expanded anomaly detection data set contains at least 10,000 RGB images; Different abnormal situations are distinguished and marked by labelImg.exe software, and the area to be detected is drawn; Using Python language, divide the annotated anomaly detection dataset into a training set, a test set, and a validation set in a ratio of 8:1:1, wherein the training set contains at least 8,000 images, the test set contains at least 1,000 images, and the validation set contains at least 1,000 images; Randomly extract pictures from the annotated anomaly detection dataset, repeat the operation at least 25,000 times, and score each picture from 0 to 100 to construct an anomaly scoring dataset, wherein a larger score value indicates a more reliable anomaly, and divide the anomaly scoring dataset into a training set, a test set, and a validation set in a ratio of 8:1:1, wherein the training set contains at least 20,000 pictures, the test set contains at least 2,500 pictures, and the validation set contains at least 2,500 pictures.

5. The anomaly detection method based on TOF and RGB deep fusion perception as claimed in claim 1, characterized in that: Use detection and classification methods to build neural network models respectively, and use the divided anomaly detection data set for training and testing to obtain a fusion perception model, including: Use the Pytorch framework and Python code, use the Yolov8 framework to build a detection network, and use the anomaly detection dataset for training and testing to obtain the detection network; Use the Pytorch framework and Python code to build a classification network using the MobileNetV3 framework, and use the anomaly scoring dataset for training and testing to obtain a classification network; The detection network and the classification network are connected to generate the fusion perception model.

6. The anomaly detection method based on TOF and RGB deep fusion perception as claimed in claim 1, characterized in that: The fusion perception model is deployed on the hardware platform, and the TOF sensor and RGB sensor are connected to perform fusion perception of the depth data and image data acquired in real time, including: According to the target detection scenario, the TOF sensor and RGB sensor are installed with the goal of minimizing the blind spot, and a communication connection with the hardware platform is established to transmit data; Deploy the trained fusion perception model to the hardware platform and transplant the deep data modeling results to the hardware platform; Activate the TOF sensor and RGB sensor, synchronously collect and acquire real-time depth data and real-time image data, and input them into the fusion perception model for parallel detection classification and data fusion to obtain abnormal detection results.

7. The anomaly detection method based on TOF and RGB deep fusion perception as claimed in claim 6, characterized in that: During synchronous acquisition, the RGB sensor will track the pedestrian's walking trajectory and eliminate the impact of the pedestrian's walking on the TOF sensor.

8. The anomaly detection method based on TOF and RGB deep fusion perception as claimed in claim 1, characterized in that: If there is an abnormal situation, an alarm message is generated based on the abnormal detection result, the alarm message is transmitted to the cloud platform, and an abnormal alarm is issued by sound and light, including: Performing abnormality classification according to the abnormality detection result to obtain the abnormality classification result; Perform transmission judgment based on the abnormal classification result, and if the abnormal classification result meets the transmission constraint, generate the alarm information and transmit it to the cloud platform; A local alarm is determined based on the abnormal classification result. If the abnormal classification result satisfies the local alarm constraint, a sound alarm and a light alarm are issued according to the alarm information.

9. Anomaly detection system based on TOF and RGB deep fusion perception, characterized in that: The system comprises: A sample data module is used to simulate abnormal situations based on target detection scenarios, collect original depth information using a TOF sensor, calculate depth data from the object to the camera through an algorithm, continuously collect depth data at a preset frame rate, obtain a depth data set, collect image data through an RGB sensor, and construct an abnormal detection data set; A depth modeling module, which is used to model the depth data and determine the depth change when an abnormal situation occurs; A pre-processing module, which is used to clean and annotate the anomaly detection data set and divide it into a training set, a validation set, and a test set; A perception training module, which is used to construct a neural network model using detection and classification methods respectively, and use the divided anomaly detection data set for training and testing to obtain a fusion perception model; An abnormal fusion perception module, which is used to deploy the fusion perception model to the hardware platform, connect the TOF sensor and the RGB sensor, perform fusion perception on the depth data and image data acquired in real time, determine whether there is abnormal behavior, and accurately locate the abnormal position through coordinates to obtain the abnormal detection result, wherein the abnormal detection result includes the abnormal type and abnormal position; The transmission alarm module is used to generate alarm information based on the abnormal detection result if an abnormal situation occurs, transmit the alarm information to the cloud platform, and issue an abnormal alarm through sound and light.

Citation Information

Cited By

  • Board splicing machine anomaly detection method and system based on vision intelligence

    CN121147642A

  • A vision intelligence-based abnormal detection method and system for a board splicing machine

    CN121147642B