A method for monitoring industrial spillage based on deep learning and graph neural network

Abstract

The present invention discloses an industrial spillage monitoring method based on deep learning and graph neural network, which relates to the field of intelligent monitoring and computer vision, including analysis and first target detection based on video stream, using deep learning model to identify and track engineering vehicles, and using OCR technology to extract license plate information; expanding the detection area for identifying and tracking engineering vehicles, using spillage recognition model to perform a second detection of suspected spillage in the vehicle area; performing a third detection based on the area of ​​suspected spillage to determine whether there is spillage. The present invention uses a deep learning model to identify engineering vehicles entering a specific area, identifies suspected spillage and records features; performs a third detection by calculating waiting time and image registration technology to eliminate false detections; constructs a heterogeneous graph through a graph neural network algorithm, extracts features and optimizes models, generates alarm levels and processing priorities, and improves safety monitoring capabilities in industrial scenarios.

Description

Technical Field

[0001] The present invention relates to the fields of intelligent monitoring and computer vision, and in particular to an industrial spillage monitoring method based on deep learning and graph neural network. Background Art

[0002] With the continuous development of industrial automation and intelligent monitoring technology, the monitoring and management of engineering vehicles in industrial scenarios has become increasingly important. Existing monitoring systems usually rely on fixed cameras and sensors to record the entry and exit of vehicles through manual monitoring or simple image processing technology. However, this approach has significant limitations, especially when it comes to the detection and management of vehicle spills. Traditional monitoring systems have difficulty in real-time and efficient detection and tracking of engineering vehicle spills, mainly because they cannot effectively cope with complex and changing industrial environments and dynamic vehicle behaviors. In recent years, with the rapid development of deep learning and computer vision technology, intelligent monitoring systems based on video streams have gradually become a research hotspot. These systems can detect, identify and track target objects through real-time processing of video data, greatly improving the automation and accuracy of monitoring.

[0003] Although the existing intelligent monitoring technology has made certain progress in vehicle detection and tracking, there are still many deficiencies in vehicle spill detection and information association. First, the existing technology has major problems in the accuracy and robustness of spill detection, especially in complex industrial scenarios, where spills and environmental backgrounds are easily confused, resulting in high false detection rates and missed detection rates. Secondly, in terms of the association between vehicle information and spill information, most existing methods rely on simple spatiotemporal matching algorithms, which cannot fully utilize the advantages of deep learning and graph neural networks, resulting in low accuracy and efficiency of information association. In addition, the existing system also has obvious deficiencies in processing multi-target tracking and real-time performance, and it is difficult to cope with the complex situation of multiple vehicles operating simultaneously in industrial scenarios. In contrast, the present invention proposes an industrial spill monitoring method based on deep learning and graph neural networks, which uses deep learning models and graph neural network association algorithms to achieve efficient and accurate detection and early warning of engineering vehicle spills through multiple detections and information association. Summary of the invention

[0004] In view of the problems existing in the existing intelligent monitoring system in vehicle spilled object detection and information association, the present invention is proposed.

[0005] Therefore, the problem to be solved by the present invention is how to achieve accurate detection of objects spilled by engineering vehicles and effective association of information in complex industrial scenarios.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In the first aspect, an embodiment of the present invention provides an industrial spillage monitoring method based on deep learning and graph neural networks, which includes: analyzing engineering vehicles entering a specific area based on a video stream and performing a first detection of the target; based on the first detection result, using a deep learning model to identify and track the engineering vehicles, and using OCR technology to extract license plate information; expanding the detection area for identifying and tracking engineering vehicles, and using a spillage recognition model to perform a second detection of suspected spillage in the vehicle area; performing a third detection based on the area of ​​suspected spillage to determine whether there is spillage; based on the three detection results, using a graph neural network association algorithm to associate the spillage information and the vehicle information; and generating an alarm information level and processing priority according to the association result.

[0008] As a preferred solution of the industrial spillage monitoring method based on deep learning and graph neural network described in the present invention, the first target detection includes the following steps: frame extraction of the input video stream; contrast adjustment and denoising of the extracted frames to enhance the image, and adjust the image to the input size required by the deep learning model; the relevant formula for the contrast adjustment is as follows:

[0009] ;

[0010] in, For the enhanced image, is the contrast gain factor, is the original image, is the brightness adjustment factor.

[0011] Use the pre-trained deep learning model to detect vehicles, set the detection threshold, and output the first detection result; classify engineering vehicles through the ResNet50 classification model, and retain the detection results classified as engineering vehicles; predefine the polygon coordinates of the monitoring area, make area entry judgments, and check whether the center point of the engineering vehicle boundary box is within this polygon; when the center point of the engineering vehicle boundary box is within this polygon, it is recorded as an entry event; when the center point of the engineering vehicle boundary box is outside this polygon, the center point of the engineering vehicle boundary box is recalibrated.

[0012] As a preferred solution of the industrial spillage monitoring method based on deep learning and graph neural network described in the present invention, the DeepSORT algorithm is used to track vehicles and a unique ID is assigned to each tracked vehicle; the vehicle tracking includes the following steps: The relevant formula for vehicle tracking is as follows:

[0013] ;

[0014] in, is the state vector at time k, is the state transfer matrix at time k, is the control input matrix at time k, is the control vector at time k, is the process noise at time k.

[0015] The Kalman filter algorithm is used to predict the position of the vehicle in the next frame, and the Hungarian algorithm is used to match the predicted position with the first detection result; the deep learning model is used to locate the license plate area within the vehicle boundary box; the positioning result is tilted, and the CRNN model is used to perform end-to-end license plate OCR and obtain the final license plate string through CTC decoding; the relevant formula for the tilt correction is as follows:

[0016] ;

[0017] in, is the tilt angle, and are the coordinates of two points on the edge of the license plate.

[0018] Use regular expressions to validate the license plate format and filter out incorrect recognition results; associate the recognized license plate information with the vehicle tracking ID and store it in a temporary database.

[0019] As a preferred solution of the industrial spillage monitoring method based on deep learning and graph neural network described in the present invention, the detection area is expanded, and the expansion range is dynamically adjusted according to the size and speed of the vehicle. and the standard deviation of the extended range The detection area expansion includes the following steps: The relevant formula for the detection area expansion is as follows:

[0020] ;

[0021] in, is the expanded detection area, To expand the range, is the standard deviation of the extended range, is a random variable, is the vehicle area, is the vehicle speed, and is the adjustment factor.

[0022] Use the Mask R-CNN model trained for spilled objects to perform instance segmentation in the extended area and set the initial detection threshold; classify each detected spilled object instance and record the relevant features of the suspected spilled object; the relevant features include the time, location, size and shape of occurrence; calculate the relative position relationship between the suspected spilled object and the vehicle; perform a preliminary screening of the suspected spilled objects based on predefined rules; and temporarily store the screened results pending the third test.

[0023] As a preferred solution of the industrial spillage monitoring method based on deep learning and graph neural network of the present invention, the third detection includes the following steps: calculating the waiting time, the specific formula is as follows:

[0024] ;

[0025] in, For waiting time, is the detected vehicle height, is the acceleration due to gravity, To monitor the area length, is the vehicle speed, is the adjustment factor.

[0026] Use image registration technology to map the previously recorded suspected spilled object location to the current frame, perform regional relocation, and appropriately expand the detection range; use the Mask R-CNN model to perform a third detection; perform shape and texture analysis on the detection results to eliminate false detections, and compare the two detection results; the false detections include leaves and shadows.

[0027] As a preferred embodiment of the industrial spillage monitoring method based on deep learning and graph neural network described in the present invention, the comparison of the two detection results includes: when a vehicle is detected to enter the monitoring area, the vehicle classification process is started; if the vehicle is classified as an engineering vehicle, the license plate is located and identified; if the license plate recognition is successful, the license plate information and the vehicle ID are associated and stored; if the license plate recognition fails, the vehicle is marked as an unrecognized license plate and continues to be tracked; when the engineering vehicle is driving in the monitoring area, the engineering vehicle location information is continuously updated; if the engineering vehicle stays for more than a preset threshold, a parking event warning is triggered; when multiple engineering vehicles are detected in the monitoring area at the same time, a multi-target tracking algorithm is started; if ID switching occurs in multi-target tracking, appearance features are used for reassociation; when the engineering vehicle is about to leave the monitoring area, the engineering vehicle is marked as a state to be observed; if the state to be observed lasts for more than a preset time, the vehicle in this process is removed from the tracking list; when the engineering vehicle enters the core position of the monitoring area, the detection range is expanded; if the detection range exceeds the camera field of view, it is automatically adjusted to the maximum visible range; If an object is detected in the extended area, the spill classification model is activated; if the object is classified as a potential spill, the relevant features of the potential spill are recorded; when several potential spills are detected, they are prioritized according to the relevant features of the spills; if the highest priority spill exceeds the danger threshold, an early warning is triggered immediately; when the third detection result is consistent with the second detection result, the object is marked as a confirmed spill; if the third detection result is inconsistent with the second detection result, a manual review is performed; when the existence of the spill is confirmed, the graph neural network association algorithm is immediately activated; if the association algorithm outputs several responsible vehicles, all vehicles and vehicle probabilities are listed; when the spill is confirmed and the responsible vehicle is identified, a complete incident report is generated; if the responsible vehicle cannot be determined, a general spill alarm is generated; when the incident report is generated, the severity of the incident is calculated according to the scoring system; if the severity exceeds the preset threshold, the emergency response process is initiated; when all detections are completed and report generation is generated, the system status is reset and the next round of detection is performed; if the system memory usage exceeds the threshold, the garbage collection program is triggered to release resources.

[0028] As a preferred solution of the industrial spillage monitoring method based on deep learning and graph neural network described in the present invention, wherein: a heterogeneous graph is constructed through spillage information and vehicle information; the nodes of the heterogeneous graph include vehicles, license plates and spillage; the construction of the heterogeneous graph includes the following steps: the weights of the heterogeneous graph edges are calculated based on spatiotemporal proximity and feature similarity to extract relevant features for the nodes; the relevant features include vehicle trajectories and spillage features; the features are reduced in dimension using an autoencoder to obtain nodes of uniform dimension, and the relationship between nodes is learned using a GAT model; training is performed using manually annotated association samples, and model hyperparameters are optimized through cross-validation; the association probability between the spillage node and the vehicle node is calculated using the trained GAT model, and the Hungarian algorithm is used for optimal matching to obtain the final association result; the relevant formula of the GAT model is as follows:

[0029] ;

[0030] in, is the attention coefficient of node i to node j, is an exponential function used to convert the attention score into a positive value, is a leaky linear rectifier unit activation function used to introduce nonlinearity. is the attention vector, which linearly transforms the input features. and are the feature vectors of node i and node j respectively, is the updated feature vector of node i, is the vector concatenation operation, is the neighbor set of node i, is the weight matrix, is the ReLU activation function.

[0031] In the second aspect, an embodiment of the present invention provides an industrial spill monitoring system based on deep learning and graph neural networks, which includes: a first target detection module, which analyzes engineering vehicles entering a specific area and performs a first target detection based on a video stream; an identification module, which uses a deep learning model to identify and track engineering vehicles based on the first detection result, and uses OCR technology to extract license plate information; a second detection module, which is used to expand the detection area for identifying and tracking engineering vehicles, and uses a spill recognition model to perform a second detection of suspected spills in the vehicle area; a third detection module, which performs a third detection based on the area of ​​suspected spills to determine whether there are spills; an association module, which uses a graph neural network association algorithm to associate spill information and vehicle information based on the three detection results; and a generation module, which generates an alarm information level and processing priority according to the association result.

[0032] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, the steps of the industrial spillage monitoring method based on deep learning and graph neural network as described in the first aspect of the present invention are implemented.

[0033] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, the steps of the industrial spillage monitoring method based on deep learning and graph neural network as described in the first aspect of the present invention are implemented.

[0034] The beneficial effects of the present invention are as follows: the present invention identifies engineering vehicles entering a specific area through frame extraction, image enhancement and deep learning models of video streams; uses the DeepSORT algorithm and Kalman filtering to track vehicles, and extracts license plate information through OCR technology to associate vehicle identity and trajectory; dynamically adjusts the detection area and uses the Mask R-CNN model for a second detection to identify suspected spilled objects and record features; performs a third detection by calculating waiting time and using image registration technology to eliminate false detections; constructs a heterogeneous graph through a graph neural network algorithm, extracts features and optimizes models, generates alarm levels and processing priorities, and improves safety monitoring capabilities in industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0036] Figure 1 This is a flow chart of the industrial spill monitoring method based on deep learning and graph neural network in Example 1.

[0037] Figure 2 This is a flowchart of the first target detection of the industrial spillage monitoring method based on deep learning and graph neural network in Example 1. DETAILED DESCRIPTION

[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0039] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0040] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0041] Example 1

[0042] Reference Figure 1~Figure 2 , which is the first embodiment of the present invention, and provides an industrial spillage monitoring method based on deep learning and graph neural network, comprising:

[0043] S1: Analysis of engineering vehicles entering a specific area and first detection of targets based on video streams.

[0044] Specifically, video stream preprocessing includes the following steps: extract frames from the input video stream, usually extracting 10-30 frames per second; the original video frame rate can be adjusted, and the system will automatically calculate the appropriate sampling frame rate; the relevant formula for frame extraction is as follows:

[0045] ;

[0046] in, is the sampling frame rate, is the original video frame rate.

[0047] Furthermore, the extracted frames are contrast adjusted and denoised to enhance the image and resize the image to the input size required by the deep learning model; this can be adjusted using the slider and Value, real-time view image enhancement effect, contrast adjustment related formula is as follows:

[0048] ;

[0049] in, For the enhanced image, is the contrast gain factor, is the original image, is the brightness adjustment factor.

[0050] Furthermore, a pre-trained deep learning model is used for vehicle detection, and a detection threshold is set. For example, a detection is considered valid only when the confidence level is > 0.6. Non-maximum suppression (NMS) is performed on the detection results to eliminate duplicate detections.

[0051] Specifically, a pre-trained deep learning model is used for vehicle detection, and a detection threshold is set to output the first detection result. The engineering vehicles are classified through the ResNet50 classification model, and the detection results classified as engineering vehicles are retained. The polygon coordinates of the monitoring area are predefined, and the area entry judgment is performed to check whether the center point of the engineering vehicle boundary box is within this polygon. When the center point of the engineering vehicle boundary box is within this polygon, it is recorded as an entry event. When the center point of the engineering vehicle boundary box is outside this polygon, the center point of the engineering vehicle boundary box is recalibrated.

[0052] It should be noted that the steps to build a deep learning model include: collecting and preprocessing data, including data cleaning, normalization and segmentation into training sets, validation sets and test sets; selecting the deep learning model architecture and defining the parameters of each layer; training the model by calculating the output through forward propagation, measuring the error using the loss function, and updating the weights using the backpropagation algorithm; evaluating the model performance using the validation set and test set, and adjusting the hyperparameters and model structure to improve the model accuracy and generalization ability.

[0053] S2: Based on the first detection results, a deep learning model is used to identify and track engineering vehicles, and OCR technology is used to extract license plate information.

[0054] Specifically, the DeepSORT algorithm is used for vehicle tracking, and a unique ID is assigned to each tracked vehicle; the relevant formula for vehicle tracking is as follows:

[0055] ;

[0056] in, is the state vector at time k, is the state transfer matrix at time k, is the control input matrix at time k, is the control vector at time k, is the process noise at time k.

[0057] Furthermore, the Kalman filter algorithm is used to predict the position of the vehicle in the next frame, and the Hungarian algorithm is used to match the predicted position with the first detection result; the deep learning model is used to locate the license plate area within the vehicle boundary box; the positioning result is tilted, and the CRNN model is used to perform end-to-end license plate OCR and obtain the final license plate string through CTC decoding; the relevant formula for tilt correction is as follows:

[0058] ;

[0059] in, is the tilt angle, and are the coordinates of two points on the edge of the license plate.

[0060] It should be noted that the CRNN (Convolutional Recurrent Neural Network) model is a deep learning architecture that combines convolutional neural networks (CNN) and recurrent neural networks (RNN). It is mainly used to process sequence data, such as time series or spatial sequence data, and has the ability to capture local features and long-term dependencies at the same time. The CRNN model first extracts the feature representation of the input sequence through the convolutional layer, and then uses the recurrent layer to learn the temporal dependencies in the sequence, effectively modeling the long-term dependencies in the sequence, and is suitable for tasks such as speech recognition, video analysis, and text processing.

[0061] Furthermore, a regular expression is used to verify the license plate format and filter out erroneous recognition results; the recognized license plate information is associated with the vehicle tracking ID and stored in a temporary database.

[0062] It should be noted that the system can display the original license plate image and the corrected image to confirm the correction effect.

[0063] Specifically, when a vehicle is detected entering the monitoring area, the vehicle classification process is started; if the vehicle is classified as a construction vehicle, the license plate is located and identified; if the license plate recognition is successful, the license plate information and vehicle ID are associated and stored; if the license plate recognition fails, the vehicle is marked as an unrecognized license plate and continues to be tracked; when the construction vehicle is traveling in the monitoring area, the construction vehicle location information is continuously updated; if the construction vehicle stays for longer than a preset threshold, a parking event warning is triggered; when multiple construction vehicles are detected in the monitoring area at the same time, a multi-target tracking algorithm is started; if ID switching occurs in multi-target tracking, appearance features are used for re-association; when the construction vehicle is about to leave the monitoring area, the construction vehicle is marked as being in a state to be observed; if the state to be observed continues for more than a preset time, the vehicle in this process is removed from the tracking list.

[0064] S3: Expand the detection area for identifying and tracking engineering vehicles, and use the spilled object recognition model to perform a second detection of suspected spilled objects in the vehicle area.

[0065] Specifically, the detection area is expanded, and the expansion range is dynamically adjusted according to the size and speed of the vehicle and the standard deviation of the extended range Ensure that the extended area does not exceed the camera's field of view; the relevant formula for detecting area expansion is as follows:

[0066] ;

[0067] in, is the expanded detection area, To expand the range, is the standard deviation of the extended range, is a random variable, is the vehicle area, is the vehicle speed, and is the adjustment factor.

[0068] It should be noted that by adjusting and Change the size of the detection area.

[0069] Furthermore, the Mask R-CNN model trained for spilled objects is used to perform instance segmentation in the extended area and set the initial detection threshold; each detected instance of spilled objects is classified and the relevant features of the suspected spilled objects are recorded; the relevant features include the time, location, size and shape of occurrence; the relative position relationship between the suspected spilled objects and the vehicle is calculated; the suspected spilled objects are preliminarily screened based on predefined rules; and the screened results are temporarily stored pending the third detection.

[0070] It should be noted that the Mask R-CNN (Mask Region-based Convolutional Neural Network) model is a deep learning model for target detection and instance segmentation. By introducing an additional segmentation branch, it is possible to accurately generate segmentation masks for each instance while detecting the target. Mask R-CNN combines a region proposal network (RPN) for target bounding box proposals and a convolutional neural network (CNN) for feature extraction, while adding a fully convolutional network (FCN) branch to generate pixel-level segmentation masks. This enables Mask R-CNN to efficiently complete the tasks of target detection and precise instance segmentation within a single framework, and is widely used in image and video analysis fields, such as medical image segmentation, obstacle recognition in autonomous driving, and pedestrian detection in video surveillance.

[0071] Furthermore, when an engineering vehicle enters the core position of the monitoring area, the detection range is expanded; if the detection range exceeds the camera's field of view, it is automatically adjusted to the maximum visible range; if an object is detected in the expanded area, the spill classification model is activated; if the object is classified as a potential spill, the relevant features of the potential spill are recorded; when several potential spills are detected, they are prioritized according to size and location; if the highest priority spill exceeds the danger threshold, an early warning is immediately triggered.

[0072] S4: Conduct a third inspection based on the area suspected of being spilled to determine whether there is any spilled object.

[0073] Specifically, the waiting time is calculated using the following formula:

[0074] ;

[0075] in, For waiting time, is the detected vehicle height, is the acceleration due to gravity, To monitor the area length, is the vehicle speed, is the adjustment factor.

[0076] Furthermore, image registration technology is used to map the previously recorded location of suspected spilled objects to the current frame, perform regional relocation, and appropriately expand the detection range; the Mask R-CNN model is used for a third detection; the detection results are subjected to shape and texture analysis to eliminate false detections, and the two detection results are compared; false detections include leaves and shadows.

[0077] Furthermore, when a construction vehicle is about to leave the monitoring area, it is marked as a state to be observed; if the state to be observed lasts for more than a preset time, the vehicle in this process is removed from the tracking list; when the construction vehicle enters the core position of the monitoring area, the detection range is expanded; if the detection range exceeds the camera field of view, it is automatically adjusted to the maximum visible range; if an object is detected in the extended area, the spill classification model is activated; if the object is classified as a potential spill, the relevant features of the potential spill are recorded; when several potential spills are detected, they are prioritized according to the relevant features of the spills; if the highest priority spill exceeds the danger threshold, an early warning is triggered immediately; when the third detection result is consistent with the second detection result, The object is marked as confirmed spilled object; if the third test result is inconsistent with the second test result, a manual review is carried out; when the existence of spilled objects is confirmed, the graph neural network association algorithm is immediately started; if the association algorithm outputs several responsible vehicles, all vehicles and vehicle probabilities are listed; when the spilled object is confirmed and the responsible vehicle is identified, a complete incident report is generated; if the responsible vehicle cannot be determined, a general spilled object alarm is generated; when the incident report is generated, the severity of the incident is calculated according to the scoring system; if the severity exceeds the preset threshold, the emergency response process is initiated; when all tests are completed and report generation is completed, the system status is reset and the next round of tests is carried out; if the system memory usage exceeds the threshold, the garbage collection program is triggered to release resources.

[0078] S5: Based on the three detection results, the graph neural network association algorithm is used to associate the spilled object information and vehicle information.

[0079] Specifically, a heterogeneous graph is constructed through the information of spilled objects and vehicles; the nodes of the heterogeneous graph include vehicles, license plates and spilled objects; the weights of the edges of the heterogeneous graph are calculated based on the spatiotemporal proximity and feature similarity to extract relevant features for the nodes; the relevant features include vehicle trajectories and spilled object features; the autoencoder is used to reduce the dimension of the features to obtain nodes of uniform dimension, and the GAT model is used to learn the relationship between the nodes; the manually annotated associated samples are used for training, and the model hyperparameters are optimized through cross-validation; the relevant formulas of the GAT model are as follows:

[0080] ;

[0081] in, is the attention coefficient of node i to node j, is an exponential function used to convert the attention score into a positive value, is a leaky linear rectifier unit activation function used to introduce nonlinearity. is the attention vector, which linearly transforms the input features. and are the feature vectors of node i and node j respectively, is the updated feature vector of node i, is the vector concatenation operation, is the neighbor set of node i, is the weight matrix, is the ReLU activation function.

[0082] It should be noted that the GAT model (Graph Attention Network) is a graph neural network that uses the attention mechanism to process graph structure data. It effectively captures nonlinear dependencies in complex graphs by learning the relationship weights between nodes. Unlike traditional graph convolutional networks, the GAT model introduces an attention mechanism to assign different attention weights when aggregating neighbor node features, thereby extracting key information more accurately.

[0083] Furthermore, the association probability between the spilled object node and the vehicle node is calculated through the trained GAT model, and the Hungarian algorithm is used for optimal matching to obtain the final association result.

[0084] It should be noted that the system can visualize the attention weights to understand the strength of association between different nodes.

[0085] S6: Generate alarm information level and processing priority according to the correlation result.

[0086] Specifically, a multi-factor scoring system is set up, in which the hazard degree of the spill is based on a predefined list of hazardous substances, and a hazard score of 0-10 is assigned; the spill volume is assigned a magnitude score of 0-5 according to the detected spill area; the location importance is a predefined area importance map, and a location score of 0-3 is assigned according to the spill location; the vehicle record is to check the historical records of the vehicle in the database, and 1-2 points will be added if there is a bad record; the association credibility is to map the confidence of the GAT output to 0-2 points.

[0087] Furthermore, the total score is calculated using the weighted summation method; the alarm information is generated by dividing the level and setting the processing time limit according to the total score; the level is divided into: <10 points for low level, 10-20 points for medium level, and >20 points for high level; the processing time limit is set: low level within 24 hours, medium level within 2 hours, and high level within 30 minutes;

[0088] Furthermore, according to the preset notification matrix, the recipients of different levels of alarms are determined; alarm information is sent simultaneously through multiple channels (system notification, SMS and email); an alarm confirmation mechanism is implemented to track the processing status of each alarm. Furthermore, this embodiment also provides an industrial spillage monitoring system based on deep learning and graph neural network, including: a target first detection module, which analyzes the entry of engineering vehicles into a specific area and performs a first target detection based on a video stream; an identification module, based on the first detection result, uses a deep learning model to identify and track engineering vehicles, and uses OCR technology to extract license plate information; a second detection module, which is used to expand the detection area for identifying and tracking engineering vehicles, and uses a spillage recognition model to perform a second detection of suspected spillage in the vehicle area; a third detection module, which performs a third detection based on the area of ​​suspected spillage to determine whether there is spillage; an association module, based on the three detection results, uses a graph neural network association algorithm to associate spillage information and vehicle information; a generation module, which generates an alarm information level and processing priority according to the association result.

[0089] This embodiment also provides a computer device, which is suitable for the industrial spillage monitoring method based on deep learning and graph neural network, including a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the industrial spillage monitoring method based on deep learning and graph neural network proposed in the above embodiment.

[0090] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0091] The present embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps: performing analysis of engineering vehicles entering a specific area and first detecting a target based on a video stream; based on the result of the first detection, using a deep learning model to identify and track engineering vehicles, and using OCR technology to extract license plate information; expanding the detection area for identifying and tracking engineering vehicles, and using a spilled object recognition model to perform a second detection of suspected spilled objects in the vehicle area; performing a third detection based on the area of ​​suspected spilled objects to determine whether there are any spilled objects; based on the results of the three detections, using a graph neural network association algorithm to associate the spilled object information with the vehicle information; and generating an alarm information level and processing priority based on the association result.

[0092] In summary, the present invention identifies engineering vehicles entering a specific area through frame extraction, image enhancement and deep learning models of video streams; uses the DeepSORT algorithm and Kalman filtering for vehicle tracking, extracts license plate information through OCR technology, and associates vehicle identity and trajectory; dynamically adjusts the detection area and uses the Mask R-CNN model for a second detection to identify suspected spilled objects and record features; performs a third detection by calculating waiting time and image registration technology to eliminate false detections; constructs a heterogeneous graph through a graph neural network algorithm, extracts features and optimizes the model, generates alarm levels and processing priorities, and improves safety monitoring capabilities in industrial scenarios.

[0093] Example 2

[0094] Referring to Table 1, which is the second embodiment of the present invention, this embodiment provides an industrial spillage monitoring method based on deep learning and graph neural network. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0095] Specifically, a 30-day field test was conducted in a large industrial park. The test area was a main road about 1 km long in the park, which has a large traffic volume and often has engineering vehicles passing through. The test system includes 5 high-definition cameras distributed at key locations on the road, covering the entire monitoring area. The system hardware configuration is a high-performance server equipped with NVIDIA Tesla V100 GPU for deep learning model inference.

[0096] Furthermore, the camera is precisely calibrated; according to step S1 in the invention content, the video stream is preprocessed. After multiple debugging, the sampling frame rate is finally determined to be 25fps, the contrast gain factor α is set to 1.2, and the brightness adjustment factor β is set to 0.8. According to step S2, the YOLOv5 model is used for vehicle detection, and the detection threshold is set to 0.65. The ResNet50 classification model has undergone transfer learning and is specifically used to identify engineering vehicles. The DeepSORT algorithm is used for vehicle tracking, in which the process noise covariance matrix Q and the measurement noise covariance matrix R of the Kalman filter are repeatedly tuned to adapt to complex industrial scenarios.

[0097] Furthermore, the license plate recognition module uses an improved CRNN model to improve the recognition accuracy of dirty and tilted license plates on the locally collected industrial vehicle license plate dataset. The tilt correction algorithm uses Hough transform combined with RANSAC, which can effectively handle tilt angles of up to 45 degrees. In steps S3 and S4, the dynamic expansion algorithm of the spill detection area is adaptively adjusted according to the vehicle type and speed. In the calculation formula of the expansion range, the adjustment coefficient k1 is set to 0.8 and k2 is set to 0.5; the Mask R-CNN model improves the recognition ability of various industrial wastes on the locally collected industrial spill dataset.

[0098] Specifically, the GAT model in step S5 uses a two-layer attention mechanism, with the hidden layer dimension set to 64, a learning rate of 0.001, and is trained using the Adam optimizer. The training data set contains 1,000 manually annotated spillage event samples, and the model hyperparameters are optimized through 5-fold cross-validation. The multi-factor scoring system in step S6 was determined after multiple rounds of discussions with park safety experts, and the weight distribution was optimized through sensitivity analysis. During the test, the system detected a total of 5,000 engineering vehicles passing through, including 200 suspected spillage events.

[0099] Furthermore, as shown in Table 1, the present invention is significantly superior to the prior art in all key indicators. The vehicle recognition accuracy rate increased by 6.7%, and the license plate recognition accuracy rate increased by 9.3%, thanks to the advanced deep learning model combination and image processing technology. Particularly outstanding is the substantial improvement of 21.1% in the accuracy of spilled object detection and the significant reduction of 84.2% in the false alarm rate, which is mainly due to the innovative multiple detection strategy and dynamic expansion of the detection area. The system response time was shortened by 78.9%, and the number of daily failures was reduced by 93.3%, which fully demonstrates the excellent performance of the present invention in algorithm efficiency and system stability. These data strongly demonstrate the significant advantages of the present invention in improving the safety management and operational efficiency of industrial parks.

[0100] Table 1 Comparison between the present invention and the prior art

[0101] Performance Indicators The present invention Prior art Performance improvement (%) Performance Vehicle Identification Accuracy (%) Index 98.5 92.3 6.7 License plate recognition accuracy (%) 97.5 89.5 9.3 Accuracy of spilled object detection (%) 95.2 78.6 21.1 False alarm rate of spilled objects (%) 2.5 15.8 84.2 Average response time (seconds) 1.2 5.7 78.9 System stability (number of failures per day) 0.1 1.5 93.3

[0102] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for monitoring industrial spillage based on deep learning and graph neural network, characterized in that: include, Analyze the entry of engineering vehicles into specific areas and detect targets for the first time based on video streams; Based on the first detection results, a deep learning model is used to identify and track engineering vehicles, and OCR technology is used to extract license plate information; Expand the detection area for identifying and tracking engineering vehicles, and use the spilled object recognition model to conduct a second detection of suspected spilled objects in the vehicle area; Conduct a third inspection based on the suspected spilled area to determine whether there is any spilled object; Based on the three detection results, the graph neural network association algorithm is used to associate the spilled object information with the vehicle information; Generate alarm information level and processing priority according to the correlation results; Use the DeepSORT algorithm for vehicle tracking and assign a unique ID to each tracked vehicle; The vehicle tracking comprises the following steps: The relevant formula for the vehicle tracking is as follows: ； in, is the state vector at time k, is the state transfer matrix at time k, is the control input matrix at time k, is the control vector at time k, is the process noise at time k; Use the Kalman filter algorithm to predict the vehicle's position in the next frame, and use the Hungarian algorithm to match the predicted position with the first detection result; Use a deep learning model to locate the license plate area within the vehicle bounding box; Perform tilt correction on the positioning result, use the CRNN model to perform end-to-end license plate OCR, and obtain the final license plate string through CTC decoding; The relevant formula for the tilt correction is as follows: ； in, is the tilt angle, and are the coordinates of two points on the edge of the license plate; Use regular expressions to verify the license plate format and filter out incorrect recognition results; Associating the identified license plate information with the vehicle tracking ID and storing it in a temporary database; Construct a heterogeneous graph through the information of spilled objects and vehicles; The nodes of the heterogeneous graph include vehicles, license plates and spilled objects; The process of constructing a heterogeneous graph comprises the following steps: The weights of the edges in the heterogeneous graph are calculated based on spatiotemporal proximity and feature similarity to extract relevant features for the nodes; The relevant features include vehicle trajectory and spilled object features; Use autoencoders to reduce the dimension of features to obtain nodes of uniform dimension, and use the GAT model to learn the relationship between nodes; Use manually annotated correlation samples for training and optimize model hyperparameters through cross-validation; The trained GAT model is used to calculate the association probability between the spilled object node and the vehicle node, and the Hungarian algorithm is used for optimal matching to obtain the final association result. The relevant formula of the GAT model is as follows: ； in, is the attention coefficient of node i to node j, is an exponential function used to convert the attention score into a positive value, is a leaky linear rectifier unit activation function used to introduce nonlinearity. is the attention vector, which linearly transforms the input features. and are the feature vectors of node i and node j respectively, is the updated feature vector of node i, is the vector concatenation operation, is the neighbor set of node i, is the weight matrix, is the ReLU activation function.

2. The industrial spillage monitoring method based on deep learning and graph neural network according to claim 1, characterized in that: The first target detection comprises the following steps: Extract frames from the input video stream; Perform contrast adjustment and denoising on the extracted frames to enhance the images and resize the images to the input dimensions required by the deep learning model; The relevant formula for contrast adjustment is as follows: ； in, For the enhanced image, is the contrast gain factor, is the original image, is the brightness adjustment factor; Use the pre-trained deep learning model to detect vehicles, set the detection threshold, and output the first detection result; The engineering vehicles are classified by the ResNet50 classification model, and the detection results classified as engineering vehicles are retained; Predefine the polygon coordinates of the monitoring area, make area entry judgment, and check whether the center point of the engineering vehicle boundary box is within this polygon; When the center point of the engineering vehicle bounding box is within the polygon, it is recorded as an entry event; when the center point of the engineering vehicle bounding box is outside the polygon, the center point of the engineering vehicle bounding box is recalibrated.

3. The industrial spillage monitoring method based on deep learning and graph neural network according to claim 2, characterized in that: Detection area expansion, dynamically adjust the expansion range according to vehicle size and speed and the standard deviation of the extended range The detection area expansion comprises the following steps: The relevant formula for the detection area expansion is as follows: ； in, is the expanded detection area, To expand the range, is the standard deviation of the extended range, is a random variable, is the vehicle area, is the vehicle speed, and is the adjustment factor; Use the Mask R-CNN model trained for spilled objects to perform instance segmentation in the expanded area and set the initial detection threshold; Classify each detected spill instance and record the relevant features of the suspected spill; The relevant features include appearance time, location, size and shape; Calculate the relative position of the suspected spilled object and the vehicle; Conduct preliminary screening of suspected spills based on predefined rules; The screening results are temporarily stored and await the third test.

4. The industrial spillage monitoring method based on deep learning and graph neural network according to claim 3, characterized in that: The third detection comprises the following steps: Calculate the waiting time. The specific formula is as follows: ； in, For waiting time, is the detected vehicle height, is the acceleration due to gravity, To monitor the area length, is the vehicle speed, is the adjustment factor; Use image registration technology to map the previously recorded suspected spilled object locations to the current frame, perform regional relocation, and appropriately expand the detection range; Use the Mask R-CNN model for the third detection; Perform shape and texture analysis on the detection results to eliminate false detections and compare the two detection results; The false detections include foliage and shadows.

5. The industrial spillage monitoring method based on deep learning and graph neural network according to claim 4, characterized in that: The comparison of the two test results comprises: When a vehicle is detected entering the monitoring area, the vehicle classification process is initiated; If the vehicle is classified as an engineering vehicle, license plate location and recognition is performed; If the license plate recognition is successful, the license plate information and vehicle ID are associated and stored; If the license plate recognition fails, the vehicle is marked as unrecognized and continues to be tracked; when the construction vehicle is driving in the monitoring area, the location information of the construction vehicle is continuously updated; if the construction vehicle stays for longer than the preset threshold, a parking event warning is triggered; when multiple construction vehicles are detected in the monitoring area at the same time, the multi-target tracking algorithm is started; if ID switching occurs in multi-target tracking, the appearance features are used for re-association; When the engineering vehicle is about to leave the monitoring area, it will be marked as a waiting state; If the observation state lasts longer than the preset time, the vehicle will be removed from the tracking list; when the engineering vehicle enters the core position of the monitoring area, the detection range will be expanded; if the detection range exceeds the camera's field of view, it will automatically adjust to the maximum visible range; If an object is detected in the extended area, the spill classification model is activated; if the object is classified as a potential spill, the relevant features of the potential spill are recorded; When several potential spills are detected, they are prioritized based on their relevant features; If the highest priority spilled objects exceed the danger threshold, an early warning is triggered immediately; When the third test result is consistent with the second test result, the object is marked as a confirmed spilled object; If the third test result is inconsistent with the second test result, a manual review will be conducted; when it is confirmed that spilled objects exist, the graph neural network association algorithm will be immediately started; if the association algorithm outputs several responsible vehicles, all vehicles and vehicle probabilities will be listed; When the spill is confirmed and the responsible vehicle is identified, a full incident report is generated; if the responsible vehicle cannot be determined, a general spill alert is generated; When an incident report is generated, the incident severity is calculated based on the scoring system; If the severity exceeds the preset threshold, the emergency response process is initiated; When all detections and report generation are completed, the system status is reset and the next round of detection is performed; if the system memory usage exceeds the threshold, the garbage collection program is triggered to release resources.

6. An industrial spillage monitoring system based on deep learning and graph neural network, based on the industrial spillage monitoring method based on deep learning and graph neural network according to any one of claims 1 to 5, characterized in that: include, The target first detection module analyzes the engineering vehicles entering a specific area and detects the target for the first time based on the video stream; The recognition module uses a deep learning model to identify and track engineering vehicles based on the first detection results, and uses OCR technology to extract license plate information; A second detection module is used to expand the detection area for identifying and tracking engineering vehicles, and use the spilled object recognition model to perform a second detection of suspected spilled objects in the vehicle area; The third detection module performs a third detection based on the area suspected of being spilled to determine whether there is any spilled object; The association module uses a graph neural network association algorithm to associate the spilled object information with the vehicle information based on the three detection results; The generation module generates the alarm information level and processing priority according to the correlation results.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the industrial spill monitoring method based on deep learning and graph neural network described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the industrial spill monitoring method based on deep learning and graph neural network described in any one of claims 1 to 5 are implemented.

Images