Factory management and control method and system based on artificial intelligence
By introducing automated processing based on artificial intelligence into the factory management and control system, including target detection, tracking and trajectory prediction, the problems of low efficiency and lack of intelligent prediction of traditional methods are solved, and efficient and accurate factory management and safety management are achieved.
Patent Information
- Application Number
- CN202411700876.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-03-14
AI Technical Summary
Traditional factory management and control methods rely on manual patrols and simple monitoring systems, which are inefficient and slow in response, making it difficult to accurately identify and track multiple targets, especially in complex environments, and lack intelligent prediction and early warning mechanisms, making it difficult to effectively monitor whether targets will cross the boundary or enter dangerous areas.
Using an artificial intelligence-based factory management and control method, we obtain surveillance videos, conduct target detection, target tracking and trajectory prediction, build a space-time map, and determine whether the target is crossing the boundary. The method includes video processing, object detection, object tracking, trajectory prediction and out-of-bounds judgment, and uses the trained model to perform automated processing.
Automatic monitoring and control of goals in the factory is achieved, manual intervention is reduced, management efficiency is improved, target behavior patterns can be accurately captured in complex environments, accurate prediction and trajectory planning are carried out, and the safety and management response speed of the factory are improved.
Smart Images
Figure CN119623980A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of plant area management and control, and in particular to a plant area management and control method and system based on artificial intelligence. Background Art
[0002] Artificial intelligence (AI) refers to the simulation and enhancement of human intelligence, which enables computers or machines to perform tasks that usually require human intelligence. AI technologies include machine learning, deep learning, natural language processing, computer vision and other fields. A factory area refers to the area where an enterprise or factory is located, which is usually a physical space dedicated to production, processing, assembly, research and development and other activities. The factory area usually includes infrastructure such as production lines, warehousing, equipment, offices, and employee areas.
[0003] Traditional methods rely on manual inspections and manual surveillance video analysis. Managers need to monitor surveillance images for a long time to identify targets and make behavioral judgments. This not only increases the workload of managers, but also easily leads to omissions or misjudgments due to fatigue or distraction. Therefore, the efficiency is low and the response speed is slow. In traditional methods, target detection and tracking usually rely on manual or simple monitoring systems. This method is difficult to accurately identify and track multiple targets, especially in complex environments. The behavior of targets may be complicated and interactive. Traditional methods often cannot effectively capture the subtle changes of these targets, resulting in poor monitoring effects. In addition, due to the lack of intelligent prediction and early warning mechanisms, traditional methods are difficult to effectively monitor whether targets will cross boundaries or enter dangerous areas. Especially when there are many interactions between targets and the factory environment is relatively complex, it is easy to ignore potential safety risks, resulting in an increased probability of accidents and affecting the safety management level of the factory. Summary of the invention
[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a plant area management and control method and system based on artificial intelligence.
[0005] The technical solution adopted to solve the above technical problems is: a factory area control method based on artificial intelligence, including:
[0006] Acquire a surveillance video of a target surveillance area, acquire a video frame sequence of the surveillance video, and preprocess the video frame sequence to obtain a standard video frame sequence;
[0007] Performing target detection on the standard video frame sequence based on the trained target detection model to obtain a target detection result for each standard video frame in the standard video frame sequence;
[0008] Performing target tracking on the target detection result of each standard video frame in the standard video frame sequence to obtain a target motion trajectory of each target in the standard video frame sequence;
[0009] Constructing a spatiotemporal graph based on the target motion trajectory of each target in the standard video frame sequence, and performing trajectory prediction on the spatiotemporal graph based on a trained trajectory prediction model to obtain a target predicted motion trajectory of each target in the standard video frame sequence;
[0010] The local spatial coordinates of multiple preset target control areas in the target monitoring area are obtained, the local spatial coordinates of the target predicted motion trajectory are calculated, and based on the local spatial coordinates of the target predicted motion trajectory and the local spatial coordinates of the target control area, it is determined whether each target in the standard video frame sequence has crossed the boundary.
[0011] Preferably, preprocessing the video frame sequence to obtain a standard video frame sequence includes:
[0012] Calculate the total pixel value of a single channel in the video frame, wherein the calculation formula of the total pixel value of the single channel is as follows:
[0013]
[0014] Among them, S c It represents the total pixel value of a single channel in the video frame, M and N represent the spatial resolution of the video frame, c represents the three channels of R, G, and B, and I c (i,j) represents the value of the c-channel pixel (i,j);
[0015] The total pixel value of a single channel in the video frame is normalized to obtain a normalized average value of the total pixel value of the single channel in the video frame, wherein the calculation formula of the normalized average value of the total pixel value of the single channel is as follows:
[0016]
[0017] Among them, AVG c Represents the normalized average value of the total pixel value of a single channel;
[0018] The pixel average value of all channels in the video frame is calculated based on the normalized average value of the total pixel value of a single channel in the video frame, and the equalization coefficients of the G and B channels are calculated based on the pixel average values of all channels in the video frame, wherein the calculation formula of the equalization coefficients of the G and B channels is as follows:
[0019]
[0020] Among them, θ cIt represents the equalization coefficient of G and B channels, and AVG represents the pixel average value of all channels in the video frame.
[0021] Preferably, preprocessing the video frame sequence to obtain a standard video frame sequence further includes:
[0022] Based on the equalization coefficients of the G and B channels, the G and B channels in the video frame are equalized to obtain an equalized video frame, wherein the expression of the equalization processing is as follows:
[0023]
[0024] in, Represents the value of the pixel (i, j) in the c channel of the equalized video frame;
[0025] The attenuation coefficient is constructed based on the difference of the R, G, and B channels in the equalized video frame. The calculation formula of the attenuation coefficient is as follows:
[0026]
[0027] Among them, AM represents the attenuation coefficient, AM GR represents the difference between the G and B channels in the equalized video frame, and AM BR represents the difference between the B and R channels in the equalized video frame, and
[0028] Color correction is performed on the R channel in the equalized video frame based on the attenuation matrix to obtain a standard video frame, wherein the expression of the color correction is as follows:
[0029]
[0030] in, Represents the value of the R channel pixel (i, j) of the standard video frame, Indicates the maximum value of the color channel in the equalized video frame. Indicates the minimum value of the color channels in the equalized video frame.
[0031] Preferably, the target detection model adopts an improved YOLOv7-tiny model, and introduces a visual channel attention mechanism into the backbone feature extraction network of the YOLOv7-tiny model, wherein the visual channel attention mechanism is used to perform a global average pooling operation on the input feature map, and the size of the feature map is C*H*W, wherein C represents the number of channels of the feature map, H represents the height of the feature map, and W represents the width of the feature map. The expression of the visual channel attention mechanism is as follows:
[0032]
[0033] Among them, z c represents the output of the visual channel attention mechanism, u c (i,j) represents the output of feature map pixel (i,j).
[0034] Preferably, performing target tracking on the target detection result of each standard video frame in the standard video frame sequence to obtain a target motion trajectory of each target in the standard video frame sequence includes:
[0035] Initialize a Kalman filter tracker for each target according to the target detection result of the first frame in the standard video frame sequence, assign an ID to each Kalman filter tracker, and track the motion trajectory of the target under the ID;
[0036] According to the target detection result of the second frame in the standard video frame sequence, predict the target position information from the existing Kalman filter tracker, and calculate the IoU measurement matrix between the target box and the predicted target box;
[0037] The Hungarian algorithm is used to optimally associate the target with the Kalman filter tracker. For targets and Kalman filter trackers that match each other, the parameters of the matching Kalman filter are updated according to the target detection results.
[0038] The target frame is used as the tracking result. For the target that is detected but not matched to the Kalman filter tracker, the target is regarded as unmatched detection, and the Kalman filter tracker is initialized for the target. During the tracking process, the number of matches between the target and the Kalman filter tracker is counted. If the number reaches the set threshold, the Kalman filter tracker is successfully created, otherwise the creation fails.
[0039] For the unmatched Kalman filter tracker, it indicates that the target detection model has missed detection, and the number of consecutive unmatched times of the Kalman filter tracker is counted during the tracking process;
[0040] If the number of consecutive mismatches exceeds a set threshold, the target corresponding to the Kalman filter tracker is determined to be lost, and the corresponding Kalman filter tracker is deleted. When the set threshold is not reached, the Kalman filter tracker continues to track the target through Kalman filtering.
[0041] Preferably, the spatiotemporal graph includes multiple nodes and connecting edges, wherein the nodes correspond to targets in the standard video frame sequence, the connecting edges correspond to the interaction relationship between targets, the target detection results include target category, target frame coordinates, target frame size and confidence score, and the target motion trajectory includes the tracking number of the tracked target, the moving trajectory of the bottom center point of the target frame, the target frame size and confidence score information.
[0042] Preferably, performing trajectory prediction on the spatiotemporal graph based on a trained trajectory prediction model to obtain a target predicted motion trajectory of each target in the standard video frame sequence includes:
[0043] Calculate the interaction matrix of each target in the standard video frame sequence, where the expression of the interaction matrix is as follows:
[0044]
[0045] in, represents the interaction coefficient between the i-th and j-th targets in the standard video frame sequence at time t, and represents the motion speed of the i-th and j-th targets in the standard video frame sequence at time t;
[0046] Based on the trained trajectory prediction model, trajectory prediction is performed on the interaction matrix of each target in the standard video frame sequence and the space-time graph to obtain the target predicted motion trajectory of each target in the standard video frame sequence.
[0047] Preferably, the trajectory prediction model adopts a graph neural network, and the trajectory prediction model includes a two-dimensional space-time graph convolution module and a time extrapolation convolution module. The two-dimensional space-time graph convolution module is used to stack the interaction matrix and the space-time graph of each target in the standard video frame sequence to obtain the space-time feature matrix of the target motion trajectory of each target in the standard video frame sequence, and the time extrapolation convolution module is used to perform a one-dimensional convolution operation on the time series information of the space-time feature matrix of the target motion trajectory of each target in the standard video frame sequence, and record and utilize the time information of the trajectory through causal convolution to obtain the target predicted motion trajectory of each target in the standard video frame sequence.
[0048] Preferably, calculating the local spatial coordinates of the predicted motion trajectory of the target includes:
[0049] Define a transformation formula between the camera coordinate system and the local space coordinate system, where the transformation formula is as follows:
[0050]
[0051] Among them, S represents the camera oblique depth, (i, v) represents the coordinates in the camera coordinate system, (x, y, z) represents the coordinates in the local space coordinate system, R represents the rotation matrix, T represents the translation matrix, and f x and f y represents the horizontal and vertical focal length of the camera, c x and c yrepresents the principal point coordinates of the camera's image, i.e. the position of the image center, and f represents the focal length of the camera, h represents the height of the camera, Indicates the attitude angle of the camera;
[0052] The local space coordinates of the target predicted motion trajectory are obtained based on the predefined coordinate origin of the local space coordinate system and the transformation formula.
[0053] The technical solution adopted to solve the above technical problems is: a plant area control system based on artificial intelligence, which is applicable to the plant area control method based on artificial intelligence, including:
[0054] A video processing unit, the video processing unit is used to obtain a surveillance video of a target surveillance area, obtain a video frame sequence of the surveillance video, and pre-process the video frame sequence to obtain a standard video frame sequence;
[0055] An object detection unit, the object detection unit is used to perform object detection on the standard video frame sequence based on a trained object detection model to obtain an object detection result for each standard video frame in the standard video frame sequence;
[0056] A target tracking unit, the target tracking unit is used to perform target tracking on the target detection result of each standard video frame in the standard video frame sequence to obtain a target motion trajectory of each target in the standard video frame sequence;
[0057] A trajectory prediction unit, the trajectory prediction unit is used to construct a spatiotemporal graph based on the target motion trajectory of each target in the standard video frame sequence, and perform trajectory prediction on the spatiotemporal graph based on a trained trajectory prediction model to obtain a target predicted motion trajectory of each target in the standard video frame sequence;
[0058] An out-of-bounds judgment unit is used to obtain the local spatial coordinates of multiple preset target control areas in the target monitoring area, calculate the local spatial coordinates of the target predicted motion trajectory, and judge whether each target in the standard video frame sequence has crossed the boundary based on the local spatial coordinates of the target predicted motion trajectory and the local spatial coordinates of the target control area.
[0059] The beneficial effects of the present invention are as follows: (1) The present invention automatically processes surveillance videos, including target detection, target tracking, and trajectory prediction. The method can realize automatic monitoring and control of various targets (such as personnel, equipment, vehicles, etc.) within the factory area, reduce manual intervention, and can automatically identify and track the movement trajectory of the target. At the same time, preventive control is carried out according to the future movement prediction of the target, thereby improving the efficiency of factory management. By constructing and predicting the movement trajectory of the target based on the space-time graph, the relationship between the spatial position and time change of the target can be considered, making the prediction model more accurate. This method can better capture the behavior pattern of the target in a complex factory environment, especially when multiple targets interact, so as to perform more accurate cross-border prediction and trajectory planning; (2) The present invention can automatically identify and track the movement trajectory of the target, and can automatically identify and track the movement trajectory of the target. By combining the target prediction trajectory with the preset target control area, the method can predict in advance whether the target is likely to cross the boundary or enter the dangerous area. By calculating the relationship between the local spatial coordinates of the target and the control area, the target behavior can be monitored in real time, and an alarm can be issued in advance or intervention measures can be taken to avoid safety hazards and improve the safety of the factory. (3) In the daily operation of the factory, whether the target (such as personnel, equipment, etc.) crosses the boundary or enters the dangerous area is directly related to the safety of the factory. Through real-time monitoring and intelligent early warning, the response speed and accuracy of the factory management can be greatly improved, the probability of accidents can be reduced, and the overall safety can be improved. At the same time, the method can reduce the burden of manual inspections, allowing managers to have more time to handle other tasks and improve production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 A schematic flow chart of the steps of an overall method in an embodiment of the present invention;
[0061] Figure 2 A schematic diagram of the system architecture of an overall system in an embodiment of the present invention.
[0062] Figure numerals: 1. Video processing unit; 2. Target detection unit; 3. Target tracking unit; 4. Trajectory prediction unit; 5. Out-of-bounds judgment unit. DETAILED DESCRIPTION
[0063] Embodiment 1, as Figure 1 As shown, the present invention proposes a plant area management and control method based on artificial intelligence, comprising:
[0064] S1. Obtain a surveillance video of a target surveillance area, obtain a video frame sequence of the surveillance video, and preprocess the video frame sequence to obtain a standard video frame sequence;
[0065] S2. performing target detection on the standard video frame sequence based on the trained target detection model to obtain a target detection result for each standard video frame in the standard video frame sequence;
[0066] S3, performing target tracking on the target detection result of each standard video frame in the standard video frame sequence to obtain a target motion trajectory of each target in the standard video frame sequence;
[0067] S4, constructing a spatiotemporal graph based on the target motion trajectory of each target in the standard video frame sequence, and performing trajectory prediction on the spatiotemporal graph based on the trained trajectory prediction model to obtain the target predicted motion trajectory of each target in the standard video frame sequence;
[0068] S5. Obtain the local spatial coordinates of multiple preset target control areas in the target monitoring area, calculate the local spatial coordinates of the target predicted motion trajectory, and determine whether each target in the standard video frame sequence has crossed the boundary based on the local spatial coordinates of the target predicted motion trajectory and the local spatial coordinates of the target control area.
[0069] In the present invention, the target monitoring area refers to an area in a factory or scene that is monitored and tracked using monitoring equipment (such as a camera). This is usually a specific area divided according to security or management requirements, and may include the entrance of the factory, equipment areas, personnel flow areas, etc.; a video frame sequence refers to a continuous image frame extracted from a monitoring video. A video is composed of a series of frames (pictures), which are arranged in chronological order to form a video stream, and each frame can be regarded as a static image; target detection is a technology in computer vision that aims to identify and locate target objects from images or video frames. Usually, target detection marks the target position in the image and assigns a category label to it, for example, detecting targets such as people and vehicles in a factory video; target tracking refers to continuously tracking and predicting the position of a target in a video stream. Unlike target detection, target tracking maintains the identity of the target through continuous frame analysis, thereby finding the target's motion trajectory in the video; a spatiotemporal graph is a graph structure that combines target position and time information. The graph can be used to capture the dynamic relationship of the target at different times and spaces for more accurate trajectory analysis and prediction; the trajectory prediction model predicts the future trajectory of the target based on the historical trajectory of the target; local spatial coordinates refer to the coordinate system of a specific position in the monitoring area or factory area. It is the coordinate relative to a reference point or area (such as a surveillance camera or a target control area). These coordinates are usually used to represent the relative position of the target in space; the target control area is a specific area designated within the factory area for the management and monitoring of targets. For example, in certain high-risk areas, restricted areas or areas around important equipment, the system monitors the target's trajectory in real time and determines whether the target enters or crosses these areas to ensure safety; out-of-bounds judgment refers to judging whether the target has entered a preset control area boundary through the target's trajectory. For example, if a target enters a dangerous area or a restricted area, the system needs to issue an alarm in time. The out-of-bounds judgment is usually based on the comparison between the predicted trajectory of the target and the spatial coordinates of the control area.
[0070] Embodiment 2, a plant area control method based on artificial intelligence proposed by the present invention, compared with embodiment 1, this embodiment further includes: preprocessing the video frame sequence to obtain a standard video frame sequence, including:
[0071] A1. Calculate the total pixel value of a single channel in the video frame, where the calculation formula for the total pixel value of a single channel is as follows:
[0072]
[0073] Among them, S c It represents the total pixel value of a single channel in the video frame, M and N represent the spatial resolution of the video frame, c represents the three channels of R, G, and B, and I c(i,j) represents the value of the c-channel pixel (i,j);
[0074] A2. Normalize the total pixel value of a single channel in the video frame to obtain a normalized average value of the total pixel value of the single channel in the video frame, wherein the calculation formula of the normalized average value of the total pixel value of the single channel is as follows:
[0075]
[0076] Among them, AVG c Represents the normalized average value of the total pixel value of a single channel;
[0077] A3. Calculate the pixel average of all channels in the video frame based on the normalized average value of the total pixel value of a single channel in the video frame, and calculate the equalization coefficients of the G and B channels based on the pixel average values of all channels in the video frame, wherein the calculation formula of the equalization coefficients of the G and B channels is as follows:
[0078]
[0079] Among them, θ c It represents the equalization coefficient of G and B channels, and AVG represents the pixel average value of all channels in the video frame.
[0080] In this embodiment, the total pixel value refers to the sum of the values of all pixels in the video frame. Each pixel is composed of values of multiple channels, usually three color channels of RGB (red, green, and blue). When calculating the total pixel value of a single channel, only the sum of the pixel values of a certain color channel is considered; the spatial resolution refers to the size or resolution of the video frame, usually expressed as "width×height". For example, a frame of video may have a resolution of 1920×1080, which means that each frame has a width of 1920 pixels and a height of 1080 pixels; the equalization coefficient is usually a coefficient in image processing, which is used to balance or adjust the influence of different channels.
[0081] In an optional embodiment, preprocessing the video frame sequence to obtain a standard video frame sequence further includes:
[0082] A4. Perform equalization processing on the G and B channels in the video frame based on the equalization coefficients of the G and B channels to obtain an equalized video frame, wherein the expression of the equalization processing is as follows:
[0083]
[0084] in, Represents the value of the pixel (i, j) in the c channel of the equalized video frame;
[0085] A5. Construct an attenuation coefficient based on the difference between the R, G, and B channels in the equalized video frame. The calculation formula of the attenuation coefficient is as follows:
[0086]
[0087] Among them, AM represents the attenuation coefficient, AM GR represents the difference between the G and B channels in the equalized video frame, and represents the difference between the B and R channels in the equalized video frame, and
[0088] A6. Perform color correction on the R channel in the normalized video frame based on the attenuation matrix to obtain a standard video frame, wherein the expression of color correction is as follows:
[0089]
[0090] in, Represents the value of the R channel pixel (i, j) of the standard video frame, Indicates the maximum value of the color channel in the equalized video frame. Indicates the minimum value of the color channels in the equalized video frame.
[0091] It should be noted that the attenuation coefficient refers to the factor used to adjust the difference in color channels in an image. The calculation of the attenuation coefficient is usually based on the difference between channels, that is, the difference in each channel (such as red, green or blue) after equalization; color correction is an image processing technology that makes the color of the image look more natural and consistent by adjusting the brightness and contrast of the colors in the image.
[0092] In an optional embodiment, the target detection model adopts an improved YOLOv7-tiny model, and introduces a visual channel attention mechanism in the backbone feature extraction network of the YOLOv7-tiny model, wherein the visual channel attention mechanism is used to perform a global average pooling operation on the input feature map, and the size of the feature map is C*H*W, wherein C represents the number of channels of the feature map, H represents the height of the feature map, and W represents the width of the feature map. The expression of the visual channel attention mechanism is as follows:
[0093]
[0094] Among them, z c represents the output of the visual channel attention mechanism, u c (i,j) represents the output of feature map pixel (i,j).
[0095] It should be noted that YOLOv7-tiny is a lightweight version of the YOLO (You Only Look Once) series of target detection models. Compared with the standard YOLOv7, YOLOv7-tiny has been optimized in terms of computing resource requirements and inference speed, making it more suitable for mobile devices or resource-constrained environments. The model reduces the amount of calculation by reducing the depth and width of the network while trying to maintain high detection accuracy; the main purpose of the visual channel attention mechanism is to enhance the attention of the convolutional neural network to important feature channels. The basic idea is that the model dynamically adjusts the feature maps of each channel by calculating the importance of each channel, thereby strengthening the response of key feature channels and suppressing unimportant feature channels; global average pooling (GAP) is a pooling operation that averages the feature maps of each channel by their spatial dimensions (i.e., height and width) to obtain a scalar value. After the feature map is globally averaged pooled, each channel uses a scalar to represent its importance.
[0096] In an optional embodiment, target tracking is performed on the target detection result of each standard video frame in the standard video frame sequence to obtain a target motion trajectory of each target in the standard video frame sequence, including:
[0097] B1. Initialize a Kalman filter tracker for each target based on the target detection result of the first frame in the standard video frame sequence, assign each Kalman filter tracker an ID, and track the target based on the motion trajectory of the target under the ID;
[0098] B2. According to the target detection result of the second frame in the standard video frame sequence, the target position information is predicted from the existing Kalman filter tracker, and the IoU metric matrix between the target box and the predicted target box is calculated;
[0099] B3. Use the Hungarian algorithm to optimally associate the target with the Kalman filter tracker. For targets and Kalman filter trackers that match each other, update the parameters of the matching Kalman filter according to the target detection result.
[0100] B4. Use the target frame as the tracking result. For the target that is detected but not matched to the Kalman filter tracker, the target is regarded as unmatched detection, and the Kalman filter tracker is initialized for the target. During the tracking process, the number of matches between the target and the Kalman filter tracker is counted. If the number reaches the set threshold, the Kalman filter tracker is successfully created, otherwise the creation fails.
[0101] B5. For unmatched Kalman filter trackers, it indicates that the target detection model has missed detection. During the tracking process, the number of consecutive unmatched Kalman filter trackers is counted;
[0102] B6. If the number of consecutive mismatches exceeds the set threshold, the target corresponding to the Kalman filter tracker is determined to be lost, and the corresponding Kalman filter tracker is deleted. When the set threshold is not reached, the Kalman filter tracker continues to track the target through Kalman filtering.
[0103] It should be noted that the Kalman filter is a recursive algorithm for state estimation of linear dynamic systems. It estimates the state of the target through two steps: prediction and update. In target tracking, Kalman filters are used to predict the position and velocity of the target and update the state by comparing it with the actual detected position. The main advantage of Kalman filters is that they can handle uncertainty and noise, so they are suitable for estimation tasks in dynamic systems. The target motion trajectory refers to the path of the target position changing over time in the video frame sequence. In the target tracking process, the position of the target in each frame is obtained by predicting and updating the Kalman filter, thereby constructing the target's motion trajectory. IoU is an indicator used to measure the degree of overlap between two target frames. The Hungarian algorithm is an algorithm used to solve the optimal matching problem and is often used in allocation problems. For example, in target tracking, it is used to find the most suitable Kalman filter for matching each target detection result. In the task of combining target detection and tracking, the Hungarian algorithm calculates the IoU value of the target and the predicted frame to generate an IoU matrix, and then finds the optimal matching pair based on the values in the matrix. In target tracking, unmatched detection refers to the target appearing in the video frame that does not match the predicted target frame of any existing Kalman filter. This may occur when the target's appearance changes, is partially occluded, or is missed. For unmatched targets, a new Kalman filter is usually created for tracking.
[0104] In an optional embodiment, the spatiotemporal graph includes multiple nodes and connecting edges, wherein the nodes correspond to targets in a standard video frame sequence, the connecting edges correspond to the interaction relationship between the targets, the target detection results include target category, target frame coordinates, target frame size and confidence score, and the target motion trajectory includes the tracking number of the tracked target, the moving trajectory of the bottom center point of the target frame, the target frame size and confidence score information.
[0105] In an optional embodiment, trajectory prediction is performed on the spatiotemporal graph based on a trained trajectory prediction model to obtain a target predicted motion trajectory of each target in a standard video frame sequence, including:
[0106] C1. Calculate the interaction matrix of each target in the standard video frame sequence, where the expression of the interaction matrix is as follows:
[0107]
[0108] in, represents the interaction coefficient between the i-th and j-th targets in the standard video frame sequence at time t, and represents the motion speed of the i-th and j-th targets in the standard video frame sequence at time t;
[0109] C2. Based on the trained trajectory prediction model, the interaction matrix and spatiotemporal graph of each target in the standard video frame sequence are predicted to obtain the target predicted motion trajectory of each target in the standard video frame sequence.
[0110] It should be noted that the interaction matrix is a way of describing the interaction between targets. Each matrix element represents the degree of interaction or influence between one target and another target. For example, in a scene with multiple people or multiple objects moving, the movement of one target may be affected by other targets, and the interaction matrix can be used to quantify this influence.
[0111] In an optional embodiment, the trajectory prediction model adopts a graph neural network, and the trajectory prediction model includes a two-dimensional space-time graph convolution module and a time extrapolation convolution module. The two-dimensional space-time graph convolution module is used to stack the interaction matrix and space-time graph of each target in the standard video frame sequence to obtain the space-time feature matrix of the target motion trajectory of each target in the standard video frame sequence. The time extrapolation convolution module is used to perform a one-dimensional convolution operation on the temporal information of the space-time feature matrix of the target motion trajectory of each target in the standard video frame sequence, and record and utilize the time information of the trajectory through causal convolution to obtain the predicted target motion trajectory of each target in the standard video frame sequence.
[0112] It should be noted that graph neural network is a deep learning model used to process graph structured data. In trajectory prediction, the interactions between targets can be regarded as nodes and edges in the graph. Nodes represent targets and edges represent the relationships between targets. GNN updates the features of each node by aggregating the information of neighboring nodes, thereby realizing the processing of complex structured data. In this scenario, graph neural network is used to process the interactive relationships and spatiotemporal information between targets to perform trajectory prediction.
[0113] In an optional embodiment, calculating the local space coordinates of the predicted motion trajectory of the target includes:
[0114] D1. Define the transformation formula between the camera coordinate system and the local space coordinate system, where the transformation formula is as follows:
[0115]
[0116] Among them, S represents the camera oblique depth, (i, v) represents the coordinates in the camera coordinate system, (x, y, z) represents the coordinates in the local space coordinate system, R represents the rotation matrix, T represents the translation matrix, and f x and f y represents the horizontal and vertical focal length of the camera, c x and c y represents the principal point coordinates of the camera's image, i.e. the position of the image center, and f represents the focal length of the camera, h represents the height of the camera, Indicates the attitude angle of the camera;
[0117] D2. Obtain the local space coordinates of the target predicted motion trajectory based on the coordinate origin and transformation formula of the pre-defined local space coordinate system.
[0118] Embodiment three, as Figure 2 As shown, the present invention proposes an artificial intelligence-based plant area control system, which is applicable to the artificial intelligence-based plant area control method, including:
[0119] The video processing unit 1 is used to obtain the monitoring video of the target monitoring area, obtain the video frame sequence of the monitoring video, and pre-process the video frame sequence to obtain a standard video frame sequence;
[0120] The target detection unit 2 is used to perform target detection on the standard video frame sequence based on the trained target detection model to obtain the target detection result of each standard video frame in the standard video frame sequence;
[0121] The target tracking unit 3 is used to track the target detection result of each standard video frame in the standard video frame sequence to obtain the target motion trajectory of each target in the standard video frame sequence;
[0122] The trajectory prediction unit 4 is used to construct a spatiotemporal graph based on the target motion trajectory of each target in the standard video frame sequence, and perform trajectory prediction on the spatiotemporal graph based on the trained trajectory prediction model to obtain the target predicted motion trajectory of each target in the standard video frame sequence;
[0123] The out-of-bounds judgment unit 5 is used to obtain the local spatial coordinates of multiple preset target control areas in the target monitoring area, calculate the local spatial coordinates of the target predicted motion trajectory, and judge whether each target in the standard video frame sequence has crossed the boundary based on the local spatial coordinates of the target predicted motion trajectory and the local spatial coordinates of the target control area.
[0124] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.
Claims
1. A plant area management and control method based on artificial intelligence, characterized in that: include: Acquire a surveillance video of a target surveillance area, acquire a video frame sequence of the surveillance video, and preprocess the video frame sequence to obtain a standard video frame sequence; Performing target detection on the standard video frame sequence based on the trained target detection model to obtain a target detection result for each standard video frame in the standard video frame sequence; Performing target tracking on the target detection result of each standard video frame in the standard video frame sequence to obtain a target motion trajectory of each target in the standard video frame sequence; Constructing a spatiotemporal graph based on the target motion trajectory of each target in the standard video frame sequence, and performing trajectory prediction on the spatiotemporal graph based on a trained trajectory prediction model to obtain a target predicted motion trajectory of each target in the standard video frame sequence; The local spatial coordinates of multiple preset target control areas in the target monitoring area are obtained, the local spatial coordinates of the target predicted motion trajectory are calculated, and based on the local spatial coordinates of the target predicted motion trajectory and the local spatial coordinates of the target control area, it is determined whether each target in the standard video frame sequence has crossed the boundary.
2. The method for plant area management and control based on artificial intelligence according to claim 1, characterized in that: Preprocessing the video frame sequence to obtain a standard video frame sequence includes: Calculate the total pixel value of a single channel in the video frame, wherein the calculation formula of the total pixel value of the single channel is as follows: Among them, S c It represents the total pixel value of a single channel in the video frame, M and N represent the spatial resolution of the video frame, c represents the three channels of R, G, and B, and I c (i,j) represents the value of the c-channel pixel (i,j); The total pixel value of a single channel in the video frame is normalized to obtain a normalized average value of the total pixel value of the single channel in the video frame, wherein the calculation formula of the normalized average value of the total pixel value of the single channel is as follows: Among them, AVG c Represents the normalized average value of the total pixel value of a single channel; The pixel average value of all channels in the video frame is calculated based on the normalized average value of the total pixel value of a single channel in the video frame, and the equalization coefficients of the G and B channels are calculated based on the pixel average values of all channels in the video frame, wherein the calculation formula of the equalization coefficients of the G and B channels is as follows: Among them, θ c It represents the equalization coefficient of G and B channels, and AVG represents the pixel average value of all channels in the video frame.
3. The method for plant area management and control based on artificial intelligence according to claim 2 is characterized in that: Preprocessing the video frame sequence to obtain a standard video frame sequence also includes: Based on the equalization coefficients of the G and B channels, the G and B channels in the video frame are equalized to obtain an equalized video frame, wherein the expression of the equalization processing is as follows: in, Represents the value of the pixel (i, j) in the c channel of the equalized video frame; The attenuation coefficient is constructed based on the difference of the R, G, and B channels in the equalized video frame. The calculation formula of the attenuation coefficient is as follows: Among them, AM represents the attenuation coefficient, AM GR represents the difference between the G and B channels in the equalized video frame, and AM BR represents the difference between the B and R channels in the equalized video frame, and Color correction is performed on the R channel in the leveled video frame based on the attenuation matrix to obtain a standard video frame, wherein the expression of the color correction is as follows: in, Represents the value of the R channel pixel (i, j) of the standard video frame, Indicates the maximum value of the color channel in the equalized video frame. Indicates the minimum value of the color channels in the equalized video frame.
4. The method for plant area management and control based on artificial intelligence according to claim 1, characterized in that: The target detection model adopts an improved YOLOv7-tiny model, and introduces a visual channel attention mechanism into the backbone feature extraction network of the YOLOv7-tiny model, wherein the visual channel attention mechanism is used to perform a global average pooling operation on the input feature map, and the size of the feature map is C*H*W, wherein C represents the number of channels of the feature map, H represents the height of the feature map, and W represents the width of the feature map. The expression of the visual channel attention mechanism is as follows: Among them, z c represents the output of the visual channel attention mechanism, u c (i,j) represents the output of feature map pixel (i,j).
5. The method for plant area management and control based on artificial intelligence according to claim 1, characterized in that: Performing target tracking on the target detection result of each standard video frame in the standard video frame sequence to obtain a target motion trajectory of each target in the standard video frame sequence includes: Initialize a Kalman filter tracker for each target according to the target detection result of the first frame in the standard video frame sequence, assign an ID to each Kalman filter tracker, and track the motion trajectory of the target under the ID; According to the target detection result of the second frame in the standard video frame sequence, predict the target position information from the existing Kalman filter tracker, and calculate the IoU measurement matrix between the target box and the predicted target box; The Hungarian algorithm is used to optimally associate the target with the Kalman filter tracker. For targets and Kalman filter trackers that match each other, the parameters of the matching Kalman filter are updated according to the target detection results. The target frame is used as the tracking result. For the target that is detected but not matched to the Kalman filter tracker, the target is regarded as unmatched detection, and the Kalman filter tracker is initialized for the target. During the tracking process, the number of matches between the target and the Kalman filter tracker is counted. If the number reaches the set threshold, the Kalman filter tracker is successfully created, otherwise the creation fails. For the unmatched Kalman filter tracker, it indicates that the target detection model has missed detection, and the number of consecutive unmatched times of the Kalman filter tracker is counted during the tracking process; If the number of consecutive mismatches exceeds a set threshold, the target corresponding to the Kalman filter tracker is determined to be lost, and the corresponding Kalman filter tracker is deleted. When the set threshold is not reached, the Kalman filter tracker continues to track the target through Kalman filtering.
6. The method for plant area management and control based on artificial intelligence according to claim 1, characterized in that: The spatiotemporal graph includes multiple nodes and connecting edges, wherein the nodes correspond to the targets in the standard video frame sequence, the connecting edges correspond to the interaction relationship between the targets, the target detection results include target category, target frame coordinates, target frame size and confidence score, and the target motion trajectory includes the tracking number of the tracked target, the moving trajectory of the center point at the bottom of the target frame, the target frame size and confidence score information.
7. The method for plant area management and control based on artificial intelligence according to claim 6, characterized in that: Performing trajectory prediction on the spatiotemporal graph based on the trained trajectory prediction model to obtain a target predicted motion trajectory of each target in the standard video frame sequence, including: Calculate the interaction matrix of each target in the standard video frame sequence, where the expression of the interaction matrix is as follows: in, represents the interaction coefficient between the i-th and j-th targets in the standard video frame sequence at time t, and represents the motion speed of the i-th and j-th targets in the standard video frame sequence at time t; Based on the trained trajectory prediction model, trajectory prediction is performed on the interaction matrix of each target in the standard video frame sequence and the space-time graph to obtain the target predicted motion trajectory of each target in the standard video frame sequence.
8. The method for plant area management and control based on artificial intelligence according to claim 7 is characterized in that: The trajectory prediction model adopts a graph neural network, and the trajectory prediction model includes a two-dimensional space-time graph convolution module and a time extrapolation convolution module. The two-dimensional space-time graph convolution module is used to stack the interaction matrix and the space-time graph of each target in the standard video frame sequence to obtain the space-time feature matrix of the target motion trajectory of each target in the standard video frame sequence. The time extrapolation convolution module is used to perform a one-dimensional convolution operation on the time series information of the space-time feature matrix of the target motion trajectory of each target in the standard video frame sequence, and record and utilize the time information of the trajectory through causal convolution to obtain the target predicted motion trajectory of each target in the standard video frame sequence.
9. The method for plant area management and control based on artificial intelligence according to claim 1, characterized in that: Calculating the local spatial coordinates of the predicted motion trajectory of the target, including: Define a transformation formula between the camera coordinate system and the local space coordinate system, where the transformation formula is as follows: Among them, S represents the camera oblique depth, (u, v) represents the coordinates in the camera coordinate system, (x, y, z) represents the coordinates in the local space coordinate system, R represents the rotation matrix, T represents the translation matrix, and f x and f y represents the horizontal and vertical focal length of the camera, c x and c y represents the principal point coordinates of the camera's image, i.e. the position of the image center, and represents the focal length of the camera, h represents the height of the camera, Indicates the attitude angle of the camera; The local space coordinates of the target predicted motion trajectory are obtained based on the predefined coordinate origin of the local space coordinate system and the transformation formula.
10. A plant area control system based on artificial intelligence, which is applicable to the plant area control method based on artificial intelligence according to any one of claims 1 to 9, characterized in that: include: A video processing unit (1), the video processing unit (1) is used to obtain a monitoring video of a target monitoring area, obtain a video frame sequence of the monitoring video, and pre-process the video frame sequence to obtain a standard video frame sequence; An object detection unit (2), the object detection unit (2) being used to perform object detection on the standard video frame sequence based on a trained object detection model, so as to obtain an object detection result for each standard video frame in the standard video frame sequence; A target tracking unit (3), the target tracking unit (3) is used to perform target tracking on the target detection result of each standard video frame in the standard video frame sequence, so as to obtain a target motion trajectory of each target in the standard video frame sequence; A trajectory prediction unit (4), the trajectory prediction unit (4) is used to construct a spatiotemporal graph based on the target motion trajectory of each target in the standard video frame sequence, and perform trajectory prediction on the spatiotemporal graph based on a trained trajectory prediction model to obtain a target predicted motion trajectory of each target in the standard video frame sequence; The cross-border judgment unit (5) is used to obtain the local spatial coordinates of multiple preset target control areas in the target monitoring area, calculate the local spatial coordinates of the target predicted motion trajectory, and judge whether each target in the standard video frame sequence has crossed the border based on the local spatial coordinates of the target predicted motion trajectory and the local spatial coordinates of the target control area.