Intelligent field monitoring management method for food processing workshop

Abstract

The invention relates to the technical field of intelligent monitoring, and discloses an intelligent field monitoring management method for a food processing workshop, and the method comprises the following steps: S1, deploying a multi-mode camera and a monocular industrial camera in the workshop; s2, acquiring visible light and thermal infrared video streams of a multi-mode camera and a monocular industrial camera video stream; s3, the preprocessed video stream is transmitted to an improved YOLOv5 model to obtain personnel pixel coordinates; s4, the coordinates are input into an improved DeepSORT model; and S5, extracting joint points by using the improved OpenPose model to judge falling. According to the intelligent on-site monitoring management method for the food processing workshop, video streams of two cameras are obtained firstly, visible light and thermal infrared video streams are subjected to wavelet transform and then adaptive weighted fusion, adaptive median filtering denoising is performed on the fused video stream and a monocular industrial camera video stream, and the video streams of the two cameras are obtained. The image quality in complex environments such as steam and dust is improved, a clear image basis is provided for subsequent personnel behavior recognition, and the problem that a traditional visible light monitoring system is low in personnel behavior recognition rate is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent monitoring, in particular to an intelligent on-site monitoring management method for a food processing workshop. BACKGROUND

[0002] A food processing workshop is a core place for food production, and the on-site management quality directly affects food quality and safety and the personal safety of operating personnel.

[0003] At present, the mainstream monitoring technology in the food processing workshop mainly includes a visible light monitoring system and a positioning system based on a wireless signal, wherein the visible light monitoring system collects workshop on-site images by deploying a camera to realize visual viewing of the on-site situation; the wireless positioning system mainly uses WiFi or RFID technology to try to obtain the position information of personnel in the workshop.

[0004] However, the existing food processing workshop monitoring technology has the following deficiencies: first, the image quality of the traditional visible light monitoring system is seriously degraded in the complex environment such as steam and dust commonly seen in the food processing workshop, resulting in a low personnel behavior recognition rate; second, the traditional monitoring system lacks intelligent recognition and early warning capabilities for key personnel behaviors such as falling, area intrusion, and point inspection violations, and cannot realize real-time supervision of personnel behavior; third, the traditional monitoring system cannot automatically identify the department to which the personnel belong, making it difficult to carry out targeted management; therefore, an intelligent on-site monitoring management method for a food processing workshop is proposed. SUMMARY

[0005] (I) Technical problems solved In view of the deficiencies of the prior art, the present application provides an intelligent on-site monitoring management method for a food processing workshop, which has the advantages of realizing high-precision recognition of personnel behavior in a complex environment through multi-modal camera fusion and adaptive video processing technology, realizing intelligent early warning of falling, area intrusion, and point inspection violations through an improved deep learning model, and realizing automatic determination of the department to which the personnel belong through dust-free clothing color recognition technology, thereby solving the problems of low personnel behavior recognition rate in a complex environment, lack of intelligent recognition and early warning capabilities for key personnel behaviors, and inability to automatically identify the department to which the personnel belong, making it difficult to carry out targeted management.

[0006] (II) Technical solutions To achieve the above-mentioned through multi-modal camera fusion and adaptive video processing technology realizes the high precision recognition of personnel behavior in complex environment;Through the improved deep learning model realizes the intelligent early warning of falling, regional invasion, point inspection violation;Through the color recognition technology of dust-free clothes realizes the automatic determination of the department to which the personnel belong, the present application provides the following technical scheme: An intelligent on-site monitoring and management method for food processing workshop, comprising the following steps: S1: In the raw material processing area, equipment operation area, quality detection area, point inspection work area of the workshop, multi-modal camera and monocular industrial camera are deployed, the multi-modal camera integrates visible light and thermal infrared module, and the monocular industrial camera supports RTSP protocol;The Zhang calibration method is used to calibrate the monocular camera to obtain the intrinsic matrix and distortion coefficient, and the two modules of the multi-modal camera are calibrated respectively; S2: Obtain the visible light video stream, thermal infrared video stream of the multi-modal camera and the video stream of the monocular industrial camera through the RTSP protocol;After wavelet transform, the visible light video stream and the thermal infrared video stream are adaptively weighted and fused, and the fused video stream and the monocular industrial camera video stream are adaptively median filtered; S3: The preprocessed video stream is input into the improved YOLOv5 model to output the personnel pixel coordinates, the improved YOLOv5 is based on the YOLOv5s framework, uses CIOU loss function to calculate the positioning loss, and uses BCEWithLogistsLoss to calculate the classification loss and confidence loss; S4: The personnel pixel coordinates are input into the improved DeepSORT model, which uses an 8-dimensional vector to construct a state estimation model, uses GIOU to construct a cost matrix, combines "Mahalanobis distance + cosine distance" for matching, and extracts personnel ID and motion trajectory through cascading matching; S5: The improved OpenPose model is used to extract the human joint points and determine the falling, and the AlexNet model is used to identify the color of the dust-free clothes to determine the department to which the personnel belong, and then determine the regional invasion and point inspection violation; S6: Based on the intrinsic matrix of the monocular industrial camera, the PnP algorithm is used to solve the external parameter, and the conversion model of pixel coordinates and workshop actual coordinates is established, and the actual coordinates of the personnel are calculated; S7: When falling or violation behavior is detected, store abnormal information, and call enterprise WeChat API to send early warning notification; S8: Based on Flask, a management platform is built, and the real-time video module of the platform uses multi-thread processing.

[0007] Preferably, the specific rules of adaptive weighted fusion in S2 are as follows: The low frequency component is weighted according to the regional variance and energy, and the regional variance calculation formula is: The energy calculation formula is: wherein, is the area size, is the pixel value, is the area mean value; the thermal infrared image area energy is set as , the visible light image area energy is , when > , the thermal infrared image weight is , the visible light image weight is , when > , the opposite is true; The high-frequency component selects the component with the largest absolute value in the two modalities; the adaptive median filtering rule is that when the noise density is ≤20%, a 3*3 window is used, when the noise density is >20%, a 6*6 window is used, and when the window size exceeds 6 and the noise density is still high, the center pixel original value is output.

[0008] Preferably, the structure improvement of the improved YOLOv5 model in S3 is: Replace the Focus module with a 6*6 convolution module, the step size of the 6*6 convolution module is 2, and the padding is 2; replace the BottleneckCSP module with a C3 module, the C3 module contains 3 convolution layers and 4 Bottleneck modules; replace the SPP module with an SPPF module, the SPPF module contains 3 5*5 maximum pooling layers; replace the FPN+PAN module with a BiFPN module, the BiFPN module removes the node with only one input edge, adds channels between the input node and the output node, and repeats the feature fusion block 3 times; The model training data set is 7000 food processing workshop personnel images, which are divided into a training set and a test set in a ratio of 8:2, the training set uses Mosaic data enhancement, the input image size is 640*640, Batchsize=64, the initial learning rate is 0.01, the weight decay is 0.0005, the momentum is 0.937, the training rounds are 300, and the model detection speed reaches 100 FPS after training is completed.

[0009] Preferably, the specific design of the improved DeepSORT model in S4 is: The 8-dimensional state vector is , wherein is the target center horizontal and vertical pixel coordinates, is the target box width-height ratio, is the target box height, is the speed of , respectively. The formula for calculating GIOU is as follows: ,in To enclose the prediction box With real frame The smallest bounding box; The formula for calculating Mahalanobis distance is: ,in For the first Each detection bounding box vector For the first A tracking box vector, The inverse of the covariance matrix between the detection box and the tracking box; The formula for calculating cosine distance is: ,in For the first Normalized feature vectors of each bounding box. For the first In the set of feature vectors that were successfully tracked, the first... k 1 eigenvector; Cascade Matching The trajectory begins, The frame difference between the current frame of the trajectory and the last successfully matched frame. A new trajectory needs to be successfully matched for 3 consecutive frames to become a deterministic state.

[0010] Preferably, the design of the improved OpenPose model in S5 is as follows: The MobileNetV3 network replaces the original VGGNet. MobileNetV3 includes 16 Bneck modules, and the Bneck modules adopt... The activation function, where: The 7×7 convolutional kernel in the dual-branch structure is replaced with one 1×1 convolutional kernel and two 3×3 convolutional kernels. In the feature refinement stage, three 3×3 convolutional layers with 128 channels are used. A temporal frame merging module and a pose residual fusion module are added. The frame fusion formula for the time frame merging module is: For video frame time index; The residual formula for the pose residual fusion module is: For feature connection operations; The criteria for determining a fall are: a descent speed of both knee joints > 0.1 m / s for 3 frames, an aspect ratio of the bounding rectangle of the human body > 0.8, and a human body axis angle < 63°; The formula for the descent velocity of both knee joints is: , is the right knee joint Y coordinate of the current / previous frame, is the corresponding time; The aspect ratio formula is , , , , is the maximum value of the circumscribed rectangle; The axis angle formula is , is the neck-pelvis vector, is the horizontal unit vector.

[0011] Preferably, the specific implementation of monocular vision positioning in S6 is: The camera imaging model formula is: wherein is the scale factor, is the intrinsic matrix, is the rotation matrix, is the translation vector, is the world coordinate; The intrinsic matrix is , and the distortion coefficient is =-0.37068473, =0.20948198, =0.00908307, p1 =-0.00526021, p2 =2.00344022; The extrinsic is solved by PnP algorithm, and the rotation matrix is: The translation vector T=[33.74424264, 88.03384245, 931.9332833]; When calculating the actual coordinates of the personnel, it is set that: Substituting the imaging model, the positioning error is ≤0.1m.

[0012] Preferably, the design and operation process of the AlexNet model in S5 is: The model contains 5 convolutional layers, 3 pooling layers, and 3 fully connected layers, the convolutional layer adopts ReLU activation function, and the fully connected layer adds Dropout and the dropout rate is 0.5; ​The training data set is 500 images of dust-free clothing of A, B, C and D department personnel, the image size is 224*224, the training round is 100, the initial learning rate is 0.001, the optimizer is Adam, and the loss function is , is the real label, is the prediction probability; When running, the personnel detection box image is input into the model, and the color corresponding to the department is output: black corresponds to A department, blue corresponds to B department, white corresponds to C department, and red corresponds to D department.

[0013] Preferably, the implementation process of the linkage early warning in S7 is: The WeChat API calling parameters include (enterprise ID), (application ID), (application secret key); The abnormal information storage adopts SQLServer database, and the storage table field contains (record number), (time of occurrence), (place), (personnel ID), (abnormal type), (file path); The early warning notification includes abnormal type, time of occurrence, abnormal content, device information and file link. The abnormal content describes the department of the personnel involved, the violation behavior and the area, and the device information describes the camera IP and the model.

[0014] Preferably, the multi-thread design of the real-time video module in S8 is: Create a sub-thread to read the RTSP video stream, and control the frame rate to 30FPS; The main thread runs the improved YOLOv5 and DeepSORT models, and receives the personnel coordinate and ID output by the sub-thread video frame; Create a thread pool of 4 threads to process AlexNet color classification and pose residual calculation respectively; The main thread returns the video stream through with a delay of ≤100ms, supporting computer, tablet and mobile phone access.

[0015] Preferably, the optimization for the food processing workshop environment in S2 is: Under the steam environment, the contrast of the visible light image is calculated , is the maximum value of the gray scale, <30 is the weight of the thermal infrared 0.7, and the weight of the visible light is 0.3; Under the dust environment, the gray scale standard deviation is calculated , >50 is high dust, filter initial window 5x5, noise density every 5% increase window +1, maximum 8x8.

[0016] (Three) beneficial effects Compared with the prior art, the intelligent field monitoring management method of the food processing plant has the following beneficial effects: 1. The intelligent field monitoring management method of the food processing plant, by deploying a multi-modal camera integrated with visible light and thermal infrared modules and a monocular industrial camera supporting the RTSP protocol, first acquires the video streams of the two cameras, performs wavelet transform on the visible light and thermal infrared video streams, and then adaptively weighted fusion, and then uses adaptive median filtering to denoise the fused video stream and the monocular industrial camera video stream, improves the image quality in complex environments such as steam and dust, provides a clear image basis for subsequent personnel behavior recognition, and solves the problem of low personnel behavior recognition rate of traditional visible light monitoring systems.

[0017] 2. The intelligent field monitoring management method of the food processing plant, the preprocessed video stream is input into the improved YOLOv5 model to output personnel pixel coordinates, the coordinates are input into the improved DeepSORT model to extract personnel ID and motion trajectory, and the OpenPose model is improved to extract human joint points to determine falling, and the AlexNet model is used to identify the color of the dust-free clothes to determine the department to which the personnel belong to determine regional intrusion and point inspection violations. When an anomaly is detected, the enterprise WeChat API is called to send an early warning, realizing intelligent recognition and real-time supervision of personnel key behaviors, and solving the problem of lack of such capability in traditional systems.

[0018] 3. The intelligent field monitoring management method of the food processing plant, the improved YOLOv5 model is used to obtain the personnel detection frame, and the image in the detection frame is input into the AlexNet model containing 5 convolution layers, 3 pooling layers and 3 fully connected layers. The model determines the department to which the personnel belong by identifying the color of the dust-free clothes, and clearly defines the activity range and authority of personnel in each department, providing a basis for precise management of different departments, and solving the problem that traditional monitoring systems cannot automatically identify the department to which the personnel belong and are difficult to manage specifically. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The flowchart of the intelligent field monitoring management method of the food processing plant of the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the embodiments of the present application and the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the scope of the present application.

[0021] Please refer to Figure 1 An intelligent field monitoring and management method for a food processing plant, comprising the following steps: S1: deploying multi-modal cameras and monocular industrial cameras in the raw material processing area, equipment operation area, quality detection area and point inspection work area of the plant, the multi-modal cameras integrating visible light and thermal infrared modules, and the monocular industrial cameras supporting the RTSP protocol; calibrating the monocular cameras to obtain an intrinsic matrix and distortion coefficients by Zhang's calibration method, and respectively calibrating the two modules of the multi-modal cameras; S2: acquiring visible light video streams, thermal infrared video streams of the multi-modal cameras and video streams of the monocular industrial cameras through the RTSP protocol; adaptively weighting and fusing the visible light video streams and the thermal infrared video streams after wavelet transform; and denoising the fused video streams and the video streams of the monocular industrial cameras by adaptive median filtering; S3: inputting the preprocessed video streams into an improved YOLOv5 model to output personnel pixel coordinates, the improved YOLOv5 taking YOLOv5s as a basic framework, calculating a positioning loss by using a CIOU loss function, and calculating a classification loss and a confidence loss by using a BCEWithLogistsLoss; S4: inputting the personnel pixel coordinates into an improved DeepSORT model, the model constructing a state estimation model by using an 8-dimensional vector, constructing a cost matrix by using a GIOU, matching by combining "Mahalanobis distance + cosine distance", and extracting personnel IDs and motion trajectories by cascading matching; S5: extracting human joint points and determining falls by using an improved OpenPose model, identifying the colors of dust clothes to determine the departments to which the personnel belong by using an AlexNet model, and then determining regional intrusion and point inspection violations; S6: solving external parameters by using a PnP algorithm based on the intrinsic matrix of the monocular industrial cameras, establishing a conversion model of pixel coordinates and actual coordinates of the plant, and calculating actual coordinates of the personnel; S7: storing abnormal information when falls or violations are detected, and simultaneously calling an enterprise WeChat API to send an early warning notice; S8: constructing a management platform based on Flask, and adopting multi-thread processing for a real-time video module of the platform.

[0022] Embodiment one: The embodiment surrounds camera deployment and calibration, video stream acquisition and preprocessing, personnel detection, personnel tracking, behavior recognition and violation determination, personnel positioning, early warning and storage, management platform construction, etc. Combined with video stream adaptive weighted fusion and filtering, improved YOLOv5 model, improved DeepSORT model, improved OpenPose and AlexNet, monocular vision positioning, AlexNet department identification, linkage early warning and storage, Flask multithreading, special environment optimization and other technical requirements, the core methods of workshop personnel detection, tracking, positioning and multi-modal fusion, fall detection are integrated to realize intelligent management of the whole process of personnel in the food processing workshop.

[0023] A multi-modal camera integrating visible light and thermal infrared modules, and a monocular industrial camera supporting the RTSP protocol are selected. The two cameras are deployed in the raw material processing area, the equipment operation area, the quality detection area, and the point inspection work area respectively. The installation coordinates, height, and angle of each area are determined to ensure complete monitoring coverage. The Zhang calibration method is used to calibrate the monocular industrial camera. A 12x9 checkerboard calibration board is made, which is composed of black and white squares with a side length of 30mm. 20 calibration images are taken at different positions such as 1m, 3m, and 5m, and at different angles such as 0°, 15°, and 30°. The OpenCV toolkit is used to extract the inner corner points and optimize the sub-pixel coordinates. The intrinsic matrix and distortion coefficient are calculated, and the calibration error is controlled within 0.5 pixels. The visible light module of the multi-modal camera is calibrated according to the same steps, and the thermal infrared module is calibrated using a 50mm square grid to collect 15 calibration images.

[0024] The RTSP protocol is used to obtain the video stream of each camera. The visible light, thermal infrared, and monocular industrial cameras of the multi-modal camera correspond to different RTSP addresses. The VideoCapture function of OpenCV is used to read the video stream and set the frame rate to 30FPS. At the same time, the format conversion from BGR to RGB, and from single-channel thermal infrared image to 3-channel grayscale image is completed. The db4 wavelet basis is used to decompose the visible light and thermal infrared images to the third level, obtaining the low-frequency component and the high-frequency component. The low-frequency component is used to calculate the variance and energy of the 16x16 pixel region to determine the weight of the thermal infrared and visible light images and to fuse them. The variance calculation method is to sum the difference between each pixel value and the regional mean value, and then divide by the total number of pixels in the region. The energy calculation method is to sum the square of each pixel value in the region, and then divide by the total number of pixels in the region. If the energy of the thermal infrared image region is greater than that of the visible light image region, the weight of the thermal infrared image is calculated by an exponential function, and the weight of the visible light image is 1 minus the weight of the thermal infrared image. Otherwise, the weight calculation method is swapped. The high-frequency component selects the component with the largest absolute value in the two modalities, and the inverse wavelet transform is used to obtain the fused image. Adaptive median filter is used for the fused image and monocular camera video stream, and the noise density in the window is counted. When the noise density is less than 20%, a 3*3 window is used, and when the noise density is more than 20%, a 6*6 window is used. If the window size exceeds 6 and the noise density is still high, the original value of the center pixel is kept; The visible light contrast is calculated in the steam environment. The contrast is the difference between the maximum and minimum gray values divided by the sum of the maximum and minimum gray values. When the contrast is less than 30%, the thermal infrared weight is set to 0.7 and the visible light weight is set to 0.3. In the dust environment, the gray standard deviation is calculated. The standard deviation is the square root of the sum of the squares of the difference between each pixel value in the region and the regional mean, divided by the square root of the total number of pixels in the region. When the standard deviation is greater than 50, the initial filter window is set to 5*5. The window size increases by 1 for every 5% increase in noise density, and the maximum window size is 8*8.

[0025] The YOLOv5 model is improved based on the YOLOv5s framework. The Focus module is replaced with a 6*6 convolution module with a step size of 2 and a padding of 2. The BottleneckCSP module is replaced with a C3 module, which contains three convolution layers and four Bottleneck modules. The SPP module is replaced with a SPPF module, which contains three 5*5 max pooling layers. The FPN+PAN module is replaced with a BiFPN module, which removes the node with only one input edge. A channel is added between the input node and the output node, and the feature fusion block is repeated three times. A food processing plant personnel image dataset of 7000 images is created, divided into training and test sets in the ratio of 8:2. The training set uses Mosaic data augmentation, and the input image size is set to 640*640. The Batchsize is set to 64, the initial learning rate is set to 0.01, the weight decay is set to 0.0005, the momentum is set to 0.937, the training rounds are set to 300, the CIOU loss function is used to calculate the positioning loss, the BCEWithLogistsLoss is used to calculate the classification loss and confidence loss, and the model detection speed reaches 100FPS after training. The preprocessed video stream is input into the model to output the personnel pixel coordinates.

[0026] The improved DeepSORT model constructs a state estimation model with an 8-dimensional vector, which includes the target center horizontal pixel coordinate, the target center vertical pixel coordinate, the target box width-height ratio, the target box height, and the speed of the four parameters. A GIOU is used to construct a cost matrix, which is IOU minus the difference between the minimum bounding box area of the predicted box and the true box and the difference between the union area of the predicted box and the true box divided by the minimum bounding box area. The Mahalanobis distance and the cosine distance are combined for matching. The Mahalanobis distance is the transpose of the difference between the detection box vector and the tracking box vector multiplied by the inverse of the detection box and tracking box covariance matrix, and then multiplied by the difference between the detection box vector and the tracking box vector. The cosine distance is 1 minus the minimum value of the product of the normalized feature vector of the target box and the feature vector in the successful tracking feature vector set. According to the cascade matching strategy, the matching is started from the track with the frame difference being 0, missing age the track with the frame difference being 0, missing age the frame difference between the current frame and the last successful matching frame of the track, and the threshold is set to 70. A new track needs to be successfully matched for 3 consecutive frames to be converted to a determined state. Finally, the personnel ID and motion trajectory are extracted.

[0027] The improved OpenPose model uses MobileNetV3 instead of the original VGGNet. MobileNetV3 includes 16 Bneck modules, and the Bneck module uses an h-swish activation function. The activation function is the input value multiplied by the ReLU6 function processing input value plus 3, and then divided by 6. The ReLU6 function limits the input value to between 0 and 6. Replace the 7x7 convolution kernel in the double-branch structure with 1x1 convolution kernel and 2x3 convolution kernel. In the feature refinement stage, 3x3 convolution layers are used with 128 channels. Time frame merging module and pose residual fusion module are added. The time frame merging module realizes frame fusion by weighted sum according to the time index weight of the current frame and the previous and next frames. The pose residual fusion module calculates the residual by feature connection operation after weighting the time index weight of the pose difference between the current frame and the previous and next frames. The fall determination needs to meet three conditions: the knee joint descending speed is greater than 0.1 m / s and lasts for 3 frames, the human body bounding box width-height ratio is greater than 0.8, and the human body axis angle is less than 63°. The knee joint descending speed is the difference between the current frame knee joint Y coordinate and the previous frame knee joint Y coordinate divided by the difference between the current frame time and the previous frame time. The human body bounding box width-height ratio is the difference between the maximum X coordinate and the minimum X coordinate of the bounding box divided by the difference between the maximum Y coordinate and the minimum Y coordinate of the bounding box. The human body axis angle is the inverse sine value of the product of the absolute value of the dot product of the neck to pelvis vector and the horizontal unit vector divided by the product of the modulus of the neck to pelvis vector and the modulus of the horizontal unit vector. The AlexNet model includes 5 convolution layers, 3 pooling layers, and 3 fully connected layers. The convolution layer uses the ReLU activation function, and the fully connected layer adds Dropout with a dropout rate of 0.5. 500 images of A, B, C, D department personnel dust clothes are used as training data set, the image size is set to 224x224, the training round is 100, the initial learning rate is 0.001, the Adam optimizer is used, the loss function is cross entropy loss, and the cross entropy loss is the negative sum of the product of the true label and the logarithm of the predicted probability; When running, the personnel detection box image input model is intercepted, the personnel belonging department is determined according to the rule that black corresponds to A department, blue corresponds to B department, white corresponds to C department, and red corresponds to D department, and then the area intrusion and point inspection violation are judged, the area intrusion is that the non-D department personnel enters the quality detection area, and the point inspection violation is that the non-C department personnel enters the point inspection work area or the C department personnel does not complete the point inspection according to the specification.

[0028] The intrinsic matrix of the monocular industrial camera is: The rotation matrix and translation vector are obtained by combining the PnP algorithm to solve the extrinsic parameters: The translation vector T = [33.74424264, 88.03384245, 931.9332833]; Set the world coordinate Z axis value of the personnel's ground as 0, calculate and value, is the result of multiplying the inverse matrix of the rotation matrix and the inverse matrix of the intrinsic matrix, and then multiplying the pixel coordinate vector, is the result of multiplying the inverse matrix of the rotation matrix and the translation vector, and the scale factor Zc is the sum of the world coordinate Z axis value and the third item value divided by the third item value, the pixel coordinates are substituted into the camera imaging model to obtain the actual coordinates of the personnel, and the camera imaging model is the scale factor times the pixel coordinate vector is equal to the intrinsic matrix multiplied by the extrinsic matrix containing the rotation matrix and the translation vector, and then multiplied by the world coordinate vector, the positioning error is controlled within 0.1 meters.

[0029] When detecting a fall or a violation, the WeChat API is called to send a warning notice, the API calling parameters include enterprise ID, application ID and application key, the access_token is obtained through the enterprise ID and application key, and then the warning content is sent using the access_token, the warning content includes the type of anomaly, the time of occurrence, the content of the anomaly, the device information and the file link; The abnormal information is stored in a SQL Server database, and the database table fields include record number, occurrence time, location, personnel ID, abnormal type, and file path. The snapshot picture or video file is stored in a specified path and the file path is recorded. Based on Flask, a management platform is built, a sub-thread reads the RTSP video stream and controls the frame rate to be 30FPS, the main thread runs the improved YOLOv5 and DeepSORT models, a thread pool of 4 threads is created to process the AlexNet color classification and pose residual calculation respectively, the main thread returns the video stream in a specific way to ensure that the delay does not exceed 100ms, and supports computer, tablet, mobile phone and other devices to access.

[0030] Embodiment Two: This embodiment is aimed at point inspection violation judgment and personnel tracking optimization. Combined with cascade matching, fall judgment, point inspection logic and point inspection area positioning technology details, it solves the problems of unstable personnel tracking and inaccurate violation judgment in point inspection work area, and ensures that the point inspection behavior supervision meets the specifications.

[0031] The five key points of the workshop point inspection work area are accurately calibrated, and the actual coordinates of the five point inspection points are measured, such as point inspection point P1 The actual coordinates are 1.0 meters and 9.0 meters, P2 2.0 meters and 10.0 meters, P3 3.0 meters and 9.0 meters, P4 4.0 meters and 10.0 meters, P5 2.0 meters and 11.0 meters, the corresponding pixel coordinates of each point inspection point are obtained by shooting with a monocular industrial camera, such as P1 The corresponding pixel coordinates are 800 and 450, P2 850 and 500, P3 900 and 450, P4 950 and 500, P5 850 and 550, these coordinate data are stored in a special configuration file for subsequent algorithm calling.

[0032] At the same time, the pixel boundary of the point inspection area is clearly marked in the image, such as the left boundary of the pixel coordinates of the point inspection area is 750, the right boundary is 1000, the upper boundary is 400, and the lower boundary is 600. The algorithm only tracks and judges the behavior of the personnel appearing in this area, reducing the interference of irrelevant personnel outside the area on the judgment result.

[0033] In the point inspection period of 9:00-9:05 in the morning and 21:00-21:05 in the evening, the multi-target tracking parameters are optimized, missing ageThe threshold was adjusted from 70 to 50 to shorten the retention time when the trajectory does not match, avoid long-term invalid trajectory retention due to personnel obstructing the equipment, and improve the real-time performance and accuracy of tracking. The Re-ID feature extraction network was replaced with ResNet50, extracting 256-dimensional feature vectors to enhance feature representation capabilities. Compared to the original 128-dimensional feature vectors, this provides a more accurate description of personnel appearance. 300 images of personnel in inspection scenarios were added to the training set. These images include personnel from Department C wearing white cleanroom suits performing inspections. The feature matching module was retrained, and the cosine distance threshold was reduced from 0.3 to 0.25 to further improve the accuracy of feature matching. After optimization, the ID jump rate was reduced by 40%, ensuring that inspection personnel are stably tracked throughout the inspection process.

[0034] Design multi-dimensional inspection judgment rules, calculate the Euclidean distance between the tracking trajectory and each inspection point. If the distance is less than or equal to 0.3 meters, the inspection point is considered to be covered. For example, if the tracking trajectory passes through positions at 1.2 meters and 9.1 meters, the distance between the tracking trajectory and the inspection point is considered to be less than or equal to 0.3 meters. P1 The Euclidean distance is approximately 0.22 meters, which is considered as coverage. P1 ; The inspection completion time is set at 5 minutes, starting from when personnel from Department C enter the inspection area. Failure to complete all inspection points within the time limit constitutes a violation. If personnel from other departments enter the inspection area, or if Department C personnel fail to cover inspection points in the correct order (e.g., skipping points), the violation will be penalized. P3 Enter directly P4 The area will immediately trigger a violation determination to ensure compliance only when personnel from Department C complete the inspection according to the specifications, thus avoiding issues such as cheating or inadequate inspection.

[0035] When a violation occurs during inspection, the system automatically captures an image of the violator, stores the image in a specified path on the server, and records the file path. Simultaneously, it calls the WeChat API to send an alert to administrators. The alert clearly indicates the type of "inspection anomaly" and the department to which the violator belongs, such as "personnel from non-Department C have entered the inspection area" or "Personnel from Department C have not covered the inspection points." P3 The system must ensure that the coordinates of the inspection points, the time of occurrence, and the associated camera information, including the camera IP address, are not covered, so that management personnel can obtain details of violations in a timely manner and take appropriate action.

[0036] Example 3: This embodiment focuses on multi-camera deployment and coordinate transformation, combining monocular visual positioning extrinsic parameter solving and cross-camera ID association feature matching technology requirements to solve the problems of time asynchrony, spatial disconnection, and ID discontinuity under multi-camera monitoring, and realize collaborative supervision of the entire workshop.

[0037] After deploying multi-modal cameras and monocular industrial cameras in the raw material processing area, equipment operation area, quality detection area, and point inspection work area of the workshop, deploying an NTP server and setting a fixed IP address, such as IP address 192.168.1.10, logging into the Web management interface of each camera, configuring NTP synchronization parameters in the network settings, synchronizing the time of the camera to the NTP server, regularly checking the synchronization status through the camera management interface, and ensuring that the time error of all cameras is controlled within 10 ms, providing a unified reference for subsequent multi-camera data time alignment, and avoiding trajectory breakage caused by time deviation.

[0038] Based on the coordinate conversion model, the multi-camera space is spliced. In the workshop, fixed and easily identifiable objects such as columns are selected as public control points, such as three columns Q1 、 Q2 、 Q3 The actual world coordinates of these control points are accurately measured using professional measurement tools, such as Q1 5 meters and 5 meters, Q2 10 meters and 5 meters, Q3 10 meters and 10 meters.

[0039] Find these public control points in the monitoring screen of different cameras respectively, and record their corresponding pixel coordinates, such as Q1 700 and 400 in camera 1, Q2 800 and 400, Q3 800 and 500 in camera 2, Q1 900 and 300, Q2 1000 and 300, Q3 1000 and 400, calculate the perspective transformation matrix from each camera to the world coordinates using the functions provided by OpenCV, and use the matrix to convert the images collected by each camera into bird's-eye view in the world coordinates. According to the actual layout of the workshop, the bird's-eye view corresponding to different cameras is spliced together to form a complete 15x15m workshop panoramic view, realizing seamless connection of multi-camera monitoring area.

[0040] For multi-target tracking, cross-camera ID association is realized, and the comprehensive feature vector of the personnel is extracted, combined with the color feature output by the AlexNet model and the Re-ID feature extracted by the ResNet50 model. The AlexNet color feature is a 4-dimensional vector, and the ResNet50 Re-ID feature is a 256-dimensional vector. The two feature vectors are spliced to form a 320-dimensional comprehensive feature vector, which can more comprehensively describe the appearance information of the personnel and reduce the matching error caused by a single feature. When the same person enters the monitoring area of another camera from the monitoring area of one camera, the 320-dimensional integrated feature vector of the person is extracted after the person is detected in the new camera, the cosine similarity is calculated with the feature vector of the tracked person in other cameras, and if the similarity is greater than 0.9, it is determined that it is the same person, the person ID in the new camera is associated with the original ID, the continuous tracking across cameras is realized, and the continuity of tracking when the person moves in the workshop is ensured.

[0041] The pixel coordinates of the person output by each camera are converted into world coordinates through the corresponding perspective transformation matrix, and a coordinate system with a Z-axis value of 0 on the ground of the workshop is uniformly adopted to ensure that the actual coordinates of the same person under different cameras are consistent and to avoid positioning deviation caused by different camera angles. When the person is in the overlapping area covered by multiple cameras, the average value of the actual coordinates calculated by multiple cameras is taken to further reduce the positioning error and control the error within 0.08 meters, thereby improving the global positioning accuracy.

[0042] Embodiment Four This embodiment is optimized for special environments and model adaptation, combining dust environment filtering, improved YOLOv5 dataset and anchor box, and tracking parameter adjustment techniques to solve the problems of poor image quality and low detection and tracking accuracy in high-dust environments in food processing workshops, and to ensure that the robustness of the algorithm meets the performance standards.

[0043] The image preprocessing process is enhanced. First, linear gray scale stretching is performed on the dust image to expand the gray scale range from 50 to 200 to 0 to 255, thereby improving the contrast of the image through redistribution of the gray scale value, making the person's outline and details clearer, and facilitating subsequent detection algorithm recognition. Before adaptive median filtering, the image is first subjected to bilateral filtering processing, with the diameter of the bilateral filter set to 9, sigmaColor set to 75, and sigmaSpace set to 75. This filtering method can effectively remove dust noise while preserving image edge information, avoiding edge blurring caused by traditional filtering. After processing, the PSNR of the image is improved from 22 dB to 28 dB, providing high-quality image data for subsequent detection.

[0044] In summary, the intelligent on-site monitoring and management method for the food processing workshop first acquires video streams from two types of cameras, namely, a multi-modal camera integrating visible light and thermal infrared modules and a monocular industrial camera supporting the RTSP protocol. After wavelet transform of the visible light and thermal infrared video streams, adaptive weighted fusion is performed. Then, adaptive median filtering is used to denoise the fused video stream and the monocular industrial camera video stream, thereby improving the image quality in complex environments such as steam and dust, providing a clear image basis for subsequent personnel behavior recognition, and solving the problem of low personnel behavior recognition rate in traditional visible light monitoring systems.

[0045] And, the intelligent field monitoring management method of the food processing plant, the improved YOLOv5 model outputs personnel pixel coordinates after preprocessing the video stream input, the coordinates input the improved DeepSORT model to extract personnel ID and motion trajectory, and the improved OpenPose model extracts the human joint node to determine the fall, the AlexNet model identifies the color of the dust-free clothes to determine the department to which the personnel belong to determine the regional intrusion and point inspection violation, and the enterprise WeChat API is called to send an early warning when an anomaly is detected, realizing intelligent identification and real-time supervision of personnel key behaviors, solving the problem of lack of such capabilities in traditional systems.

[0046] And, the intelligent field monitoring management method of the food processing plant, the improved YOLOv5 model obtains the personnel detection frame, the image in the detection frame is intercepted and input into the AlexNet model containing 5 convolution layers, 3 pooling layers and 3 fully connected layers, the model determines the department to which the personnel belong to by identifying the color of the dust-free clothes, and the activity range and authority of personnel in each department are clear, providing a basis for precise management of different departments, solving the problem that the traditional monitoring system cannot automatically identify the department to which the personnel belong to and is difficult to manage specifically, and solving the problem that the existing food processing plant monitoring technology has low personnel behavior recognition rate in complex environments, lacks intelligent identification and early warning capabilities for personnel key behaviors, and cannot automatically identify the department to which the personnel belong to, making it difficult to manage specifically.

[0047] The related modules involved in the system are hardware system modules or functional modules combined with computer software programs or protocols and hardware in the prior art. The computer software programs or protocols involved in the functional modules are known to those skilled in the art and are not improvements of the system. The improvement of the system is the interaction or connection relationship between the modules, i.e., the improvement of the overall structure of the system to solve the corresponding technical problems of the system.

[0048] Although embodiments of the present application have been shown and described, it is to be understood that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An intelligent on-site monitoring and management method for a food processing workshop, characterized in that, Includes the following steps: S1: Deploy multimodal cameras and monocular industrial cameras in the raw material processing area, equipment operation area, quality inspection area, and spot inspection work area of ​​the workshop. The multimodal camera integrates visible light and thermal infrared modules, and the monocular industrial camera supports the RTSP protocol. Use Zhang's calibration method to calibrate the monocular camera to obtain the intrinsic parameter matrix and distortion coefficients, and calibrate the two modules of the multimodal camera separately. S2: Obtain the visible light video stream, thermal infrared video stream, and monocular industrial camera video stream via the RTSP protocol; perform wavelet transform on the visible light video stream and thermal infrared video stream and then adaptively weighted fuse them; apply adaptive median filtering to denoise the fused video stream and the monocular industrial camera video stream. S3: Input the preprocessed video stream into the improved YOLOv5 model to output the pixel coordinates of the person. Improve YOLOv5 based on the YOLOv5s framework, use the CIOU loss function to calculate the localization loss, and use BCEWithLogistsLoss to calculate the classification loss and confidence loss. S4: Input the pixel coordinates of the personnel into the improved DeepSORT model. This model constructs a state estimation model with an 8-dimensional vector, uses GIOU to construct a cost matrix, and combines "Madara distance + cosine distance" for matching. The personnel ID and motion trajectory are extracted through cascaded matching. S5: An improved OpenPose model is used to extract human joints and determine falls. An AlexNet model is used to identify the color of cleanroom garments to determine the department to which the personnel belong, thereby determining area intrusion and inspection violations. S6: Based on the intrinsic parameter matrix of a monocular industrial camera, the extrinsic parameters are solved by combining the PnP algorithm, a conversion model between pixel coordinates and actual workshop coordinates is established, and the actual coordinates of personnel are calculated. S7: When a fall or violation is detected, store abnormal information and simultaneously call the WeChat Work API to send an alert notification; S8: The management platform is built on Flask, and the platform's real-time video module uses multi-threaded processing.

2. The intelligent on-site monitoring and management method for a food processing workshop according to claim 1, characterized in that, The specific rules for adaptive weighted fusion in S2 are as follows: The low-frequency components are weighted using a weighting factor calculated based on the regional variance and energy. The formula for calculating the regional variance is as follows: The energy calculation formula is: in, For area size, For pixel values, This is the regional mean. Let the energy of the thermal infrared image region be... The energy of the visible light image region is ,when > At that time, thermal infrared image weight Visible light image weights ,when > Sometimes the opposite is true; The high-frequency component is selected from the two modes with the largest absolute value; the adaptive median filtering rule is: a 3×3 window is used when the noise density is ≤20%, a 6×6 window is used when the noise density is >20%, and the original value of the center pixel is maintained when the window size exceeds 6 and the noise density is still high.

3. The intelligent on-site monitoring and management method for a food processing workshop according to claim 1, characterized in that, The structural improvement of the YOLOv5 model in S3 is as follows: The Focus module is replaced with a 6×6 convolutional module, which has a stride of 2 and a padding of 2. The BottleneckCSP module is replaced with a C3 module, which contains 3 convolutional layers and 4 Bottleneck modules. The SPP module is replaced with an SPPF module, which contains 3 5×5 max pooling layers. The FPN+PAN module is replaced with a BiFPN module. The BiFPN module removes nodes with only one input edge, adds a channel between the input and output nodes, and repeats the feature fusion block 3 times. The model training dataset consists of 7,000 images of people in a food processing workshop, divided into training and testing sets in an 8:2 ratio. The training set uses Mosaic data augmentation. The input image size is 640×640, the batch size is 64, the initial learning rate is 0.01, the weight decay is 0.0005, the momentum is 0.937, and the training epochs are 300. After training, the model's detection speed reaches 100 FPS.

4. The intelligent on-site monitoring and management method for a food processing workshop according to claim 1, characterized in that, The specific design of the improved DeepSORT model in S4 is as follows: The 8-dimensional state vector is ,in The horizontal and vertical pixel coordinates of the target center. The aspect ratio of the target bounding box. The height of the target bounding box. They are respectively speed; The formula for calculating GIOU is as follows: ,in To enclose the prediction box With real frame The smallest bounding box; The formula for calculating Mahalanobis distance is: ,in For the first Each detection bounding box vector For the first A tracking box vector, The inverse of the covariance matrix between the detection box and the tracking box; The formula for calculating cosine distance is: ,in For the first Normalized feature vectors of each bounding box. For the first In the set of feature vectors that were successfully tracked, the first... k 1 eigenvector; Cascade Matching from The trajectory begins, The frame difference between the current frame of the trajectory and the last successfully matched frame. A new trajectory needs to be successfully matched for 3 consecutive frames to become a deterministic state.

5. The intelligent on-site monitoring and management method for a food processing workshop according to claim 1, characterized in that, The improved OpenPose model in S5 is designed as follows: The MobileNetV3 network replaces the original VGGNet. MobileNetV3 includes 16 Bneck modules, and the Bneck modules adopt... The activation function, where: The 7×7 convolutional kernel in the dual-branch structure is replaced with one 1×1 convolutional kernel and two 3×3 convolutional kernels. In the feature refinement stage, three 3×3 convolutional layers with 128 channels are used. A temporal frame merging module and a pose residual fusion module are added. The frame fusion formula for the time frame merging module is: For video frame time index; The residual formula for the pose residual fusion module is: For feature connection operations; The criteria for determining a fall are: a descent speed of both knee joints > 0.1 m / s for 3 frames, an aspect ratio of the bounding rectangle of the human body > 0.8, and a human body axis angle < 63°; The formula for the descent velocity of both knee joints is: , The Y-coordinate of the right knee joint in the current / previous frame. For the corresponding time; The aspect ratio formula is , , , , Find the maximum and minimum values ​​of the circumscribed rectangle; The formula for the axis angle is: , The neck-pelvis vector. It is a horizontal unit vector.

6. The intelligent on-site monitoring and management method for a food processing workshop according to claim 1, characterized in that, The specific implementation of monocular vision localization in S6 is as follows: The camera imaging model formula is: in As a scale factor, This is the intrinsic parameter matrix. For rotation matrix, It is a translation vector. World coordinates; Intrinsic parameter matrix The distortion coefficient is =-0.37068473, =0.20948198, =0.00908307, p1 =-0.00526021, p2 =2.00344022; The extrinsic parameters are solved using the PnP algorithm. The rotation matrix is: Translation vector T = [33.74424264, 88.03384245, 931.9332833]; When calculating the actual coordinates of personnel, set as follows: Substituting into the imaging model yields The positioning error is ≤0.1m.

7. The intelligent on-site monitoring and management method for a food processing workshop according to claim 1, characterized in that, The design and operation flow of the AlexNet model in S5 is as follows: The model contains 5 convolutional layers, 3 pooling layers, and 3 fully connected layers. The convolutional layers use the ReLU activation function, and the fully connected layers have Dropout with a dropout rate of 0.

5. The training dataset consists of 500 images of cleanroom garments worn by personnel in departments A, B, C, and D, each image measuring 224×224 pixels. There are 100 training epochs with an initial learning rate of 0.

001. The optimizer is Adam, and the loss function is... , For real labels, To predict probabilities; At runtime, the image of the personnel detection box is captured and input into the model. The output color corresponds to the department: black corresponds to department A, blue corresponds to department B, white corresponds to department C, and red corresponds to department D.

8. The intelligent on-site monitoring and management method for a food processing workshop according to claim 1, characterized in that, The implementation process of the linkage early warning in S7 is as follows: Enterprise WeChat API call parameters include (Company ID) (Application ID) (Application Key); Exception information is stored using an SQL Server database, and the storage table fields include... (Record Number) (Time of occurrence) (Place), (Personnel ID) (Abnormal type) (File path); The warning notification includes the type of anomaly, the time of occurrence, the content of the anomaly, the equipment information, and the file link. The content of the anomaly describes the personnel, department, violation, and area involved, while the equipment information describes the camera's IP address and model.

9. The intelligent on-site monitoring and management method for a food processing workshop according to claim 1, characterized in that, The multi-threaded design of the real-time video module in S8 is as follows: Create a child thread to read the RTSP video stream, controlling the frame rate to 30 FPS; The main thread runs an improved YOLOv5 and DeepSORT model, and receives video frames from the sub-thread to output the coordinates and IDs of the personnel. Create a thread pool with 4 threads to handle AlexNet color classification and pose residual calculation respectively; The main thread passes Returns video stream with a latency of ≤100ms, and supports access from computers, tablets, and mobile phones.

10. The intelligent on-site monitoring and management method for a food processing workshop according to claim 1, characterized in that, The optimization of the food processing workshop environment in S2 is as follows: Calculating the contrast of visible light images in a steam environment , This represents the maximum and minimum grayscale values. When the temperature is <30°C, the weighting for thermal infrared light is 0.7 and the weighting for visible light light is 0.

3. Calculate the standard deviation of grayscale in a dusty environment , >50 indicates high dust levels. The initial filter window is 5×5. For every 5% increase in noise density, the window size increases by 1, up to a maximum of 8×8.

Images