Intelligent belt conveyor monitoring system

Abstract

The invention relates to an intelligent belt conveyor monitoring system which is characterized in that a material image is acquired, a laser stripe center line skeleton is extracted, a laser stripe center position is determined, an extraction result is optimized, a closed curve is obtained, and an energy-saving optimization model of a belt conveyor is constructed according to a material quantity detection result. Self-adaptive speed regulation control is realized, so that the accuracy of material quantity detection and the operation efficiency of the belt conveyor are effectively improved, and powerful support is provided for optimizing energy-saving control; the system utilizes nine joint points to generate a confidence map, predicts the correlation degree between key points through a part of affinity fields, constructs a human body skeleton, utilizes a time clue to estimate the positions of the joint points when the joint points are shielded, combines a pose residual fusion module to improve the accuracy of pose estimation, and combines a spatial domain constraint condition to calculate the knee point descending speed. Fall and false fall actions can be accurately distinguished, an alarm can be triggered when fall is detected, and related personnel are notified through a digital twinning control system.

Description

Technical Field

[0001] The invention relates to the field of belt conveyors, and in particular to an intelligent belt conveyor monitoring system. Background Art

[0002] In actual industrial production, belt conveyors are an important type of material handling equipment. Their working principle is to transport materials from one place to another through a continuous conveyor belt. However, in the actual working process, the output frequency of such equipment is usually preset to a fixed value, and it is always kept in a high-speed operation or stopped state. This long-term high-speed operation and frequent start and stop will undoubtedly aggravate the wear of the equipment, and will also cause unnecessary power consumption, which will affect the stability and economy of the equipment. In addition, most of the current belt conveyors cannot accurately judge the state of the materials on the conveyor belt. Therefore, it is impossible to adjust the running speed of the belt conveyor in real time according to the actual state of the materials to achieve the purpose of energy saving and consumption reduction. In this case, the operating efficiency and energy utilization rate of the equipment will be greatly affected, which not only increases the operating costs of the enterprise, but also is inconsistent with the concept of energy conservation, emission reduction and green development advocated by our country. Therefore, in response to these problems, we need to find effective solutions to improve the operating efficiency of belt conveyors, reduce energy consumption, and achieve stable operation of equipment.

[0003] In the daily operation of the belt conveyor, since the belt conveyor is usually in a relatively closed environment, the environment around the entire corridor is usually poor. In such an environment, if a person encounters an accident during inspection, such as falling or falling due to a sudden illness, it is difficult to detect it in time. First, the light in the workshop varies greatly, which leads to poor imaging quality of the camera in dim or night scenes, and it is impossible to ensure that the fall behavior can be identified in the factory all day long. Secondly, the noise source in the workshop is very complex, and traditional equipment is easily affected by factors such as large and medium-sized equipment, electromagnetic interference, and dust in the factory, which may cause the input video to be defocused or the worker's posture to be partially blocked, making it more difficult to extract features. Finally, the action set of the inspection worker includes some pseudo-fall actions, and there are also some situations similar to the fall action in the worker's daily actions. Since the fall features extracted by the traditional algorithm cannot accurately identify the pseudo-fall action, the possibility of misjudgment increases. Therefore, we need an algorithm that can accurately identify the fall behavior to improve the efficiency and accuracy of the inspection. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the deficiencies of the prior art, the present invention provides an intelligent belt conveyor monitoring system to solve the problems raised in the above background technology.

[0006] (II) Technical solution

[0007] To achieve the above object, the present invention provides the following technical solution: an intelligent belt conveyor monitoring system, including a laser stripe detection device, a camera, a controller, a computing device and an adaptive speed control subroutine, the system can achieve the following functions:

[0008] The pixel point information of the center line skeleton of the laser stripe is obtained by the laser stripe detection device. The steps of obtaining the pixel point information of the center line skeleton of the laser stripe by the laser stripe detection device are as follows: setting the region of interest to obtain the light stripe image, using bilateral filtering to add a pixel value weight term on the basis of Gaussian filtering, and expressing the pixel value weight as G r , the spatial distance weight is represented by G s , then the weight BF calculation method of a point p in the bilateral filtering process in the template window centered on point q is as follows: W q represents a two-dimensional convolutional template, I p The grayscale value of point p is represented by integrating the grayscale information of all pixels in the image. By setting a reasonable threshold, the image is binarized and segmented. The laser stripes of a certain width in the material image are converted into the center line of the laser stripes that can represent the true center position. The linear interpolation method is used to quickly repair the center line of the laser stripes. The geometric center point on each longitudinal section of the laser stripes is regarded as the center point of the laser stripes. 1 is the ordinate of the upper edge point of a column in the laser stripe, C 2 is the ordinate of the lower edge point of a column of laser stripes, then the ordinate C of the center position of the laser stripes in the column is:

[0009] The controller selects the initial i-th point and i+1-th point from the laser stripe centerline skeleton and calculates the normal direction of the line between the two points. The steps for calculating the normal direction of the line between the two points are as follows: the laser stripe centerline skeleton consists of n pixels in total, and the initial i-th point (xi, yi) and the i+1-th point (xi +1 ,yi +1 ) and calculate the normal direction of the connected straight line. According to the grayscale information of the laser stripe at the normal direction of the i+1th point, the new center point of the laser stripe (x ci ,y ci ), the normal direction m of the straight line connecting the two points is calculated according to the slope k between the two points as follows:

[0010] The controller introduces the Manhattan distance to judge the validity of the new laser stripe center point and optimizes the extraction result of the laser stripe center; the Manhattan distance represents the arithmetic sum of the horizontal and vertical coordinates of two points in the coordinate axis, and calculates the new laser stripe center point (x ci ,y ci ) and the i+1th point (x i+1 ,y i+1 ) between the Manhattan distance L, the formula is as follows, L = |x i+1 -x ci |+|y i+1 -y ci |

[0011] If L is less than or equal to 4, the new laser stripe center point is considered valid, and the new laser stripe center point is used to replace and update the second point in the original laser stripe skeleton; if L is greater than 4, the new laser stripe center point is considered invalid, and the second point in the original laser stripe skeleton is retained;

[0012] The controller traverses all the laser stripe centerline skeleton points until all the laser stripe center points are extracted;

[0013] The computing device superimposes the center line of the laser stripe of the conveyor belt contour in the unloaded image of the belt conveyor and the center line of the laser stripe of the material in the loaded image of the belt conveyor to obtain a closed curve, and calculates the instantaneous cross-sectional area S enclosed by the closed curve;

[0014] The computing device calculates the volume V of the material flowing through and the instantaneous material quantity Q within a single frame sampling time according to the sampling frame rate of the camera and the running speed of the belt conveyor. S , the computing device takes the average value of the instantaneous material quantity within multiple sampling times to obtain the material quantity Q of the belt conveyor;

[0015] The controller determines the appropriate conveyor belt running speed according to the material quantity detection result and the belt conveyor energy-saving optimization model, and realizes the adaptive speed control of the belt conveyor through the adaptive speed control subroutine.

[0016] Furthermore, the region of interest is a rectangle, the horizontal length of the region of interest should be consistent with the effective length of the laser stripes on the conveyor belt, the lowest point in the vertical direction of the region of interest should include the laser stripes at the lowest point of the surface when the conveyor belt is unloaded, and the highest point in the vertical direction of the region of interest should include the laser stripes on the material surface at the highest point when the conveyor belt is fully loaded.

[0017] Furthermore, the center line of the laser stripe of the conveyor belt contour in the unloaded image of the belt conveyor and the center line of the laser stripe of the material in the loaded image of the belt conveyor are superimposed to obtain a closed curve. The figure enclosed by the closed curve represents the instantaneous cross-sectional area S formed by the material on the conveyor belt at a certain moment. For each point on the closed curve, the relative height ΔH of the material relative to the conveyor belt surface is calculated. i , and the actual width Δx of each pixel in the horizontal direction corresponding to the height ΔHi of the material surface i , then according to the formula, S = ∑(Δx i ×ΔH i ) calculates the instantaneous cross-sectional area S enclosed by the closed curve; the running speed of the conveyor belt is v, and the sampling frame rate of the camera is f c , the instantaneous cross-sectional area is regarded as the volume component of the material in the direction of travel, then the volume V of the material flowing through the single frame sampling time is: V = [∑ (Δx i ×ΔH i )]×v / f c ; According to the density ρ of the transported material, calculate the instantaneous material quantity Q of the belt conveyor within the single frame sampling time S :Q s =ρ×v×∑(Δx i ×ΔH i ); The calculation formula for the material quantity Q of the belt conveyor is:

[0018] Furthermore, according to the belt conveyor energy-saving optimization model, the material volume is divided into n intervals [Q 0 ,Q 1 ]、[Q 1 ,Q 2 ]…[Q n-1 ,Q n ], the corresponding conveyor belt running speeds are v 0 、v 1 …v n-1 .

[0019] Furthermore, a fall detection subroutine is included, which is composed of the following modules:

[0020] a. Video acquisition module, which uses the method of fusion of visible light camera and thermal imaging camera to acquire images, and is used to collect monitoring videos of the working area of ​​the belt conveyor;

[0021] b. A detection module, which uses a human posture estimation algorithm to identify key points of the human body in the video and detects whether a fall event occurs based on the key points; wherein the human posture estimation algorithm includes:

[0022] Using confidence map C n,mDescribes the probability distribution of key points of the human body, where X n,m is the true position of the key points of the human body, σ n,m Describe the probability distribution of corresponding key points;

[0023] The degree of association between two key points is predicted using the partial affinity field, the correct connection of each type of limb is determined by the integral value H, and the limbs that share the same key points are connected to form a human skeleton;

[0024] When the human body is partially occluded, the position of the joint points when it is occluded is estimated based on the time clues, and the feature tensor φ is constructed i ;

[0025] The pose residual ψ is calculated through the residual fusion module i , as clues for pose estimation;

[0026] The threshold α of the knee point descent speed is set to distinguish between falling actions and pseudo-falling actions;

[0027] The human body axis inclination angle θ and the height-to-width ratio R of the human body circumscribed rectangular frame are introduced as spatial constraints to further confirm the fall event.

[0028] c. Detection information module, used to display the number of people in the current video, the safety status of the characters, and trigger an alarm when a fall event is detected;

[0029] d. Alarm information module. When the detection module detects a fall event, the module sends the video screenshot of the fall and the location information of the incident to the digital twin control system in the main control room, and notifies relevant personnel in the form of alarm sounds and text messages.

[0030] Furthermore, in the detection module, an image with a clear outline of a person is used as the model input, and the large 7×7 convolution kernel in the original network is replaced with three 3×3 convolution kernels. The obtained features are used as the input of the OpenPose model. After double-branch multi-level iterative network processing, key points and skeleton maps can be generated for the person in each frame of the image. The lightweight model after cropping can accurately identify the 19 joints in the human skeleton sequence. The formula for generating a confidence map of the human joints in each frame of the image is shown as follows:

[0031]

[0032] Where: m is the character number, n is the key point of the human body, Q is the confidence map C n,m Any point in X n,m is the real position of the key points of the human body, σ n,m Used to describe the probability distribution of corresponding key points.

[0033] In the further detection module, the correlation between two key points is predicted by partial affinity fields, which can be expressed by the integral value H as:

[0034]

[0035] Where: n 1 、n 2 are adjacent key points, ||n 2 -n 1 ||2 is the length of the limb, Q(t)∈[n 1 ,n 2 ], if point Q is on the limb, then k b (Q(t)) = v, where v = (m 2 -m 1 ) / ||m 2 -m 1 ||2 is a unit vector, otherwise, k b (Q(t))=0; the correct connection of each type of limb is obtained by selecting the maximum value of H, and then the limbs that share the same key points are connected into a human skeleton.

[0036] In the further detection module, there are rich temporal clues between the occluded time frame F(s) and the marked adjacent frames F(j) and F(l), so the time frame merging result F of the partially occluded person i is i (j,s,l) is:

[0037]

[0038] Where j, s, and l are time frame indices. For each joint point, specific time information can be specified and calculated and merged into a key point heat map, which can be stacked into a feature tensor φ after regrouping. i , expressed as:

[0039]

[0040] Among them: ⊕ is the connection frame operation, N is the total number of joints, and n is the joint number.

[0041] In the further detection module, the obtained different feature tensors are input into the residual fusion module. Similar to the key point time frame merging, the pose residual ψ is calculated by pose residual fusion. i , as another effective clue for pose estimation, has been realized

[0042] ψ i (j,s)=F i (s)-F i (j),

[0043] ψ i(s,l)=F i (l)-F i (s),

[0044] The process is as follows:

[0045] By concatenating the original residual ψ i (j,s),ψ i (s,l) and its corresponding time weights, and processed by a 3×3 residual block, the final position residual estimate can be obtained, so that the model can detect the accurate human key points and skeleton position features in visible light images and thermal imager images, and label the key points of body parts.

[0046] In the further detection module, α=0.1m / s is selected as the threshold for distinguishing falls and other pseudo-falls. During the distinction process, the knee point moving speed V between adjacent frames is calculated and compared with the threshold to achieve distinction. The calculation formula for the knee point moving speed is:

[0047]

[0048] The key point coordinates can be obtained by extracting the human skeleton, where the coordinates of the left knee point are (X 13 ,Y 13 ), the coordinates of the right knee point are (X 10 ,Y 10 ), so in the formula: Y 10 Y′ is the ordinate of the right knee point in the current frame; 10 Y is the vertical coordinate of the right knee point in the previous frame; 13 Y′ is the ordinate of the left knee point in the current frame; 13 is the ordinate of the left knee point in the previous frame; T is the current time; T′ is the time of the previous frame; when Φ is true, it means that at least one knee point has a sudden change in descent speed. If there is a sudden change in three consecutive frames, it is judged as a fall action, otherwise, it is a non-fall action; the human body axis inclination angle θ and the aspect ratio R of the human body circumscribed rectangular frame are used as spatial constraints, and the maximum value of the key point coordinates X is selected. max , Y max and minimum value X min , Y min Construct a rectangular frame circumscribing the human body, and obtain the aspect ratio R of the rectangular frame circumscribing the human body:

[0049] By selecting the neck keypoint (X 1 ,Y 1 ), center point (X 8 ,Y 8 ) generates a vector representing the position and direction of the human body axis As shown below: The angle between the human body axis and the unit vector of the horizontal ground can be expressed as: When R is less than the threshold value of 0.8 and the human body inclination angle satisfies 63°≤θ≤117°, the action is judged as non-falling. If this condition is not met, it is judged as falling.

[0050] (III) Beneficial effects

[0051] Compared with the prior art, the present invention provides an intelligent belt conveyor monitoring system, which has the following beneficial effects:

[0052] (1) The conveyor belt speed is precisely adjusted according to the instructions of the control platform. It controls the motor speed by fine-tuning the frequency converter, thereby realizing intelligent control of the conveyor belt speed. It can accurately determine the amount of material on the conveyor belt and calculate the optimal operating speed of each belt conveyor based on this, which not only improves efficiency but also saves energy. At the same time, by monitoring the operating status of all equipment in real time, every detail of the system is mastered to ensure the continuity and stability of production.

[0053] (2) The clarity and coherence of the real-time images of the fall detection subroutine provide rich visual materials for subsequent analysis. It adopts a human posture estimation algorithm, which can accurately identify the key points of the human body from the video screen and monitor the safety status of the people in the video at all times. Once a fall event is detected, it will immediately initiate an emergency response and present the fall information in an intuitive graphical interface, allowing monitoring personnel to respond quickly and ensure the safety of the lives of on-site personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is the unloaded diagram of the belt conveyor;

[0055] Figure 2 It is a material diagram in a complex environment;

[0056] Figure 3 is the centerline diagram of the laser stripe;

[0057] Figure 4 is the new laser stripe center point diagram;

[0058] Figure 5 Flow chart for obtaining new laser stripe center point using geometric center method;

[0059] Figure 6 Calculate the cross-sectional area S formed by the center line of the conveyor belt contour laser line and the center line of the material laser line;

[0060] Figure 7 Provide energy-saving optimization model diagram for belt conveyor;

[0061] Figure 8 It is the speed regulation flow chart of the present invention;

[0062] Fig. 9 It is the flow chart of the adaptive speed control subroutine;

[0063] Fig.10 Label the key points of body parts;

[0064] Fig.11 Block diagram of the fall subroutine. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] When inspecting materials, the laser transmitter and camera remain stationary, their relative positions do not change, and the belt conveyor continues to operate. Figure 1 As shown in the figure, the center line of the laser stripe of the conveyor belt contour in the unloaded image of the belt conveyor is compared with the center line of the light stripe in the loaded image of the belt conveyor. In the actual image processing process, only the area where the light stripe is located should be paid attention to. By setting a reasonable area of ​​interest to obtain the light stripe image, it is possible to avoid processing the entire image, reduce the workload of image processing, and shorten the time used for image processing.

[0067] When determining the region of interest, it is necessary to fully consider the possible positions of the laser stripes in the material image to avoid large deviations in the material quantity calculation due to the loss of some laser stripes in the region of interest. The region of interest determined in this system is a rectangle, which is mainly subject to three position constraints, as follows:

[0068] 1. The horizontal length of the area of ​​interest should be consistent with the effective length of the laser stripes on the conveyor belt.

[0069] 2. The lowest point in the vertical direction of the area of ​​interest should contain the laser stripe at the lowest point of the conveyor belt surface when it is unloaded.

[0070] 3. The highest point in the vertical direction of the area of ​​interest should include the laser stripes on the material surface at the highest point when the conveyor belt is fully loaded.

[0071] Laser stripes have high brightness in grayscale images. However, in complex background environments, when the laser beam is projected onto the surface of an object, multiple diffuse reflections will produce diffuse light spots, resulting in inconsistent laser stripe width, unfocused brightness, unclear edges, and other issues in the collected material image. Figure 2As shown in the figure, the difficulty of laser stripe extraction is increased. At the same time, under the influence of ambient light, the material will produce a certain amount of bright spot noise due to the reflection of the surface, and the brightness of the spot noise is close to the brightness of the laser stripe, which will affect the extraction of the laser stripe. Therefore, it is necessary to use appropriate image denoising technology to smooth the material image, retain the laser stripe detail information and remove the interference of noise in the material image.

[0072] Bilateral filtering is used to add pixel value weight items on the basis of Gaussian filtering, that is, considering the distance factor and the difference in pixel values. The principle of Gaussian filtering is to use the weighted average of pixels in the neighborhood of the sliding template to replace the value of the central pixel of the template. Assuming that in the template window centered on point q, the weight calculation method of a certain point p in the Gaussian filtering process is:

[0073]

[0074] The pixel value weight is represented as G r , the spatial distance weight is represented by G s , then the weight BF calculation method of a point p in the bilateral filtering process in the template window centered on point q is as follows:

[0075]

[0076] W q represents a two-dimensional convolutional template, Ip represents p The gray value of the point.

[0077] The grayscale information of all pixels in the comprehensive image is used to achieve binary segmentation of the image by setting a reasonable threshold. Since the laser stripes in the material image have a certain width, in order to maximize the accuracy of material quantity detection, the laser stripes of a certain width in the material image are converted into the center line of the laser stripes that can represent the true center position. The linear interpolation method is used to quickly repair the broken line in the center of the laser stripes, so that the material quantity can be calculated linearly based on the center line of the laser stripes later. The principle of the interpolation method is to combine the coordinate information of the left and right end points of the curve to be interpolated, calculate the slope between the two end points, and use the straight line formed between the two end points to quickly complete the curve.

[0078] like Figure 3 As shown, the geometric center point on each longitudinal section of the laser stripe is regarded as the center point of the laser stripe, C1 is the ordinate of the upper edge point of a column in the laser stripe, and C2 is the ordinate of the lower edge point of a column in the laser stripe. Then the ordinate C of the center position of the laser stripe in the column is:

[0079] Assume that the center line skeleton of the laser stripe consists of n pixels in total, and take its initial i-th point (x i ,yi ) and the i+1th point (x i+1 ,y i+1 ) and calculate the normal direction of the connected straight line. In the normal direction of the i+1th point, according to the grayscale information of the laser stripe, the new center point (x ci ,y ci ). The normal direction m of the line connecting two points is calculated based on the slope k between the two points as follows:

[0080]

[0081] The present invention sets the maximum width of the laser stripe cross section in the collected material image to 7 pixels, so only the coordinates of the 7 pixels (including the i+1th point) through which the normal line passes at the i+1th point can be counted, avoiding slowing down the calculation time of the algorithm due to counting too many redundant pixels. The coordinates of the 7 pixels are marked as (u 1 ,v 1 ),(u 2 ,v 2 )…(u 7 ,v 7 ), the corresponding gray value is g(u 1 ,v 1 ),g(u 2 ,v 2 )…g(u 7 ,v 7 ). Calculate the grayscale centroid value of each pixel in the horizontal and vertical directions as the horizontal x value of the new laser stripe center point. ci Coordinates and vertical y ci Coordinates, such as Figure 4 , the calculation principle is shown in the formula.

[0082]

[0083] Manhattan distance is introduced to judge the validity of the new laser stripe center point and optimize the extraction result of the laser stripe center. Manhattan distance represents the arithmetic sum of the horizontal and vertical coordinates of two points in the coordinate axis. ci ,y ci ) and the i+1th point (x i+1 ,y i+1 ) is the Manhattan distance L between them, as shown in the formula,

[0084] L=|x i+1 -x ci |+|y i+1 -y ci |

[0085] If L is less than or equal to 4, the new laser stripe center point is considered valid, and the new laser stripe center point is used to replace and update the second point in the original laser stripe skeleton; if L is greater than 4, the new laser stripe center point is considered invalid, and the second point in the original laser stripe skeleton is retained.

[0086] According to the above calculation process, all the skeleton points of the laser stripe center line are traversed, and the third point, the fourth point, ... the nth point in the skeleton of the laser stripe center line are calculated respectively until all the center points of the laser stripe are extracted. These center points constitute the center line of the laser stripe, such as Figure 5 .

[0087] The center line of the laser stripe of the conveyor belt contour in the unloaded image of the belt conveyor and the center line of the laser stripe of the material in the loaded image of the belt conveyor are superimposed to obtain a closed curve. The figure enclosed by the closed curve represents the instantaneous cross-sectional shape formed by the material on the conveyor belt at a certain moment, such as the position of S in the figure. For each point on the closed curve, the relative height ΔH of the material relative to the conveyor belt surface is calculated i , and the material surface height is ΔH i The actual width of each pixel in the horizontal direction corresponding to Δx i , then according to the formula, S = ∑(Δx i ×ΔH i )

[0088] Calculate the cross-sectional area of ​​S for the figure enclosed by the closed curve, such as Figure 6 .

[0089] It is known that the running speed of the conveyor belt is v and the sampling frame rate of the camera is f c Assuming that the instantaneous cross-sectional area of ​​the material flowing through the conveyor belt does not change during the single-frame sampling time, and the instantaneous cross-sectional area is regarded as the infinitesimal volume component of the material in the direction of travel, the volume V of the material flowing through the conveyor belt during the single-frame sampling time is:

[0090] V=[∑(Δx i ×ΔH i )]×v / f c

[0091] According to the density ρ of the transported material, calculate the instantaneous material quantity Q of the belt conveyor within the single frame sampling time S :

[0092] Q s =ρ×v×∑(Δx i ×ΔH i )

[0093] In the actual operation of the belt conveyor, the instantaneous material volume fluctuates greatly. Usually, the n instantaneous material volume Q within a sampling time isS Take the average value as the value of the material quantity Q of the belt conveyor.

[0094]

[0095] When a belt conveyor is performing speed control, a key issue is how to determine the appropriate conveyor belt speed based on the material quantity detection results, so as to ensure that the belt conveyor is operated at full load as much as possible to achieve optimized energy-saving control. Considering the dynamic characteristics of the belt conveyor, it is not suitable to perform stepless speed regulation. In order to avoid material quantity disturbance and ensure the stability of the system, the material quantity can be divided into multiple intervals to construct a belt conveyor energy-saving optimization model.

[0096] The rated flow rate of the belt conveyor is Q = 2000t / h, the rated speed is v = 3m / s, and the maximum load line density of the conveyor belt is

[0097] q m =175kg / m. The material volume increment ΔQ=200t / h is used for statistics to determine the corresponding conveyor belt running speed under different flow rates. At the same time, in order to avoid frequent start and stop of the belt conveyor, when the material volume is lower than Q min =500t / h, the conveyor belt runs at the designed minimum belt speed of 0.8m / s. Figure 7 Energy-saving optimization model for belt conveyor.

[0098] The main control program is responsible for realizing the centralized control of the entire system. After completing the controller initialization, data collection is performed, and then the belt conveyor and other detection devices are started in sequence, and adaptive speed regulation is performed according to the control instructions. The process is as follows: Figure 8 shown.

[0099] In the actual production process, the material quantity is constantly changing within the limit. In order to avoid frequent changes in the conveyor belt speed, the material quantity is divided into n intervals [Q 0 ,Q 1 ]、[Q 1 ,Q 2 ]…[Q n-1 ,Q n ], the corresponding conveyor belt running speeds are v 0 、v 1 …v n-1 The adaptive speed control subroutine adjusts the speed of the belt conveyor according to the detection results of the material quantity and the range to which it belongs. The flow chart is as follows: Fig. 9 shown.

[0100] The method of fusing visible light photography and thermal imaging is used to obtain images. When there is sufficient light, a visible light camera is used to collect images. When the environment is dim, different weighting coefficients are used to fuse thermal imaging images and visible light images according to the light intensity of the environment.

[0101] The real-time acquired thermal imaging modality images and visible light modality images are converted into grayscale images and then used to 1 , X 2 Indicates that the size is W×H. X is X 1 , X 2 Image after weighted fusion. X 1 , X 2 The weighted fusion process of two original images is as follows:

[0102] X(w,h)=k 1 X 1 (w,h)+k 2 X 2 (w,h)

[0103] Where: w is the row number of the image pixel, w = 1, 2, ..., w; h is the column number of the image pixel, h = 1, 2, ..., h; k 1 , k 2 is the weighted coefficient of image fusion at that moment,

[0104]

[0105] Where: L t is the real-time light intensity of the workshop environment, L h and L d They are the upper and lower thresholds of the light intensity in the image fusion process. By comparing the light intensity at the time of image acquisition with the set threshold, the weighting coefficient of the image is determined and the fused image at that moment is output.

[0106] The image with clear outline of the person is used as the model input, and the large 7×7 convolution kernel in the original network is replaced with three 3×3 convolution kernels. The features obtained are used as the input of the OpenPose model. After double-branch multi-level iterative network processing, key points and skeleton graphs can be generated for the person in each frame of the image. The lightweight model after cropping can accurately identify the 19 joints in the human skeleton sequence. The formula for generating the confidence map of the human joints in each frame of the image is shown as follows:

[0107]

[0108] Where: m is the character number, n is the key point of the human body, Q is the confidence map C n,m Any point in X n,m is the real position of the key points of the human body, σ n,mIt is used to describe the probability distribution of corresponding key points and predict the degree of association between two key points through partial affinity fields. It can be expressed by the integral value H as:

[0109]

[0110] Q(t)=(1-t)n 2 +tn 1 , t∈(0,1)

[0111] Where: n 1 、n 2 are adjacent key points, ||n 2 -n 1 ||2 is the length of the limb, Q(t)∈[n 1 ,n 2 ] If point Q is on the limb, then kb(Q(t)) = v, where v = (m 2 -m 1 ) / ||m 2 -m 1 ||2 is a unit vector, otherwise, k b (Q(t))=0.

[0112] The correct connection of each type of limb is obtained by selecting the maximum value of H, and then the limbs that share the same key points are connected into a human skeleton.

[0113] The workshop environment is complex. Workers are easily blocked by equipment and goods during work. Only part of the body features can be detected during work, which affects the accuracy of the detection results. The blocked time frame F(s) and the marked adjacent frames F(j) and F(l) have rich time clues. Therefore, for the partially blocked person i, the time frame merging result F i (j,s,l) is:

[0114]

[0115] Where j, s, and l are time frame indices. For each joint point, specific time information can be specified to calculate and merge into a key point heat map, which can be stacked into a feature tensor φi after regrouping, expressed as:

[0116]

[0117] in: For the connection frame operation, N is the total number of joints, and n is the joint number. This not only eliminates the interference of irrelevant joints and removes algorithm redundancy, but also estimates the position of the joint points when they are occluded based on the time clues. The different feature tensors obtained are then input into the residual fusion module. Similar to the key point time frame merging, the pose residual ψi is calculated through pose residual fusion as another effective pose estimation clue. The implementation process is as follows:

[0118] ψ i (j,s)=F i (s)-F i (j)

[0119] ψ i (s,l)=F i (l)-F i (s),

[0120]

[0121] By connecting the original residuals ψi(j,s), ψi(s,l) and their corresponding time weights, and processing them through a 3×3 residual block, the final position residual estimate can be obtained. This allows the model to detect accurate human key points and skeleton position features in visible light images and thermal imager images, and label the key points of body parts as follows Fig.10 .

[0122] The pseudo-fall action can be understood as an active fall action, and the descending speed of the knee point is always kept within a certain safety range. However, when a worker falls accidentally, he is in a passive state, and the descending speed of his knee point will suddenly change in a short period of time, thus exceeding the safe speed range. Therefore, α=0.1m / s is selected as the threshold for distinguishing falls and other pseudo-fall actions. During the discrimination process, the knee point movement speed V between adjacent frames is calculated and compared with the threshold to achieve discrimination. The knee point movement speed calculation formula is:

[0123]

[0124] The key point coordinates can be obtained by extracting the human skeleton, where the coordinates of the left knee point are (X 13 ,Y 13 ), the coordinates of the right knee point are (X 10 ,Y 10 ), so in the formula: Y 10 Y′ is the ordinate of the right knee point in the current frame; 10 Y is the vertical coordinate of the right knee point in the previous frame; 13 Y′ is the ordinate of the left knee point in the current frame; 13is the ordinate of the left knee point in the previous frame; T is the current time; T′ is the time of the previous frame; when Φ is true, it means that at least one knee point has a sudden change in descent speed. If there is a sudden change in 3 consecutive frames, it is judged as a fall action, otherwise, it is a non-fall action.

[0125] Distinguishing pseudo-fall actions only by calculating the descending speed of the knee point is prone to misjudgment, because the sudden change in the moving speed of the knee point can be actively initiated by the human body, such as squatting. It is impossible to accurately judge the state of a person only by relying on the constraints of the time domain, so the constraints of the spatial domain are proposed as the basis for distinguishing human falls. When workers are working in factory workshops, the aspect ratio of the human body's circumscribed rectangle and the inclination angle of the body axis for actions such as standing and squatting are greatly different from the state of falling; therefore, according to the physical properties satisfied when the body is in different states, the present invention proposes the inclination angle θ of the human body's circumscribed rectangle and the aspect ratio R of the human body's circumscribed rectangle as spatial constraints.

[0126] In order to more accurately mark the circumscribed rectangular frame of the human body, the maximum value of the key point coordinates X max , Y max and minimum value X min , Y min Construct a rectangular frame circumscribing the human body, and obtain the aspect ratio R of the rectangular frame circumscribing the human body:

[0127]

[0128] By selecting the neck keypoint (X 1 ,Y 1 ), center point (X 8 ,Y 8 ) generates a vector representing the position and direction of the human body axis As shown below:

[0129]

[0130] Unit vector between the body axis and the horizontal ground The angle formed can be expressed as:

[0131]

[0132] When R is less than the threshold value of 0.8 and the human body inclination angle satisfies 63°≤θ≤117°, the action is judged as non-falling. If this condition is not met, it is judged as falling.

[0133] like Fig.11The video acquisition module is used to access the monitoring video line. The detection module uses the human posture estimation algorithm to identify the key points of the human body in the video and detect whether a fall has occurred. The detection information module displays the number of people in the current video and the safety status of the people in the video, that is, whether a fall has occurred. At the same time, when a fall occurs, the alarm information is connected to the digital twin control system in the main control room and sent in the form of alarm sound and text message to remind the on-duty personnel in the main control room and notify the relevant supervisors by text message. The alarm information includes the video screenshot of the fall and the location information of the incident.

[0134] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0135] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent belt conveyor monitoring system, characterized in that: The system includes a laser stripe detection device, a camera, a controller, a computing device and an adaptive speed control subroutine. The system can achieve the following functions: The pixel point information of the center line skeleton of the laser stripe is obtained by the laser stripe detection device. The steps of obtaining the pixel point information of the center line skeleton of the laser stripe by the laser stripe detection device are as follows: setting the region of interest to obtain the light stripe image, using bilateral filtering to add a pixel value weight term on the basis of Gaussian filtering, and expressing the pixel value weight as G r , the spatial distance weight is represented by G s , then the weight BF calculation method of a point p in the bilateral filtering process in the template window centered on point q is as follows: W q represents a two-dimensional convolutional template, I p Represents the gray value of point p, integrates the gray information of all pixels in the image, realizes the binary segmentation of the image by setting a reasonable threshold, converts the laser stripes of a certain width in the material image into the center line of the laser stripes that can represent the real center position, and uses linear interpolation to quickly repair the center line of the laser stripes; regards the geometric center point on each longitudinal section of the laser stripes as the center point of the laser stripes, C1 is the ordinate of the upper edge point of a column in the laser stripes, and C2 is the ordinate of the lower edge point of a column in the laser stripes, then the ordinate C of the center position of the laser stripes in the column is: The controller selects the initial i-th point and i+1-th point from the laser stripe centerline skeleton and calculates the normal direction of the line between the two points. The steps of calculating the normal direction of the line between the two points are as follows: the laser stripe centerline skeleton consists of n pixels in total, and the initial i-th point (x i ,y i ) and the i+1th point (x i+1 ,y i+1 ) and calculate the normal direction of the connected straight line. According to the grayscale information of the laser stripe at the normal direction of the i+1th point, the new center point of the laser stripe (x ci ,y ci ), the normal direction m of the straight line connecting the two points is calculated according to the slope k between the two points as follows: The controller introduces the Manhattan distance to judge the validity of the new laser stripe center point and optimizes the extraction result of the laser stripe center; the Manhattan distance represents the arithmetic sum of the horizontal and vertical coordinates of two points in the coordinate axis, and calculates the new laser stripe center point (x ci ,y ci ) and the i+1th point (x i+1 ,y i+1 ) between the Manhattan distance L, the formula is as follows, L = |x i+1 -x ci |+|y i+1 -y ci | If L is less than or equal to 4, the new laser stripe center point is considered valid, and the new laser stripe center point is used to replace and update the second point in the original laser stripe skeleton; if L is greater than 4, the new laser stripe center point is considered invalid, and the second point in the original laser stripe skeleton is retained; The controller traverses all the laser stripe centerline skeleton points until all the laser stripe center points are extracted; The computing device superimposes the center line of the laser stripe of the conveyor belt contour in the unloaded image of the belt conveyor and the center line of the laser stripe of the material in the loaded image of the belt conveyor to obtain a closed curve, and calculates the instantaneous cross-sectional area S enclosed by the closed curve; The computing device calculates the volume V of the material flowing through and the instantaneous material quantity Q within a single frame sampling time according to the sampling frame rate of the camera and the running speed of the belt conveyor. S , the computing device takes the average value of the instantaneous material quantity within multiple sampling times to obtain the material quantity Q of the belt conveyor; The controller determines the appropriate conveyor belt running speed according to the material quantity detection result and the belt conveyor energy-saving optimization model, and realizes the adaptive speed control of the belt conveyor through the adaptive speed control subroutine.

2. An intelligent belt conveyor monitoring system according to claim 1, characterized in that: The region of interest is a rectangle, and the horizontal length of the region of interest should be consistent with the effective length of the laser stripes on the conveyor belt. The lowest point in the vertical direction of the region of interest should include the laser stripes at the lowest point of the surface when the conveyor belt is unloaded, and the highest point in the vertical direction of the region of interest should include the laser stripes on the material surface at the highest point when the conveyor belt is fully loaded.

3. An intelligent belt conveyor monitoring system according to claim 1, characterized in that: The center line of the laser stripe of the conveyor belt contour in the unloaded image of the belt conveyor and the center line of the laser stripe of the material in the loaded image of the belt conveyor are superimposed to obtain a closed curve. The figure enclosed by the closed curve represents the instantaneous cross-sectional area S formed by the material on the conveyor belt at a certain moment. For each point on the closed curve, the relative height ΔH of the material relative to the conveyor belt surface is calculated. i , and the actual width Δx of each pixel in the horizontal direction corresponding to the height ΔHi of the material surface i , then according to the formula, S = ∑(Δx i ×ΔH i ) calculates the instantaneous cross-sectional area S enclosed by the closed curve; the running speed of the conveyor belt is v, and the sampling frame rate of the camera is f c , the instantaneous cross-sectional area is regarded as the volume component of the material in the direction of travel, then the volume V of the material flowing through the single frame sampling time is: V = [∑ (Δx i ×ΔH i )]×v / f c ; According to the density ρ of the transported material, calculate the instantaneous material quantity Q of the belt conveyor within the single frame sampling time S :Qs=ρ×v×∑(Δx i ×ΔH i ); The calculation formula for the material quantity Q of the belt conveyor is:

4. An intelligent belt conveyor monitoring system according to claim 1, characterized in that: According to the belt conveyor energy-saving optimization model, the material volume is divided into n intervals [Q0,Q1], [Q1,Q2]…[Q n-1 ,Q n ], the corresponding conveyor belt running speeds are v0, v1…v n-1 .

5. An intelligent belt conveyor monitoring system according to claim 1, characterized in that: It also includes a fall detection subroutine, which consists of the following modules: a. Video acquisition module, which uses the method of fusion of visible light camera and thermal imaging camera to acquire images, and is used to collect monitoring videos of the working area of ​​the belt conveyor; b. A detection module, which uses a human posture estimation algorithm to identify key points of the human body in the video and detects whether a fall event occurs based on the key points; wherein the human posture estimation algorithm includes: Using confidence map C n,m Describes the probability distribution of key points of the human body, where X n,m is the true position of the key points of the human body, σ n,m Describe the probability distribution of corresponding key points; The degree of association between two key points is predicted using the partial affinity field, the correct connection of each type of limb is determined by the integral value H, and the limbs that share the same key points are connected to form a human skeleton; When the human body is partially occluded, the position of the joint points when it is occluded is estimated based on the time clues, and the feature tensor φ is constructed i ; The pose residual ψ is calculated through the residual fusion module i , as clues for pose estimation; The threshold α of the knee point descent speed is set to distinguish between falling actions and pseudo-falling actions; The human body axis inclination angle θ and the height-to-width ratio R of the human body circumscribed rectangular frame are introduced as spatial constraints to further confirm the fall event. c. Detection information module, used to display the number of people in the current video, the safety status of the characters, and trigger an alarm when a fall event is detected; d. Alarm information module. When the detection module detects a fall event, the module sends the video screenshot of the fall and the location information of the incident to the digital twin control system in the main control room, and notifies relevant personnel in the form of alarm sounds and text messages.

6. An intelligent belt conveyor monitoring system according to claim 5, characterized in that: In the detection module, an image with a clear human outline is used as the model input, and the large 7×7 convolution kernel in the original network is replaced with three 3×3 convolution kernels. The features obtained are used as the input of the OpenPose model. After double-branch multi-level iterative network processing, key points and skeleton maps can be generated for the characters in each frame of the image. The lightweight model after cropping can accurately identify the 19 joints in the human skeleton sequence. The formula for generating the confidence map of the human joints in each frame of the image is shown as follows: Where: m is the character number, n is the key point of the human body, Q is the confidence map C n,m Any point in X n,m is the real position of the key points of the human body, σ n,m Used to describe the probability distribution of corresponding key points.

7. An intelligent belt conveyor monitoring system according to claim 5, characterized in that: In the detection module, the correlation between two key points is predicted by partial affinity fields, which can be expressed by the integral value H as follows: Q(t)=(1-t)n2+tn1,t∈(0,1) Among them: n1, n2 are adjacent key points, ||n2-n1||2 is the length of the limb, Q(t)∈[n1,n2], if point Q is on the limb, then k b (Q(t))=v, where v=(m2-m1) / ||m2-m1||2 is a unit vector, otherwise, k b (Q(t))=0; the correct connection of each type of limb is obtained by selecting the maximum value of H, and then the limbs that share the same key points are connected into a human skeleton.

8. An intelligent belt conveyor monitoring system according to claim 5, characterized in that: In the detection module, there are rich time clues between the blocked time frame F(s) and the marked adjacent frames F(j) and F(l), so the time frame merging result F of the partially blocked person i is i (j,s,l) is: Where j, s, and l are time frame indices. For each joint point, specific time information can be specified and calculated and merged into a key point heat map, which can be stacked into a feature tensor φ after regrouping. i , expressed as: Among them: ⊕ is the connection frame operation, N is the total number of joints, and n is the joint number.

9. An intelligent belt conveyor monitoring system according to claim 5, characterized in that: In the detection module, the obtained different feature tensors are input into the residual fusion module. Similar to the key point time frame merging, the pose residual ψi is calculated by pose residual fusion as another effective pose estimation clue. The implementation process is as follows: ψ i (j,8)=F i (s)-F i (j), ψ i (s,l)=F i (l)-F i (s), By concatenating the original residual ψ i (j,s),ψ i (s,l) and its corresponding time weights, and processed by a 3×3 residual block, the final position residual estimate can be obtained, so that the model can detect the accurate human key points and skeleton position features in visible light images and thermal imager images, and label the key points of body parts.

10. An intelligent belt conveyor monitoring system according to claim 5, characterized in that: In the detection module, α=0.1 m / s is selected as the threshold for distinguishing falls and other pseudo-falls. During the distinguishing process, the knee point moving speed V between adjacent frames is calculated and compared with the threshold to achieve the distinguishing. The knee point moving speed calculation formula is: The key point coordinates can be obtained by extracting the human skeleton, where the coordinates of the left knee point are (X 13 ,Y 13 ), the coordinates of the right knee point are (X 10 ,Y 10 ), so in the formula: Y 10 Y is the ordinate of the right knee point in the current frame; ′ 10 Y is the vertical coordinate of the right knee point in the previous frame; 13 Y is the ordinate of the left knee point in the current frame; ′ 13 is the ordinate of the left knee point in the previous frame; T is the current time; T′ is the time of the previous frame; when Φ is true, it means that at least one knee point has a sudden change in descent speed. If there is a sudden change in three consecutive frames, it is determined to be a fall action, otherwise, it is a non-fall action; The human body axis inclination angle θ and the height-to-width ratio R of the human body circumscribed rectangular frame are used as spatial constraints, and the maximum value of the key point coordinates X is selected. max , Y max and minimum value X min , Y min Construct a rectangular frame circumscribing the human body, and obtain the aspect ratio R of the rectangular frame circumscribing the human body: Generate a vector representing the position and direction of the human body axis by selecting the key points of the neck (X1, Y1) and the center point (X8, Y8) As shown below: The angle between the human body axis and the unit vector of the horizontal ground can be expressed as: When R is less than the threshold value of 0.8 and the human body inclination angle satisfies 63°≤θ≤117°, the action is judged as non-falling. If this condition is not met, it is judged as falling.

Images