Image Processing Method for Construction Vehicles and Personnel at Electric Power Infrastructure Sites Using Monocular Vision

Abstract

The present invention proposes a method for processing images of construction vehicles and personnel at the power infrastructure construction site using monocular vision, which is used to realize the prediction of the trajectories of construction vehicles and personnel relying only on a single camera in a complex power infrastructure construction site environment, and to judge the estimated intersection probability of vehicles and personnel, so as to avoid the occurrence of intersection accidents between construction vehicles and personnel. The method includes: collecting image data of the trajectories of construction vehicles and personnel at the power infrastructure construction site using a monocular camera and performing image processing using various algorithms, predicting the probabilities of multiple future travel trajectories of construction vehicles and operators using the Kalman filter enhancement algorithm, and then calculating the probability of intersection with the other party under each path by the Bayesian probability model, warning and reminding the paths that exceed the probability threshold, and reducing the possibility of intersection between construction vehicles and personnel.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle and personnel trajectory intersection probability prediction, in particular to an image processing method for construction vehicles and personnel at the power infrastructure construction site using monocular vision. Background Art

[0002] Due to the influence of terrain and various building obstructions at the infrastructure construction site, there will be visual blind spots when observing the relative positions between personnel and construction machinery by the naked eye traditionally, and some potential safety hazards cannot be predicted in advance. Even with the use of positioning sensors, there is still a lack of a solution that can effectively locate the mechanical operating arm at the present stage. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide an image processing method for construction vehicles and personnel at the power infrastructure construction site using monocular vision, realizing visible positions of on-site personnel and machinery, controllable behaviors, and predictable trends, changing "post-event alarm" to "pre-event warning", and improving the management and control level of the infrastructure construction site by intelligent means.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions: An image processing method for construction vehicles and personnel at the power infrastructure construction site using monocular vision, including the following steps:

[0005] Collect image data of construction vehicles and personnel at the power infrastructure construction site using a monocular camera;

[0006] Denoise the trained image samples using the median denoising method to filter out noises such as salt-and-pepper noise in the images;

[0007] Perform deep learning training on the power infrastructure construction site image data using the YoloX object detection algorithm, and obtain the approximate positions of construction vehicles and personnel respectively;

[0008] Perform edge detection on the denoised image samples using algorithms such as the Canny edge detection algorithm, LSD line detection algorithm, and SLIC image superpixel segmentation algorithm respectively, and obtain the edge detection results under different algorithms;

[0009] Set a confidence level to fuse different edge detection results to obtain the final recognition models of construction vehicles and personnel.

[0010] In a preferred embodiment, use the trained model to detect moving construction vehicles and on-site personnel in the construction site images detected by the monocular camera;

[0011] Define corresponding identification numbers, coordinates, identification frame sizes, and speeds of the targets for the identified construction vehicles and on-site personnel respectively;

[0012] Use the Kalman filter enhancement algorithm to predict the size and coordinate position of the recognition frame for the next frame of construction vehicles and operators respectively, and visualize the recognition frame;

[0013] Use the augmenting path matching algorithm to match the current frame target group with the union of the previous frame target group and the target group that failed to match in the previous frame, and finally obtain the maximum match; output the coordinate status information for the successfully matched targets.

[0014] In a preferred embodiment, the specific formula of the Kalman filter enhancement algorithm model is as follows:

[0015] X p+1 =F p X p +G p W p

[0016] Where X p represents the recognition target motion state vector, that is, the set of the speed and position of the object in the horizontal and vertical directions, the size of the recognition frame, and the ratio of the height to the width of the recognition frame, F p represents the object state transition matrix, G p represents the input matrix, W p represents the system input;

[0017] F p The specific calculation formula is as follows:

[0018]

[0019] Among them, E represents the identity matrix, and △T represents the time difference between a certain frame and the previous frame;

[0020] Furthermore, the prior covariance matrix R p+1|p can be obtained from the posterior covariance matrix and the process noise matrix. The specific calculation formula is as follows:

[0021]

[0022] Among them, E 7×7 is the process noise covariance matrix.

[0023] In a preferred embodiment, the augmenting path matching algorithm can match the current frame target group with the union of the previous frame target group and the target group that failed to match in the previous frame. The specific steps are as follows:

[0024] Initialize the bipartite graph, that is, confirm the recognition frames of the previous frame that the current frame recognition frame may match;

[0025] Match according to the defined recognition number order above. When the previous frame target may match the current target, perform a match first;

[0026] If the matching result of the current frame target with a later order is repeated with the matching result of the current frame target with an earlier order, the matching target of the latter is replaced recursively;

[0027] If the current frame target fails to match successfully through the above method, the target is regarded as a newly emerged target;

[0028] Repeat the above steps to make the previous frame target match the current frame target one by one as much as possible, and output the coordinate status information of the successfully matched targets.

[0029] In a preferred embodiment, the power infrastructure construction site is divided into an n*n grid, and all trajectory coordinates should fall into the corresponding grids;

[0030] Using the Bayesian probability estimation formula, calculate the prior probability of each "next trajectory point" appearing in a certain grid based on a large amount of historical trajectory data of construction vehicles and operators at the power infrastructure construction site;

[0031] Put the "next frame trajectory" of the construction vehicle and the operator predicted by the above Kalman filter enhanced algorithm into the Bayesian probability calculation formula, and calculate the trajectory prediction likelihood probabilities of the two respectively;

[0032] If the prediction result of the "next frame trajectory" is the same as the actual result, that is, both are in the same grid, the posterior probability should be recalculated using the Bayesian rule to update the probability model;

[0033] If the "next frame trajectory" of the construction vehicle and the operator predicted by the above Kalman filter enhanced algorithm fall into the same grid, multiply the probabilities of the two to calculate the probability of the intersection of the two trajectories;

[0034] If the intersection probability is less than a certain threshold, the system gives an early warning in time to prevent the construction vehicle and the operator from intersecting at the power infrastructure construction site, and timely reduce the safety hazards at the power infrastructure construction site.

[0035] In a preferred embodiment, calculate the likelihood probability of each "next trajectory point" appearing in a certain grid using a large amount of historical trajectory data of construction vehicles and operators at the power infrastructure construction site. The calculation formula is as follows:

[0036]

[0037] Among them, P(A|B) represents the probability that the next frame trajectory point is in a certain grid when the grid where the previous frame trajectory point is located is known, P(A,B) represents the probability that the upper and lower frames of a certain trajectory are in the known grid, and P(B) represents the probability that the previous frame of a certain trajectory is in the known grid;

[0038] The "next-frame trajectory" of construction vehicles and operators predicted by the Kalman filter enhancement algorithm is input into the Bayesian probability calculation formula to calculate the trajectory prediction probabilities of the two respectively. The specific calculation formula is as follows:

[0039]

[0040] Among them, P(A|B) represents the posterior probability that the next-frame trajectory point is in a certain grid when the grid where the previous-frame trajectory point is located is known, P(B|A) is the likelihood probability, P(A) is the prior distribution, and P(B) is the normalization operator.

[0041] Compared with the prior art, the present invention has the following beneficial effects: The present invention is a method for predicting the intersection of the trajectories of construction vehicles and personnel at the power infrastructure construction site using monocular vision. Through steps such as image processing, trajectory prediction, and probability calculation, it accurately predicts the reliable position information and intersection probability of personnel and machinery in a future period of time, greatly avoiding the occurrence of intersection accidents between construction vehicles and operators at the power infrastructure construction site, realizing visible positions, controllable behaviors, and predictable trends of on-site personnel and machinery, and improving the management and control level of the construction site by means of intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic flow chart of a method for image processing and training of construction vehicles and personnel at the power infrastructure construction site using monocular vision in an embodiment of the present invention;

[0043] Figure 2 It is a schematic flow chart of a method for predicting the future traveling trajectories of construction vehicles and operators based on the Kalman filter enhancement algorithm in an embodiment of the present invention;

[0044] Figure 3 It is a schematic flow chart of a method for calculating and warning the intersection probability of the trajectories of construction vehicles and operators at the power infrastructure construction site based on Bayesian probability estimation in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0045] The present invention will be further described below in conjunction with the drawings and embodiments.

[0046] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs.

[0047] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0048] Referring to Figure 1 , Figure 1 This is a method for image processing and training of construction vehicles and personnel in the power infrastructure construction site using monocular vision in an embodiment of the present invention. The image processing and training method provided in this embodiment includes the following steps:

[0049] Step S10, the monocular vision is to collect image data of construction vehicles and personnel in the power infrastructure construction site using a monocular camera;

[0050] It should be noted that the trained images mainly have noises such as salt-and-pepper noise, and the salt-and-pepper noise easily changes the gray value of the affected pixel points to 255 or 0, that is, it turns into dark points or bright points. Therefore, it is necessary to denoise to weaken the above influence;

[0051] Step S11, in this embodiment, the median denoising method is used to denoise the trained image, and the specific steps are as follows:

[0052] Set a movable frame to move within the image range to ensure that the center of the movable frame coincides with each pixel point of the image;

[0053] Record the gray values of the corresponding pixel points under the movable frame at each pixel point, and sort the gray values in order;

[0054] Record the median value of the gray value sequence, and replace the gray value of the pixel point under the center of the movable frame with the median value to complete denoising.

[0055] Step S12, preferably, use the YoloX object detection algorithm to train the image data and obtain the approximate positions of construction vehicles and operating personnel in the power infrastructure construction site;

[0056] Furthermore, use the SLIC image superpixel segmentation algorithm, LSD line detection algorithm, and Canny edge detection algorithm to optimize the denoised target image respectively;

[0057] Step S13, the SLIC image superpixel segmentation algorithm can obtain the optimal image boundary by clustering color pixels, and the specific steps are as follows:

[0058] Randomly set K superpixel centers among the pixel points of the denoised target image;

[0059] It should be noted that the center should be adjusted to the minimum gradient point within a certain range of each superpixel center to prevent the superpixel point from coinciding with noise or boundaries;

[0060] Initialize the superpixel centers of all pixel points in the target image, and calculate the distances from the initialized pixel points to their respective superpixel centers;

[0061] For each superpixel center, calculate the distances between the pixel points within its range and other superpixel centers. If the above distances are less than the distance between the pixel point and the original superpixel center, then assign the pixel point to the new superpixel center. After traversing all pixel points, recalculate the positions of each superpixel center;

[0062] Repeat the above steps until each superpixel center is located exactly in the middle of all pixel points.

[0063] Step S14, the LSD line detection algorithm can obtain the local line contours of the image, and its specific steps are as follows:

[0064] Perform Gaussian blur on the denoised target image and then downsample it to reduce the aliasing effect;

[0065] Calculate the gradient values and gradient directions of each pixel point. The specific calculation formulas are as follows:

[0066]

[0067] Where G is the gradient value, LLA is the horizontal angle, and g is the gradient of the point in two directions. The specific calculation formulas are as follows:

[0068]

[0069]

[0070] Where i(a, b) is the gray value of the pixel point (a, b) on the gray image;

[0071] Perform pseudo-sorting on each pixel point according to the gradient values of each pixel point, that is, divide the gradients into 1024 blocks, and classify each pixel point into each block according to the gradient values;

[0072] Take the pixel point with the maximum gradient as the starting point, search for the pixel points within a certain range around it with gradient values greater than ρ, and generate a rectangular frame R containing each satisfied point;

[0073] Furthermore, determine whether the rectangular frame R reaches the density threshold. If not, then cut the rectangular frame R until the density threshold is satisfied;

[0074] It should be noted that the false alarm times of each of the rectangular frames should be calculated to judge the gradient consistency within the rectangular frame. The specific calculation formula is as follows:

[0075]

[0076] Where N is the number of rectangular frames in the target image with size x*y, and n is the total number of pixels of the in-class points in each rectangular frame;

[0077] Furthermore, if the false alarm times value is less than the threshold, the rectangular frame R will be output.

[0078] Step S15, the Canny edge detection algorithm can obtain the optimal boundary of the target image. The specific steps are as follows:

[0079] Use a Gaussian filter to smooth the denoised target image;

[0080] Calculate the gradient value and gradient direction of each pixel point. The specific calculation formula is the same as the above LSD line detection algorithm;

[0081] Perform non-maximum suppression on the gradient values of the pixel points, that is, within a 3*3 moving range, compare the gradient of the specified pixel point with the gradients of the two pixel points in the gradient direction. If the former gradient value is the smallest, assign the gradient value of the specified pixel point to 0, otherwise the gradient value remains unchanged;

[0082] Use the double critical point algorithm to identify the boundary, that is, set high and low critical pixel points. If the pixel point in the target image is greater than the highest value, it is determined to be a boundary, and if it is lower than the lowest value, it is determined not to be a boundary;

[0083] It should be noted that if the selected pixel point is between the high and low critical values, use the neighboring pixel points of this pixel point to determine whether it is on the boundary in the above manner.

[0084] Step S16, set the detection confidence according to the above edge detection algorithms, and finally perform confidence fusion on the three edge detection results to obtain the final recognition results of the construction vehicles and operating personnel at the power infrastructure construction site.

[0085] Refer to Figure 2 , Figure 2 is a schematic flow chart of a method for predicting the future traveling trajectories of construction vehicles and operating personnel based on the Kalman filter enhancement algorithm in an embodiment of the present invention. The future traveling trajectory prediction method provided in this embodiment includes the following steps:

[0086] Step S20, send the image data of the power infrastructure construction site obtained by the monocular camera into the above image processing and training model for training, and output the recognition results;

[0087] Step S21 defines corresponding identification numbers, coordinates, identification frame sizes, and target speeds for the identified construction vehicles and on-site personnel respectively, and adds each identified target to the detection set;

[0088] In step S22, the Kalman filter enhancement algorithm is used to predict the position and speed of the currently identified objects in the next frame and output the results;

[0089] It should be noted that the Kalman filter enhancement algorithm is a self-regressive optimization algorithm for solving the state estimation of nonlinear systems, and has high prediction accuracy and significant prediction effects for the trajectory prediction of dynamic targets;

[0090] The specific formula of the Kalman filter enhancement algorithm model is as follows:

[0091] X p+1 =F p X p +G p W p

[0092] Where X p represents the recognition target motion state vector, that is, the set of the speed and position of the object in the horizontal and vertical directions, the size of the recognition frame, and the ratio of the height to the width of the recognition frame, F p represents the object state transition matrix, G p represents the input matrix, W p represents the system input.

[0093] F p The specific calculation formula is as follows:

[0094]

[0095] Among them, E represents the identity matrix, and △T represents the time difference between a certain frame and the previous frame;

[0096] Furthermore, the prior covariance matrix R p+1|p can be obtained through the posterior covariance matrix and the process noise matrix. The specific calculation formula is as follows:

[0097]

[0098] Where E 7×7 is the process noise covariance matrix.

[0099] In step S23, the augmented path matching algorithm is used to match the current frame target group with the union of the previous frame target group and the target group that failed to match in the previous frame. The specific steps are as follows:

[0100] Initialize the bipartite graph, that is, confirm the recognition frames of the previous frame that the recognition frames of the current frame may match;

[0101] Match according to the defined recognition number order above. When the target in the previous frame may match the target at this moment, perform a match first;

[0102] If the matching result of the target in the current frame with a later order and the target in the previous frame is the same as the matching result of the target in the current frame with an earlier order, recursively replace the latter matching target;

[0103] If the target in the current frame fails to match successfully through the above method, regard this target as a newly emerged target;

[0104] Repeat the above steps to make the target in the previous frame match the target in the current frame one by one as much as possible, and output the coordinate status information of the successfully matched targets.

[0105] It should be noted that when dividing the target group into two parts, a relatively high confidence level should be ensured, otherwise normal matching will not be possible.

[0106] Refer to Figure 3 , Figure 3 , which is a schematic flow chart of a method for calculating and warning the intersection probability of the trajectories of construction vehicles and operating personnel at the power infrastructure construction site based on Bayesian probability estimation in the embodiment of the present invention. The method for calculating and warning the intersection probability provided in this embodiment includes the following steps:

[0107] Step S30: Divide the power infrastructure construction site into an n*n grid, and all trajectory coordinates should fall into the corresponding grids;

[0108] It should be noted that considering the relatively large area of the power infrastructure construction site in this embodiment, the power infrastructure construction site is divided into a 9*9 square grid in this embodiment.

[0109] Step S31: Calculate the likelihood probability that each "next trajectory point" appears in a certain grid by using a large amount of historical trajectory data of construction vehicles and operating personnel at the power infrastructure construction site. The calculation formula is as follows:

[0110]

[0111] Among them, P(A|B) represents the probability that the next frame trajectory point is in a certain grid when the grid where the previous frame trajectory point is located is known, P(A,B) represents the probability that the upper and lower frames of a certain trajectory are in the known grids, and P(B) represents the probability that the previous frame of a certain trajectory is in the known grid.

[0112] Step S32: Input the "next frame trajectory" of the construction vehicle and the operating personnel predicted by the above Kalman filter enhancement algorithm into the Bayesian probability calculation formula, and calculate the trajectory prediction probabilities of the two respectively. The specific calculation formula is as follows:

[0113]

[0114] Among them, P(A|B) represents the posterior probability that the next-frame trajectory point is in a certain grid given the grid where the previous-frame trajectory point is located. P(B|A) is the likelihood probability, P(A) is the prior distribution, and P(B) is the normalization operator.

[0115] It should be noted that the posterior probability is a continuously updated value. By using Bayes' rule and combining historical and real-time data, the value of the posterior probability can be continuously corrected in a closed loop to ensure a higher accuracy of probability calculation.

[0116] Step S33. Further, if the prediction result of the "next-frame trajectory" is the same as the actual result, that is, both are in the same grid, then the posterior probability can be recalculated using Bayes' rule to update the probability model.

[0117] Step S34. If the "next-frame trajectories" of the construction vehicle and the operating personnel predicted by the above Kalman filter enhancement algorithm fall into the same grid, then multiply their probabilities to calculate the probability of the intersection of their trajectories.

[0118] Step S35. If the intersection probability is greater than a certain threshold, the system will give an early warning in time to prevent the construction vehicle and the operating personnel from intersecting at the power infrastructure site, and timely reduce the potential safety hazards at the power infrastructure site.

[0119] The above are only embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structural or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied to other related technical fields, shall be similarly included in the patent protection scope of the present invention.

Claims

1. Image processing method for construction vehicles and personnel at power infrastructure construction sites using monocular vision, Characterized in that, It includes the following steps: Collect image data of construction vehicles and personnel at power infrastructure construction sites using a monocular camera; Use the median denoising method to denoise the trained image samples and filter out salt-and-pepper noise in the images; Use the YoloX object detection algorithm to perform deep learning training on the image data of power infrastructure construction sites and obtain the approximate positions of construction vehicles and personnel respectively; Use the Canny edge detection algorithm, LSD line detection algorithm, and SLIC image superpixel segmentation algorithm to perform edge detection on the denoised image samples respectively, and obtain the edge detection results under different algorithms; Set confidence levels to fuse different edge detection results to obtain the final recognition models for construction vehicles and personnel; Use the trained model to detect moving construction vehicles and on-site personnel in the construction site images detected by the monocular camera; Define corresponding identification numbers, coordinates, identification box sizes, and target speeds for the identified construction vehicles and on-site personnel respectively; Use the Kalman filter enhancement algorithm to predict the identification box sizes and coordinate positions of the next frame for construction vehicles and operators respectively, and perform visualization of the identification boxes; Use the augmenting path matching algorithm to match the current frame target group with the previous frame target group and the union of the target groups that failed to match with the previous frame, and finally obtain the maximum match; output the coordinate status information for the successfully matched targets; Divide the power infrastructure construction site into an n*n grid pattern, then all trajectory coordinates should fall into the corresponding grids; Use the Bayesian probability estimation formula to calculate the prior probability of each "next trajectory point" appearing in a certain grid based on a large amount of historical trajectory data of construction vehicles and operators at power infrastructure construction sites; Input the "next trajectory points" of construction vehicles and operators predicted by the Kalman filter enhancement algorithm into the Bayesian probability calculation formula to calculate the trajectory prediction likelihood probabilities of the two respectively; If the prediction results of the "next trajectory points" are the same as the actual results, that is, they are all in the same grid, then the posterior probability should be recalculated using Bayesian rules to update the probability model; If the "next trajectory points" of construction vehicles and operators predicted by the Kalman filter enhancement algorithm fall into the same grid, then multiply the probabilities of the two to calculate the probability of the intersection of the two trajectories; If the intersection probability is less than a certain threshold, the system gives an early warning in time to prevent construction vehicles and operators from intersecting at power infrastructure construction sites and reduce potential safety hazards at power infrastructure construction sites in time.

2. The image processing method for construction vehicles and personnel at power infrastructure construction sites using monocular vision according to claim 1, Characterized in that, The specific formula of the Kalman filter enhancement algorithm model is as follows: X p+1 = F p X p + G p W p where X p represents the recognized target motion state vector, that is, the set of the speed and position of the object in the horizontal and vertical directions, the size of the recognition frame, and the ratio of the height to the width of the recognition frame, F p represents the object state transition matrix, G p represents the input matrix, W p represents the system input; F p The specific calculation formula is as follows: Where, E represents the identity matrix, and △T represents the time difference between a certain frame and the previous frame; Prior covariance matrix R p+1|p Obtained from the posterior covariance matrix and the process noise matrix, and the specific calculation formula is as follows: where E 7×7 is the process noise covariance matrix.

3. The image processing method for construction vehicles and personnel at power infrastructure construction sites using monocular vision according to claim 1, Characterized in that, The augmenting path matching algorithm matches the current frame target group with the previous frame target group and the union of the target groups that failed to match with the previous frame. The specific steps are as follows: Initialize the bipartite graph, that is, confirm the recognition frames in the previous frame that the recognition frame in the current frame may match; Match according to the defined order of recognition numbers. When the target in the previous frame and the target in the current frame may match, perform a match first; If the matching result of the target in the current frame with a later order and the target in the previous frame is the same as the matching result of the target in the current frame with an earlier order, recursively replace the matching target of the latter; If the target in the current frame fails to match successfully through the above method, regard this target as a newly emerged target; Repeat the above steps to match the target in the previous frame and the target in the current frame one by one, and output the coordinate status information of the successfully matched targets.

4. The image processing method for construction vehicles and personnel at the power infrastructure construction site using monocular vision according to claim 1, characterized in that, Calculate the likelihood probability of each "next trajectory point" appearing in a certain grid by using a large amount of historical trajectory data of construction vehicles and operating personnel at the power infrastructure construction site. The calculation formula is as follows: Where P(A|B) represents the probability that the next trajectory point is in a certain grid when the grid where the previous frame trajectory point is located is known, P(A,B) represents the probability that the upper and lower frames of a certain trajectory are in the known grid, and P(B) represents the probability that the previous frame of a certain trajectory is in the known grid; Use the "next trajectory points" of construction vehicles and operating personnel predicted by the Kalman filter enhancement algorithm and input them into the Bayesian probability calculation formula to calculate the trajectory prediction probabilities of the two respectively. The specific calculation formula is as follows: Where P(A|B) represents the posterior probability that the next trajectory point is in a certain grid when the grid where the previous frame trajectory point is located is known, P(B|A) is the likelihood probability, P(A) is the prior distribution, and P(B) is the normalization operator.

Images