High slope construction danger early warning method based on machine vision and deep learning

Abstract

This invention discloses a method for early warning of hazards during highway high slope construction based on machine vision and deep learning, comprising the following steps: S01, acquiring real-time video data of the construction slope using data acquisition equipment; S02, capturing abnormal situations such as rockfall and rock mass displacement using improved classical algorithms suitable for slopes, such as the frame difference method; S03, providing the construction unit with data on abnormal events for manual backtracking; S04, establishing a backpropagation (BP) neural network using the acquired abnormal data; S05, training the algorithm to autonomously judge early warnings of hazardous situations, such as rockfall, using deep learning technology; S06, using the training results in S05, combined with traditional displacement monitoring algorithms, to jointly issue early warnings for hazardous situations, primarily rockfall and landslides. The method provided by this invention effectively assists construction units in conducting safety assessments and analyses, utilizing machine vision and neural network technologies to provide early warnings of hazardous situations, ensuring the safety of construction personnel and equipment.

Description

Technical Field

[0001] This invention relates to a method for early warning of construction hazards on high highway slopes based on machine vision and deep learning, and belongs to the field of construction safety. Background Technology

[0002] Effective and accurate monitoring of high slopes along highways can provide timely and effective safety assessments and early warnings to construction departments, helping them to take appropriate measures to ensure the safety of construction personnel and equipment when safety hazards exist. It can also help construction units to evaluate and modify construction plans in a timely manner, ensuring the smooth progress of project construction.

[0003] However, research has found that the high cost and relatively short construction period of sensor monitoring systems, as well as the complexity of slope environments compared to man-made structures, hinder the application of traditional monitoring methods on slopes. Traditional methods often rely on inefficient approaches such as manual inspections. Summary of the Invention

[0004] This invention provides a method for early warning of construction hazards on high highway slopes based on machine vision and deep learning, which solves the problems disclosed in the background art.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by this invention is: a method for early warning of construction hazards on high highway slopes based on machine vision and deep learning.

[0006] Acquire real-time video data of the construction slope;

[0007] Video data is preprocessed using digital imaging algorithms;

[0008] According to the actual engineering needs, the preprocessed image is divided into several ROI regions for motion monitoring.

[0009] Calculate the difference matrix of the ROI regions between the two frames, and convolve all elements of the matrix with convolution kernel 1 and sum them to obtain the K value used to determine whether there is motion in the video;

[0010] Digital image correlation methods are used to identify the image features of natural marker points on slopes. The displacement of the marker points is calculated by matching and identification, and the true structural displacement of the natural marker points on the slopes is obtained by displacement calibration.

[0011] The actual displacement, split expansion data, real-time rainfall, and construction site vibration data are used as the input layer of the BP neural network. The K value of 30-3000 frames after displacement is used as the output layer of the BP neural network. The hidden layer of the BP neural network is set to 1 layer to obtain the relationship between the precursory phenomenon and the occurrence of danger.

[0012] Assess the degree of danger of the slope;

[0013] Early warnings are issued based on the level of danger.

[0014] Furthermore, the process of preprocessing video data using digital imaging algorithms is as follows:

[0015] Two-dimensional Gaussian filters are used to reduce image noise, and the original RGB three-channel data of the video data is converted into grayscale data. The grayscale conversion method is shown in formula (1). In formula (1), the parameter frame is the frame to be processed, frame_grav is the frame after grayscale processing, cv2.cvtColor() and cv2.COLOR_BGRGRAY are the grayscale conversion methods and their parameters built into OpenCV. The Gaussian function is shown in formula (2), where σ is the standard deviation of the Gaussian distribution function and e is the natural logarithm. Optical image stabilization is also performed on the camera data.

[0016] The formula for grayscale conversion is:

[0017] frame_gray=cv2.cvtColor(frame,cv2.COLOR_BGRGRAY) (1)

[0018] The two-dimensional Gaussian filter function is:

[0019]

[0020] Furthermore, the digital image correlation method analyzes the grayscale changes of the target region before and after deformation based on the cross-correlation function, and performs matching and recognition for the target sub-region. The cross-correlation function is:

[0021]

[0022]

[0023] Where N and M are the frame width and frame height, respectively; f(x,y) and g(x′,y′) represent the grayscale values ​​of the images before and after deformation, respectively; and C(x,y) is the calculated cross-correlation coefficient, which is the minimum value when the two images are most similar.

[0024] Furthermore, it also includes: storing and updating video data in real time in the form of a queue container with a length of 810; initially determining whether there is motion in the picture by comparing the measured frame difference feature value and the critical threshold; when it is determined that there is motion in the picture, retrieving 810 frames before and after the judgment point; and simultaneously saving K-value data containing timestamps for subsequent use in machine learning training sets and team analysis and optimization.

[0025] Furthermore, it also includes: combining the extracted 810 frames into a video file with a frame rate of 25fps;

[0026] The trajectory of falling rocks was plotted using a modified inter-frame difference method.

[0027] Furthermore, the improved inter-frame difference method is as follows:

[0028] Extract the RGB and HSV color spaces from the video image. Perform background subtraction calculations on each of the five color channels (R, G, B, S, V). Perform a logical AND operation on the results of the R, G, and B channels to reduce noise, obtaining result B1. Perform a logical OR operation on this result with the S and V channel results to obtain the final result B2. Perform an erosion operation on B2 to remove scattered white noise, obtaining the final result B. M The trajectory of the moving object is obtained by taking the smallest bounding rectangle of the object.

[0029] Accordingly, a computer-readable storage medium storing one or more programs: the one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods described above.

[0030] Accordingly, a computing device is characterized by comprising:

[0031] One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above.

[0032] The beneficial effects achieved by this invention are as follows:

[0033] 1. This invention acquires structural deformation data based on machine vision methods, which significantly reduces costs compared to traditional sensor methods;

[0034] 2. This invention does not require the laying of a large number of lines, and obtains deformation data of multiple components and multiple locations of the structure through a small number of cameras, which has high measurement efficiency;

[0035] 3. This invention can ensure the real-time performance and reliability of monitoring.

[0036] 4. This invention can solve the construction party's need for retrospective analysis of abnormal events such as rockfalls, which is difficult to achieve with traditional methods.

[0037] 5. This invention utilizes deep learning methods to retrain the acquired and processed data, effectively utilizing the large amount of information contained in the video data, avoiding information waste, and also leveraging the high-efficiency computing characteristics of computers to improve the efficiency of security assessment. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the process of the present invention;

[0039] Figure 2 A schematic diagram of the HSV color space;

[0040] Figure 3 Flowchart for improving the inter-frame difference method and motion trajectory capture algorithm;

[0041] Figure 4 The diagram shows the results of the improved laboratory verification.

[0042] Figure 5 This is for the calibration of cracks in the experimental model and the development of crack width before failure extracted by the algorithm;

[0043] Figure 6 Experimental results of natural environment testing for improving the inter-frame difference method;

[0044] Figure 7 The image shows the results of the trajectory capture algorithm being run on the construction site.

[0045] Figure 8 The schematic diagram of the BP neural network used. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0047] like Figure 1 As shown, this invention discloses a method for early warning of construction hazards on high highway slopes based on machine vision and deep learning, comprising the following steps:

[0048] Step one involves acquiring real-time video data of the construction slope using video capture equipment. To ensure the reliability of the video data, a high-definition camera with at least 8 megapixels should be used for video data acquisition. Furthermore, to guarantee the real-time continuity of the data, the camera should operate stably 24 hours a day and use a stable network for transmission. The real-time slope data includes 4K monitoring video of the slope covering 24 hours a day and various weather and lighting conditions, used for analysis of rockfalls, landslides, rock mass displacement, etc. Due to the large amount of video data, 5G technology is used for data transmission, eliminating the need for extensive cable laying.

[0049] Step 2: The video data is preprocessed using digital image processing algorithms. Gaussian filtering is used to reduce image noise, and the original RGB three-channel data of the video image is converted into grayscale data, retaining a single color matrix to make the data easier to perform subsequent calculations. The grayscale conversion method is shown in formula (1), where frame is the frame to be processed, frame_grav is the frame after grayscale processing, cv2.cvtColor() and cv2.COLOR_BGRGRAY are the built-in grayscale conversion methods and their parameters in OpenCV, and the Gaussian function is shown in formula (2), where σ is the standard deviation of the Gaussian distribution function and e is the natural logarithm. At the same time, in order to reduce the impact of camera shake on the judgment, the camera should be fixed as much as possible, and optical image stabilization should be performed on the camera data.

[0050] frame_gray=cv2.cvtColor(frame,cv2.COLOR_BGRGRAY) (1)

[0051]

[0052] Step 3: Divide the image into regions and divide it into several ROI regions according to the actual engineering needs for motion monitoring.

[0053] Step 4: Using the absdiff method provided by OpenCV, the difference matrix of the ROI regions of the two frames is obtained by subtracting the two frames and taking the absolute value. The difference between the two grayscale matrices is then calculated, and a threshold is set to determine whether an image point is a moving point. This can be expressed by the following mathematical formula 3, where T is the threshold and f(x,y) is the pixel value at coordinates (x,y) in the difference matrix:

[0054]

[0055] When k is 1, the pixel is considered a moving point. When the number of moving points is greater than the set threshold for the number of falling rock moving points, it can be preliminarily determined that there is falling rock movement in the image.

[0056] Step 5: Using the Gaussian optical formula and the basic principles of camera imaging, the value of the characteristic value K for phenomena such as falling rocks is theoretically calculated. Before the official start of operation, a falling rock experiment is conducted on-site to verify and adjust the calculation.

[0057] Note: The Gaussian optical formulas are as follows (4) and (5), where u and v are the object distance and image distance, respectively, f is the focal length, and β is the magnification.

[0058]

[0059] Step 6: The video stream data is stored and updated in real time in the form of a queue container with a length of 810. The presence of motion in the scene is initially determined by comparing the measured frame difference feature value with the critical threshold. When motion is determined, 810 frames before and after the determination point are extracted. At the same time, the K value data containing the timestamp is saved for subsequent machine learning training set and team analysis and optimization.

[0060] Step 7: Use the videowright provided by OpenCV to synthesize the extracted 810 frames into a video file for extracting the trajectory of moving objects. To ensure stable operation in subsequent work, the frame rate of the synthesized video should be fixed at 25fps.

[0061] Step 8: Extract the video file from Step 7 and use the improved background subtraction method to draw the trajectory of moving objects in the video for easy analysis by the construction unit. The improved method used is to extract the RGB and HSV color spaces from the video image, and the mapping formula from RGB to HSV space is as follows (6). Perform background subtraction calculation on the five color channels R, G, B, S, and V respectively. Take a logical "AND" operation on the results of the R, G, and B channels to reduce noise and obtain result B1. Take a logical "OR" operation on this result and the results of the S and V channels to obtain the final result B2. Perform an erosion operation on B2 to remove scattered white noise and obtain the final result B. M The trajectory of the moving object is obtained by taking the smallest bounding rectangle of the object. The specific algorithm flow is shown in Figure ().

[0062]

[0063] Step nine involves using digital image correlation methods to identify the image features of natural marker points on the slope (such as dangerous rocks and cracks), calculating the displacement of these marker points through matching, and obtaining the actual structural displacement through displacement calibration. The main principle of the digital image correlation method is based on analyzing the grayscale changes of the target area before and after deformation using a cross-correlation function. Matching and identification are then performed on the target sub-regions. The cross-correlation function can be expressed as the formula:

[0064]

[0065] Where N and M are the frame width and frame height, respectively; f(x,y) and g(x′,y′) represent the grayscale values ​​of the images before and after deformation, respectively; and C(x,y) is the calculated cross-correlation coefficient, which is the minimum value when the two images are most similar.

[0066] Step 10: The displacement of the rocks, the overall displacement of the rock mass, the crack propagation, real-time rainfall, and construction vibration are used as the input layer of the BP neural network. The K-value (used to determine the presence of falling rocks) and the presence of falling rock hazard (represented by 0 and 1) from 30 to 3000 frames after displacement are used as the output layer of the BP neural network. The hidden layer is set to 1. The pre-analyzed data is fed into the input and output layers of the BP neural network for autonomous learning to obtain the relationship between early warning phenomena and the occurrence of hazards. The BP neural network structure is as follows: Figure 4 .

[0067] Step 11: Taking into account various factors such as rock mass displacement, environmental conditions, and successful machine learning, the degree of slope danger is determined and divided into three levels: A "Safe", B "Minor hidden danger", and C "Major hidden danger". Different alarm measures are taken for different levels. The safety level table is shown in Table 1.

[0068] A: The site is safe, and the data is being stored normally.

[0069] B: The site has minor potential hazards. Workers should try to stay away from the alarm location and the safety department should conduct a safety inspection.

[0070] C: The site has significant hidden dangers and there is a high probability of rockfall, landslides, and other dangerous situations. Workers at the alarm point should immediately stop construction, move away from the slope, and have the safety department retrieve data to eliminate the safety hazards before continuing construction.

[0071]

[0072] Table 1

[0073] Step 12: After a hazard alarm occurs, the weights of each part should be adjusted according to the alarm situation and added to the method resource library for long-term optimization and promotion of the method.

[0074] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

[0075] A computer-readable storage medium storing one or more programs, the programs including instructions that, when executed by a computing device, cause the computing device to perform a machine vision and deep learning-based method for early warning of construction hazards on high highway slopes.

[0076] A computing device includes one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for executing a method for early warning of construction hazards on high highway slopes based on machine vision and deep learning.

[0077] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0079] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0080] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0081] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. A method for highway high slope construction hazard early warning based on machine vision and deep learning, characterized by: acquiring real-time video data of the construction slope; preprocessing the video data using digital image processing algorithms; dividing the preprocessed image into several ROI regions for motion monitoring according to actual engineering needs; calculating the difference matrix of the ROI regions of the two frames, and performing convolution on all elements of the matrix according to the convolution kernel 1 and summing up to obtain the K value for determining whether there is motion in the video; using digital image correlation method to identify the image features of the natural marker points of the slope, calculating the displacement of the marker points through matching identification, and obtaining the real displacement of the natural marker points of the slope through displacement calibration; taking the real displacement, whether the crack is expanding and its rate, real-time rainfall and construction site vibration data as the input layer of the BP neural network, and the K value of the 30-3000 frames after displacement as the output layer of the BP neural network to determine whether there is a dangerous situation of rock mass falling, and setting the hidden layer of the BP neural network to one layer to obtain the relationship between the precursor and the occurrence of danger; identifying the danger degree of the slope; warning according to the danger degree; the preprocessing process of the video data using digital image processing algorithms is: using a two-dimensional Gaussian filter to reduce image noise, converting the original RGB three-channel data of the video data into grayscale data, the grayscale method is shown in formula (1), in which the parameter frame is the frame to be processed, frame_grav is the frame after grayscale processing, cv2.cvtColor() and cv2.COLOR_BGRGRAY are the built-in grayscale method and its parameters of opencv, the Gaussian function is shown in formula (2), wherein, in the formula, σ is the standard deviation of the Gaussian distribution function, e is the natural logarithm, and optical anti-shake processing is performed on the camera data; the grayscale method formula is: frame_gray=cv2.cvtColor(frame,cv2.COLOR_BGRGRAY) (1); the two-dimensional Gaussian filter function is: The digital image correlation method is based on cross-correlation function analysis of gray scale changes before and after target area deformation, and matching identification is performed on the target sub-area, and the cross-correlation function is: where N, M are frame width and frame height respectively, f(x, y) and g(x ′ ,y ′ ) represent the gray scale of the images before and after deformation respectively, C(x, y) is the cross-correlation coefficient calculated, and the minimum value is taken, the two images are most similar. 2.The method of machine vision and deep learning based highway high slope construction danger early warning according to claim 1, characterized in that, Further comprising: storing and updating the video data in the form of a queue container with a length of 810 in real time, and preliminarily determining whether the picture exists motion by comparing the measured frame difference characteristic value with the critical threshold value, when it is determined that the picture exists motion, the 810 frames before and after the judgment point are taken out; and K value data containing time stamp are saved at the same time, for subsequent training set and team analysis optimization. 3.The method of claim 2, wherein, Further comprising: merge the extracted 810 frames into a video file, and the frame rate of the synthesized video is 25fps; the first 90 frames are used to synthesize background frames, and the improved inter-frame difference method based on color space reorganization is used to draw the trajectory of the moving object in the video. 4.The method of machine vision and deep learning based highway high slope construction danger early warning according to claim 3, characterized in that, The improved inter-frame difference method is: The RGB and HSV color spaces in the video image are extracted, the R, G, B, S and V color channel matrices are respectively calculated by the background subtraction method, the results of the R, G and B channels are subjected to logical "and" operation to weaken the noise points to obtain a result B1, the result B1 is subjected to logical "or" operation with the results of the S and V channels to obtain a final result B2, and the final result B2 is subjected to appropriate erosion operation to remove scattered white noise points to obtain a final result B m , and the minimum circumscribed rectangle of the object is taken to obtain the motion object trajectory.

5. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that when executed by a computer cause the computer to perform a method comprising: The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods of claims 1-4.

6. A computing device, comprising: including: One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to, with the one or more processors, perform any of the methods of claims 1-4.

Images