Excavator work safety monitoring method based on visual aid and lidar

Abstract

The present application relates to the technical field of image processing, and proposes a method for monitoring the safety of excavator operation based on visual assistance and laser radar, which comprises: acquiring monitoring images of continuous frames during the operation of the excavator through a vehicle-mounted camera, and acquiring three-dimensional point cloud data at several time points through a laser radar; obtaining the movement degree of targets in each region, screening and acquiring several moving targets and obtaining their movement sequences; obtaining the target influence degree of each moving target at each time point; obtaining the comprehensive influence degree of the moving targets; screening the interference targets based on the comprehensive influence degree, and combining the movement sequences corresponding to each vehicle-mounted camera to obtain the interference sequence of the interference targets; and performing collision warning through an audible and visual alarm according to the interference targets and their interference sequences, so as to realize the safety monitoring of the operation of the excavator. The present application aims to solve the problem that the misidentification of computer vision and laser radar will interfere with the accuracy of the fusion results for blind areas and collision detection.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method for monitoring the safety of excavator operations based on vision assistance and lidar. Background Technology

[0002] During the safety monitoring of excavator operations, due to the limited field of vision, there will be a large blind spot during the operation of the excavator. The construction site scene where the excavator is operating is relatively complex, and collision avoidance warning is required for complex terrain environment. During the operation, LiDAR is used to detect targets near the excavator, and computer vision is used to identify the types of nearby targets, so as to achieve targeted blind spot collision avoidance warning.

[0003] The data types of image data acquired by computer vision and 3D point cloud data acquired by LiDAR differ, and multimodal data fusion needs to consider the spatiotemporal synchronization of different data types. At the same time, computer vision has the problem of misidentifying ground shadows, and a large number of similar targets can also cause visual recognition confusion. LiDAR has poor ability to identify terrain and color texture, which can lead to frequent alarms or false alarms. In addition, the working range of the excavator arm changes significantly during excavator operation, so it is necessary to consider the problem of rapid response to blind spot detection, and to promptly perform blind spot compensation and collision detection through vision assistance and LiDAR during excavator operation, and to provide timely warnings through audible and visual alarms to ensure the safety of excavator operation. Summary of the Invention

[0004] This invention provides a method for excavator operation safety monitoring based on vision assistance and LiDAR, to solve the problem that the misidentification problems of existing computer vision and LiDAR can interfere with the accuracy of fusion results in blind spot and collision detection. The specific technical solution adopted is as follows:

[0005] This invention proposes a method for monitoring the safety of excavator operations based on vision assistance and lidar, which includes the following steps:

[0006] The system acquires continuous frame monitoring images of the excavator during operation through vehicle-mounted cameras, obtains three-dimensional point cloud data at several moments through lidar, and simultaneously acquires continuous frame detection images from the robotic arm camera.

[0007] The grayscale and area changes of each region in the continuous frame monitoring images are analyzed to obtain the target movement degree of each region. Several moving targets are selected and their movement sequences are obtained. The monitoring images and 3D point cloud data are fused based on timestamps. The stereo information is quantified according to the distribution of moving targets in the 3D point cloud data. The weights of stereo information and grayscale information in the monitoring images are adjusted in combination with the movement sequences to obtain the target influence degree of each moving target at each time.

[0008] The differences in the degree of target influence of the same moving target under different vehicle-mounted cameras were analyzed. The grayscale and position of the moving target in the detection image of the robotic arm camera at the corresponding time were combined to obtain the comprehensive degree of influence of the moving target. Interference targets were screened and combined with the movement sequence corresponding to each vehicle-mounted camera to obtain the interference sequence of the interference targets.

[0009] Based on the interference target and its interference sequence, a collision warning is issued via an audible and visual alarm.

[0010] Optionally, the specific methods for obtaining the target movement degree in each region include:

[0011] For any consecutive frames of monitoring images from a vehicle-mounted camera, several regions are obtained. The same region corresponding to different frames of monitoring images is obtained. Several frames of monitoring images in which the region exists are obtained from the vehicle-mounted camera and used as the monitoring images of the region in the vehicle-mounted camera. The area and centroid coordinates of the corresponding region in each monitoring image are obtained, and the average gray value of all pixels in the corresponding region is used as the gray value of the corresponding region.

[0012] Based on the area difference, grayscale value difference, and centroid coordinate distance of the corresponding area in the two adjacent monitoring images, the area change, grayscale change, and movement distance of the region in the next monitoring image are obtained respectively. The diagonal length of the smallest bounding rectangle of the corresponding area in the next monitoring image is taken as the maximum movement distance of the region in the next monitoring image.

[0013] The target movement degree γ in region a a The calculation method is as follows:

[0014]

[0015] Where, N a Δd represents the number of monitoring images in region a. a,n D represents the distance moved by the a-th region in the n-th frame of the monitored image. a,n Δg represents the maximum movement distance of the a-th region in the n-th frame of the monitored image. a,n g represents the amount of grayscale change in the monitored image of region a in frame n. max ΔS represents the maximum grayscale value. a,n This indicates the degree of area change of the a-th region in the n-th frame of the monitored image.

[0016] Optionally, the specific method for filtering to obtain several moving targets and their movement sequences includes:

[0017] The region where the target's movement exceeds a movement threshold is defined as the moving target. The distance between the centroid of the moving target and the vehicle camera at any given moment in any monitoring image of any vehicle camera is obtained and used as an element of the moving target's movement sequence at the corresponding moment in that monitoring image. The moments corresponding to each monitoring image of any moving target under the vehicle camera are used as the analysis moments of the moving target under the vehicle camera, thus obtaining the elements of each analysis moment in the moving target's movement sequence, which constitute the moving target's movement sequence under the vehicle camera.

[0018] Optionally, the specific method for fusing the monitoring image and the 3D point cloud data based on the timestamp includes:

[0019] Each consecutive frame of the monitoring image from any vehicle-mounted camera corresponds to a collection time. The monitoring image from the vehicle-mounted camera at any time is fused with the 3D point cloud data at that time to obtain the spatial region of each moving target in the 3D point cloud data at that time.

[0020] Optionally, the specific method for obtaining the target influence degree of each moving target at each time step includes:

[0021] Analyze the point cloud distribution of moving targets in the spatial region of 3D point cloud data to obtain the three-dimensional representation of each moving target at each time; obtain the target movement degree of any moving target at each analysis time under any vehicle-mounted camera.

[0022] Based on the mean and standard deviation of gray values ​​of all pixels in the corresponding region of the monitoring image of the moving target under the vehicle camera at any analysis time, the gray complexity factor of the moving target under the vehicle camera at that analysis time is obtained. The gray complexity factor is positively correlated with the mean and standard deviation of the gray values.

[0023] The degree of target influence of the b-th moving target at the p-th analysis time under the vehicle-mounted camera is f. b,p The calculation method is as follows:

[0024]

[0025] Where, γ b,p δ represents the degree of movement of the b-th moving target at the p-th analysis time as seen by the vehicle-mounted camera. b,p h represents the degree of stereoscopic representation of the b-th moving target at the p-th analysis time under the vehicle-mounted camera. b,p Let l represent the grayscale complexity factor of the b-th moving target at the p-th analysis time as seen by the vehicle-mounted camera. b,p L represents the element at the p-th analysis time in the movement sequence of the b-th moving target within the vehicle-mounted camera. bThis represents the maximum value of all elements in the movement sequence of the b-th moving target within the vehicle-mounted camera.

[0026] Optionally, the specific method for obtaining the three-dimensional representation of each moving target at each time step includes:

[0027] The spatial region of any moving target in the 3D point cloud data at any analysis time under any vehicle-mounted camera is obtained. The variance of the coordinates of all points in the spatial region in three dimensions is calculated respectively to obtain the coordinate variance of the spatial region in each of the three dimensions. The ratio of the minimum value to the maximum value of the coordinate variance of the three dimensions is obtained as the degree of three-dimensional representation of the moving target at the analysis time.

[0028] Optionally, the specific methods for obtaining the overall impact of the moving target include:

[0029] Based on the degree of target influence of the same moving target under different vehicle cameras at the same time, we obtain several monitoring times of the moving target under each vehicle camera, as well as the similarity of the monitoring changes of the moving target.

[0030] Obtain all monitoring times of any moving target from each vehicle-mounted camera, and sort them in chronological order as several movement times of the target; obtain the target influence degree of the target at each movement time corresponding to the robotic arm camera, and the comprehensive influence degree of the i-th moving target. The calculation method is as follows:

[0031]

[0032] Among them, T i Δf represents the number of movement times for the i-th moving target. i,t c represents the difference between the maximum and minimum values ​​of the target influence levels corresponding to each vehicle-mounted camera at the t-th moment of the i-th moving target's movement. i f′ represents the similarity of monitored changes for the i-th moving target. i,t This represents the degree of target influence corresponding to the robotic arm camera at the t-th moment of the i-th moving target's movement.

[0033] Optionally, the specific methods for obtaining the moving target at several monitoring times of each vehicle-mounted camera, and the similarity of the monitoring changes of the moving target, include:

[0034] The same moving target is obtained under the monitoring images of different vehicle cameras; for any moving target, the target influence degree of the moving target at each time under each vehicle camera is obtained; for any vehicle camera, the time when the target influence degree of the moving target corresponding to that vehicle camera is the maximum value among the target influence degrees corresponding to each vehicle camera is taken as the monitoring time of the moving target at that vehicle camera.

[0035] The moving target is captured at several monitoring times of the vehicle-mounted camera, and the target influence of the moving target at each monitoring time is arranged according to the chronological order of the monitoring times to obtain the target monitoring sequence of the moving target at the vehicle-mounted camera.

[0036] Obtain the target monitoring sequence of the moving target in each vehicle camera, calculate the DTW distance of the moving target in any two vehicle camera target monitoring sequences, and use the inverse proportional normalized value of the mean DTW distance obtained between the target monitoring sequences as the monitoring change similarity of the moving target.

[0037] Optionally, the specific method for obtaining the interference sequence of the interference target includes:

[0038] For any interfering target, the minimum value among the elements of the movement sequence of each vehicle-mounted camera and robotic arm camera corresponding to the interfering target at any given time is taken as the element of the interfering target's interference sequence at that time. This process is used to obtain the elements of the interfering target's interference sequence at each time, thus forming the interfering sequence of the interfering target.

[0039] Optionally, the method for providing collision warning via an audible and visual alarm based on the interfering target and its interference sequence includes:

[0040] For any interfering target, the degree of its influence at each moment is determined by each vehicle-mounted camera and robotic arm camera. The moment corresponding to the maximum value of the target influence is taken as the size analysis moment of the interfering target, and the corresponding camera is taken as the size analysis camera of the interfering target. The area of ​​the region of the interfering target in the monitoring image of its size analysis camera at the size analysis moment is obtained, and the ratio of the area of ​​the region to the area of ​​the entire monitoring image is taken as the maximum size performance value of the interfering target. If the maximum size performance value is less than the size ratio threshold, the interfering target is deleted. Size screening is performed on all interfering targets to obtain a number of filtered interfering targets.

[0041] Based on the screened interference targets and their interference sequences, a collision risk threshold is used to determine the collision risk, and a collision signal is transmitted to the audible and visual alarm for collision warning.

[0042] The beneficial effects of this invention are as follows: This invention analyzes the grayscale and area changes of continuous frame monitoring images to perform preliminary analysis of moving targets in computer vision monitoring images, obtains the degree of target movement, and further acquires movement sequences, which reflect the movement characteristics of the moving targets. Furthermore, it analyzes the initially screened moving targets in 3D point cloud data, ensuring that the spatiotemporal information of the computer vision monitoring images and the 3D point cloud data from the LiDAR is identical during the fusion process. By analyzing the stereo information of the moving targets, it avoids misjudging shadow movement as potential interference targets. Simultaneously, the analysis comprehensively considers the distance between the target and the excavator to prevent the point cloud from being too sparse, which would affect the accuracy of stereo feature analysis. This allows for the determination of the target's influence at various times, providing a basis for subsequent analysis of the target's interference with excavator operations. The probability quantification provides the foundation; by comprehensively analyzing the target impact of moving targets through monitoring images from multiple vehicle-mounted cameras and detection images from robotic arm cameras, the target impact of moving targets increases over time, and the overall trend of change is similar under different vehicle-mounted cameras. Combined with the grayscale and position performance of the target in the detection images from the robotic arm camera, the target interference level is comprehensively quantified to screen interfering targets. Based on the movement sequence, the interference sequence of interfering targets is obtained, comprehensively reflecting the overall distance change between the interfering target and the excavator, providing a basis for subsequent early warning of whether the interfering target will collide with the excavator. By combining visual assistance and blind spot filling from multi-directional cameras and target distance monitoring from 3D point cloud data of LiDAR, collision risk monitoring and early warning can be carried out efficiently and timely, effectively ensuring the safety of excavator operation. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of a safety monitoring method for excavator operation based on vision assistance and lidar provided in one embodiment of the present invention;

[0045] Figure 2 A diagram showing the installation location of a vehicle-mounted camera;

[0046] Figure 3 A schematic diagram showing the installation location of the camera on the robotic arm;

[0047] Figure 4 Example image of dynamic 3D point cloud data detected by lidar. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Please see Figure 1 The diagram illustrates a flowchart of an excavator operation safety monitoring method based on vision assistance and lidar, according to an embodiment of the present invention. The method includes the following steps:

[0050] Step S001: Acquire continuous frame monitoring images of the excavator during operation through the vehicle-mounted camera, acquire three-dimensional point cloud data at several moments through the lidar, and simultaneously acquire continuous frame detection images from the robotic arm camera.

[0051] The purpose of this embodiment is to use 3D point cloud data detected by LiDAR during excavator operation, supplemented by video monitoring based on computer vision, to track and monitor the environment and targets around the excavator. This aims to reduce the problem of blind spots during excavator operation where close-range targets can easily interfere with or collide with the excavator. Collision warnings can be issued in a timely manner before a collision occurs, ensuring the safety of excavator operation. This requires deploying vehicle-mounted cameras at multiple locations on the excavator, and also placing cameras at the front end of the robotic arm for real-time video monitoring to reduce blind spots. Simultaneously, by combining the 3D point cloud data from LiDAR with the video monitoring from the vehicle-mounted cameras, interference from potential moving targets can be identified to reduce the risk of collision.

[0052] Specifically, on the rear side of the excavator cab, vehicle-mounted cameras are installed on the frame in three unobstructed positions in the left, right, and rear directions, such as... Figure 2 As shown, real-time video monitoring is performed. In this embodiment, video monitoring is performed by acquiring 10 frames per second, thus obtaining continuous frame monitoring images from each vehicle-mounted camera; a camera is installed at the joint of the second section of the excavator's boom as a robotic arm camera, as shown. Figure 3 As shown, video monitoring is performed using the same sampling time interval as the surveillance images to obtain continuous frames of detection images from the robotic arm's camera. LiDAR sensors are deployed on both sides of the chassis behind the cab, and real-time 3D point cloud data is acquired using the same sampling time interval as the surveillance images. This yields 3D point cloud data at several consecutive moments, as shown in the image. Figure 4 As shown.

[0053] It should be noted that both computer vision-based surveillance images and LiDAR-obtained 3D point cloud data have their own misidentification issues. Both can correctly identify abnormally moving targets or occluded targets at close range. However, computer vision has a problem with misidentifying planar shadows. By combining the computer vision with the 3D point cloud data showing the target's stereoscopic representation, it can eliminate the misjudgment of shadows as interfering targets. Meanwhile, LiDAR's neglect of color and texture, and the sparse point cloud caused by distant or small obstacles, can lead to missed judgments or misidentification as noise filtering. Therefore, combining the sparse point cloud with the corresponding target's representation in the visual surveillance image helps identify the appropriate obstacle or target. Furthermore, due to the rotation of the excavator's robotic arm and the excavator's movement, there are changes in distance and viewing angle between the excavator and various targets. Distance changes affect the 3D point cloud data, while viewing angle changes affect the image data. During the fusion process, the impact of spatial changes (i.e., changes in viewing angle and distance) under timestamp alignment needs to be considered.

[0054] Step S002: Analyze the grayscale and area changes of each region in the continuous frame monitoring images to obtain the target movement degree of each region, select and obtain several moving targets and their movement sequences; fuse the monitoring images and 3D point cloud data based on timestamps, quantify the stereo information according to the distribution of moving targets in the 3D point cloud data, and adjust the weights of stereo information and grayscale information in the monitoring images in combination with the movement sequences to obtain the target influence degree of each moving target at each time.

[0055] It should be noted that due to the presence of multi-directional vehicle-mounted cameras, blind spots during excavator operation can be filled through multi-angle video monitoring fusion. Collision warning, on the other hand, requires analysis of changes in the distance between the target and the excavator. During excavator operation, the vehicle body and robotic arm will move, and the cab and robotic arm will also rotate. The movement of the target itself will also cause changes in the corresponding areas in the continuous frame monitoring images. Therefore, by analyzing the changes in the area of ​​the region in the continuous frame monitoring images, as well as the changes in the grayscale representation of each region, that is, the changes in the target position caused by the movement of the excavator itself will affect its grayscale representation, the target movement analysis is performed, including the target's own movement and the movement of the target in the monitoring image under the movement of the excavator, and the change in movement distance is obtained to obtain the movement sequence.

[0056] Preferably, in one embodiment of the present invention, the grayscale changes and area changes of each region in a continuous frame of monitoring images are analyzed to obtain the target movement degree of each region, and several moving targets are selected and their movement sequences are obtained. The specific method includes:

[0057] For any consecutive frames of monitoring images from a vehicle-mounted camera, several regions in each frame are obtained through Canny edge detection and connected component analysis. The regions in different frames are matched and the movement trajectory is obtained through sparse optical flow method to obtain the corresponding identical regions in different frames. The edge detection algorithm is used to obtain the edges, and the regions are obtained based on the edges based on connected component analysis. Canny edge detection and connected component analysis are both well-known technologies. At the same time, the sparse optical flow method for monitoring the movement trajectory and matching the moving regions is an existing method, which will not be described in detail in this embodiment.

[0058] For any given region, several frames of monitoring images from the vehicle-mounted camera within that region are acquired as monitoring images of that region from the vehicle-mounted camera. All monitoring images are arranged according to their corresponding timestamps. It should be noted that the monitoring images described in this embodiment are all based on monitoring images acquired from the vehicle-mounted camera. The area and centroid coordinates of the corresponding region in each monitoring image are obtained (a coordinate system is established with the lower left corner of the monitoring image as the origin, and the horizontal right and vertical upward directions are positive, to obtain the coordinates of each pixel). The average grayscale value of all pixels in the corresponding region is used as the grayscale representation value of the corresponding region. For the corresponding region in any two adjacent monitoring images (the corresponding timestamps may not be adjacent, i.e., the monitoring images are arranged in timestamp order rather than sequentially, and there may be non-corresponding monitoring images), (Timestamps are not included in the sorting). The area of ​​the corresponding region in the next frame of the monitoring image is subtracted from the area of ​​the corresponding region in the previous frame of the monitoring image. The ratio of this difference to the area of ​​the corresponding region in the next frame of the monitoring image is used as the degree of area change of the region in the next frame of the monitoring image. The absolute value of the difference in grayscale values ​​of the corresponding regions in two adjacent frames of the monitoring image is obtained as the amount of grayscale change of the region in the next frame of the monitoring image. The minimum bounding rectangle of the corresponding region in the next frame of the monitoring image is obtained. The length of the diagonal of the minimum bounding rectangle is used as the maximum movement distance of the region in the next frame of the monitoring image. The distance between the coordinates of the centroid of the corresponding region in the next frame of the monitoring image and the coordinates of the centroid of the corresponding region in the previous frame of the monitoring image is used as the movement distance of the region in the next frame of the monitoring image. Then, the target movement degree γ of the a-th region is calculated. a The calculation method is as follows:

[0059]

[0060] Where, N a Δd represents the number of monitoring images in region a. a,n D represents the distance moved by the a-th region in the n-th frame of the monitored image. a,n Δg represents the maximum movement distance of the a-th region in the n-th frame of the monitored image. a,ng represents the amount of grayscale change in the monitored image of region a in frame n. max ΔS represents the maximum grayscale value (i.e., the upper limit of the grayscale value range, 255). a,n This indicates the degree of area change of the a-th region in the n-th frame of the monitored image.

[0061] It should be noted that both the degree of area change and the amount of grayscale change can reflect the movement of a region in consecutive frames of monitoring images. A positive degree of area change indicates that the corresponding region is approaching the vehicle camera. The smaller the movement distance between adjacent frames of monitoring images, the more reliable the degree of area change. When the movement distance is larger, the grayscale change of the corresponding region is more important. The larger the grayscale change, the greater the displacement relative to the vehicle camera. That is, the relative displacement causes factors such as lighting to affect the grayscale, thus producing grayscale changes. The magnitude of grayscale change and the degree of area change are both large when monitoring images are accumulated. The greater the possibility that the region has moved in consecutive frames of monitoring images, the greater the degree of target movement.

[0062] Furthermore, a preset movement threshold is established. In this embodiment, the movement threshold is described as 0.6. Areas where the target movement exceeds the movement threshold are considered moving targets. Several moving targets are then selected from several areas obtained from the continuous frame monitoring images of the vehicle-mounted camera. For the corresponding area of ​​the moving target in any monitoring image, the area, centroid position of the corresponding area, and the area of ​​the monitoring image are known, and the focal length of the vehicle-mounted camera is also known. Then, using the focal length, area, and centroid position, the distance (actual spatial distance) between the centroid of the moving target and the vehicle-mounted camera at the corresponding moment in the monitoring image is obtained, which is used as an element of the movement sequence of the moving target at the corresponding moment in the monitoring image. The moment corresponding to each monitoring image of any moving target under the vehicle-mounted camera is taken as the analysis moment of the moving target under the vehicle-mounted camera. Each moment in the movement sequence is the analysis moment of the moving target under the vehicle-mounted camera, and they are arranged in chronological order. Thus, the elements of each analysis moment in the movement sequence of the moving target are obtained, constituting the movement sequence of the moving target under the vehicle-mounted camera.

[0063] It should be further explained that after a moving target is acquired from the monitoring image, timestamp alignment is used to ensure that the spatiotemporal information of the computer vision monitoring image and the 3D point cloud data of the LiDAR is consistent. Since the analysis of moving targets through continuous frame monitoring images can accurately extract areas of change and movement without being affected by color, texture, etc., the target is extracted first from the monitoring image. Then, the corresponding point cloud information is obtained by fusion in the 3D point cloud data. The stereo information is analyzed based on the sparse distribution of the point cloud to avoid misjudging the moving part of the shadow as a moving target. At the same time, the weight is adjusted by combining the distance distribution of the movement sequence, i.e., the target. If the distance is close, the 3D point cloud data is more accurate, and its sparseness and stereo information can better reflect the impact of the target, so the weight should be larger. Conversely, the farther the distance, the more sparse the point cloud data is, and its stereo information may not be accurate enough. It is necessary to analyze it in conjunction with the grayscale performance in the monitoring image to obtain the degree of target impact at each time, which provides a basis for quantifying the possibility of the target interfering with the excavator operation.

[0064] Preferably, in one embodiment of the present invention, the monitoring image and the 3D point cloud data are fused based on timestamps, the stereo information is quantified according to the distribution of the moving target in the 3D point cloud data, and the weights of the stereo information and the grayscale information in the monitoring image are adjusted in combination with the movement sequence to obtain the target influence degree of each moving target at each time. The specific method includes:

[0065] Each consecutive frame of the monitoring image from any vehicle-mounted camera corresponds to a collection time. The monitoring image from the vehicle-mounted camera at any time is fused with the 3D point cloud data at that time to obtain the spatial region of each moving target in the 3D point cloud data at that time. It should be noted that for any moving target, only the spatial region of it in the 3D point cloud data is obtained at several analysis times under the vehicle-mounted camera.

[0066] Furthermore, the spatial region of the moving target in the 3D point cloud data at any analysis time under the vehicle-mounted camera is obtained, and the variance of the coordinates of all points in the spatial region in three dimensions is calculated respectively to obtain the variance of the coordinates of the three dimensions of the spatial region. That is, the variance of the x-axis coordinate of all points in the spatial region is calculated, and the variance of the y-axis coordinate and z-axis coordinate is calculated in the same way. The ratio of the minimum value to the maximum value of the three-dimensional variance of the three dimensions is obtained as the degree of three-dimensional representation of the moving target at the analysis time.

[0067] It should be noted that the more three-dimensional the spatial region, the greater the variance of the three-dimensional coordinates will be, which will make the ratio closer to 1. Conversely, if it is a shaded area or the spatial region tends to be a two-dimensional plane, it will result in a smaller ratio, that is, a greater difference between the minimum and maximum values. At the same time, if the variance of the coordinates is small, resulting in a ratio closer to 1, it will make the spatial region tend to be a one-dimensional point, or the point cloud will be sparse due to the distance from the lidar, and the spatial region will be small. This will be adjusted in subsequent movements.

[0068] Furthermore, the degree of target movement of the moving target corresponding to the vehicle-mounted camera is assigned to each analysis time under the vehicle-mounted camera, thus obtaining the degree of target movement of the moving target at each analysis time under the vehicle-mounted camera; the product of the mean gray value and the standard deviation of gray value of all pixels in the corresponding region of the monitoring image of the moving target at any analysis time under the vehicle-mounted camera is obtained, and the product is proportionally normalized. The result is used as the gray-level complexity factor of the moving target under the vehicle-mounted camera at that analysis time. In this embodiment, a 1-exp(-x) model is used to present the proportional relationship and normalization processing, where exp() represents an exponential function with the natural constant as the base, and x is the input of the model. The implementer can set the proportional function and the normalization function according to the actual situation; then the degree of target influence f of the b-th moving target under the vehicle-mounted camera at the p-th analysis time. b,p The calculation method is as follows:

[0069]

[0070] Where, γ b,p δ represents the degree of movement of the b-th moving target at the p-th analysis time as seen by the vehicle-mounted camera. b,p h represents the degree of stereoscopic representation of the b-th moving target at the p-th analysis time under the vehicle-mounted camera. b,p Let l represent the grayscale complexity factor of the b-th moving target at the p-th analysis time as seen by the vehicle-mounted camera. b,p L represents the element at the p-th analysis time in the movement sequence of the b-th moving target within the vehicle-mounted camera. b This represents the maximum value of all elements in the movement sequence of the b-th moving target within the vehicle-mounted camera.

[0071] It should be noted that the adjustment is made using elements in the moving sequence as reference weights. The elements in the moving sequence are the distances between the moving target and the vehicle-mounted camera. The greater the distance, the lower the reference value of the stereo information may be due to the sparse point cloud. More reference is made to the grayscale performance of the corresponding area in the monitoring image. The greater the grayscale performance and the wider the grayscale distribution range, the greater the grayscale complexity factor, and the lower the possibility of it being a planar shadow area. On the other hand, the greater the stereoscale performance, the richer the stereoscale information, and the greater the possibility that the moving target is not a two-dimensional or even one-dimensional area. The greater the credibility of its subsequent interference, the higher the degree of target influence.

[0072] Furthermore, by obtaining the degree of target influence for each moving target under each vehicle-mounted camera at its respective analysis time, the degree of target influence of each moving target at each time can be obtained.

[0073] Thus, by analyzing the grayscale and area changes of continuous frame monitoring images, a preliminary analysis of moving targets in the monitoring images under computer vision is conducted to obtain the degree of target movement and further acquire movement sequences, which reflect the movement characteristics of the moving targets. The initially screened moving targets are further analyzed in 3D point cloud data. During the fusion process, the spatiotemporal information of the computer vision monitoring images and the 3D point cloud data from the LiDAR is ensured to be consistent. By analyzing the stereo information of the moving targets, shadow movement is avoided from being misjudged as potential interference targets. The distance between the target and the excavator is also comprehensively considered during the analysis to prevent the point cloud from being too sparse, which would affect the accuracy of the stereo feature analysis. This yields the degree of target influence at each moment, providing a basis for quantifying the possibility of target interference with excavator operations.

[0074] Step S003: Analyze the differences in the degree of target influence of the same moving target under different vehicle-mounted cameras, and combine the grayscale and position performance of the moving target in the detection image of the robotic arm camera at the corresponding time to obtain the comprehensive degree of influence of the moving target; screen the interfering targets and combine the movement sequences corresponding to each vehicle-mounted camera to obtain the interference sequence of the interfering targets.

[0075] It should be noted that the degree of target influence of a moving target at different times corresponds to different vehicle-mounted cameras. 3D point cloud data can be used to fuse moving targets corresponding to different vehicle-mounted cameras. Since excavator operation involves movement and rotation of the cab and robotic arm, a comprehensive analysis of the moving target's movement is necessary. The more similar the trend of the target's influence under different vehicle-mounted cameras, the greater its interference with excavator operation safety. Combining the grayscale and position information of the moving target corresponding to the image detected by the robotic arm camera, and considering the large range of movement of the robotic arm during excavator operation, it is even more necessary to analyze the potential interference of the target in a timely manner. This quantifies the comprehensive influence of the moving target, enabling timely interference analysis and subsequent early warning during excavator operation.

[0076] Preferably, in one embodiment of the present invention, the variation of the target influence degree of the same moving target under different vehicle-mounted cameras is analyzed, and the comprehensive influence degree of the moving target is obtained by combining the grayscale and position performance of the moving target in the detection image of the robotic arm camera at the corresponding time. The specific method includes:

[0077] Since moving targets are acquired based on monitoring images from vehicle-mounted cameras, and then the spatial region corresponding to the moving target in the 3D point cloud data is obtained by fusing the monitoring images with 3D point cloud data, if moving targets acquired from different vehicle-mounted cameras correspond to the same spatial region of the moving target in the 3D point cloud data, then the same moving target is obtained from the monitoring images of different vehicle-mounted cameras. It should be noted that the 3D point cloud data includes the spatial regions corresponding to the moving targets acquired from vehicle-mounted cameras in all directions, and the monitoring images from vehicle-mounted cameras in different directions have overlapping ranges. Therefore, by taking advantage of the situation where the moving target exists in the monitoring images of two or more vehicle-mounted cameras, the spatial region of the moving target is matched in the 3D point cloud data, thereby achieving the correspondence of the same moving target under different vehicle-mounted cameras.

[0078] Furthermore, for any moving target, the target influence degree of the moving target at each moment under each vehicle-mounted camera is obtained. For any vehicle-mounted camera, the moment when the target influence degree of the moving target corresponding to that vehicle-mounted camera is the maximum value among the target influence degrees corresponding to all vehicle-mounted cameras is taken as the monitoring moment of the moving target at that vehicle-mounted camera. Several monitoring moments of the moving target at the vehicle-mounted cameras are obtained, and the target influence degree of the moving target at each monitoring moment is arranged according to the chronological order of the monitoring moments to obtain the target monitoring sequence of the moving target at the vehicle-mounted cameras. It should be noted that if the target influence degree corresponding to multiple vehicle-mounted cameras is the maximum value at a certain moment, the monitoring moment of that moment is determined based on the vehicle-mounted camera corresponding to the previous monitoring moment. It should be noted that if the target influence degree of the moving target at any moment is not present in the monitoring image of any vehicle-mounted camera, that is, the moving target does not exist in the monitoring image of that vehicle-mounted camera at that moment, that moment of the moving target is not included in its analysis moment, and that vehicle-mounted camera at that moment does not participate in the comparison of the monitoring moment acquisition.

[0079] Furthermore, the target monitoring sequences of the moving target in each vehicle-mounted camera are obtained. The DTW distance between any two vehicle-mounted camera target monitoring sequences of the moving target is calculated. The inversely proportional normalized value of the average DTW distance between the target monitoring sequences is used as the similarity of the monitoring changes of the moving target. In this embodiment, the exp(-y) model is used to present the inverse proportional relationship and normalization processing. exp() represents an exponential function with the natural constant as the base, and y is the input of the model. The implementer can set the inverse proportional function and the normalization function according to the actual situation. It should be noted that if there is only one target monitoring sequence corresponding to the moving target, the similarity of the monitoring changes of the moving target is set to 1.

[0080] Furthermore, all monitoring times of the moving target at each vehicle-mounted camera are obtained and sorted in chronological order as several movement times of the moving target. Based on the 3D point cloud data and the detection images of the robotic arm camera, the data are fused by timestamp alignment to obtain the corresponding region of the moving target in the detection image of the robotic arm camera at each movement time, thereby obtaining the target influence degree of the moving target at each movement time corresponding to the robotic arm camera. It should be noted that the corresponding region of the moving target may not exist in the detection image of the robotic arm camera at the movement time, in which case its target influence degree is 0.

[0081] Furthermore, the overall impact of the i-th moving target The calculation method is as follows:

[0082]

[0083] Among them, T i Δf represents the number of movement times for the i-th moving target. i,t c represents the difference between the maximum and minimum values ​​of the target influence levels corresponding to each vehicle-mounted camera at the t-th moment of the i-th moving target's movement. i f′ represents the similarity of monitored changes for the i-th moving target. i，t This represents the degree of target influence corresponding to the robotic arm camera at the t-th moment of the i-th moving target's movement.

[0084] It should be noted that the greater the range between the target influence levels corresponding to different vehicle-mounted cameras at different times of movement, and the greater the likelihood that the target influence is mainly distributed in the monitoring images of different vehicle-mounted cameras at different times of movement, the higher the reliability of the similarity of monitoring changes obtained based on the monitoring times of each vehicle-mounted camera. If the range is smaller, the target influence levels corresponding to different vehicle-mounted cameras are closer, and more attention should be paid to the target influence level corresponding to the robotic arm camera to obtain the final comprehensive influence level.

[0085] It should be further explained that the comprehensive impact level reflects the overall interference of moving targets on the safety of excavator operation. Therefore, the interference targets are screened based on the comprehensive impact level. At the same time, the movement sequence of the interference targets corresponding to each vehicle-mounted camera reflects the distance change between the interference targets and the excavator. After screening the interference targets, it is necessary to comprehensively extract the distance changes to obtain the interference sequence, which intuitively reflects the overall distance change between the interference targets and the excavator, and provides a basis for subsequent early warning of whether the interference targets will collide with the excavator.

[0086] Preferably, in one embodiment of the present invention, the method for filtering interference targets and combining the movement sequences corresponding to each vehicle-mounted camera to obtain the interference sequence of the interference targets includes:

[0087] A preset impact threshold is set. In this embodiment, the impact threshold is described as 0.7. Moving targets with a comprehensive impact level greater than or equal to the impact threshold are considered as interference targets. For any interference target, since the movement sequence of the corresponding continuous frame monitoring images (detection images) of each vehicle-mounted camera and robotic arm camera is known, the minimum value of the elements of each movement sequence at any given time is taken as the element of the interference sequence at that time. In this way, the elements of each time in the interference sequence of the interference target are obtained, which constitute the interference sequence of the interference target.

[0088] Thus, by comprehensively analyzing the impact of moving targets on the monitoring images from multiple vehicle-mounted cameras and the detection images from the robotic arm camera, the impact of moving targets increases over time. Furthermore, the overall trend of change is similar across different vehicle-mounted cameras. Combined with the grayscale and positional characteristics of the moving targets in the detection images from the robotic arm camera, the degree of target interference is comprehensively quantified. This allows for the screening of interfering targets, and the acquisition of interference sequences based on movement sequences comprehensively reflects the changes in the distance between the interfering targets and the excavator. This provides a foundation for subsequent early warnings regarding the potential collision between interfering targets and the excavator.

[0089] Step S004: Based on the interference target and its interference sequence, a collision warning is issued through an audible and visual alarm to achieve safety monitoring of excavator operation.

[0090] It should be noted that after identifying the interference target, it is necessary to determine the proximity of the interference target to the excavator operation by combining the interference sequence. When the distance reaches a certain range and there is a risk of collision, an early warning should be issued through an audible and visual alarm. At the same time, the size of the interference target needs to be considered. Even if a small interference target collides with the excavator, it will not affect the safety of the operation. In this case, the collision risk is further judged based on the size of the area where the interference target is located at the maximum value of the target influence degree corresponding to each camera, so as to achieve the monitoring of the excavator's operation safety.

[0091] Specifically, for any interfering target, the degree of target influence at each moment corresponds to the value of the target influence at each vehicle-mounted camera and robotic arm camera. The moment corresponding to the maximum value of the target influence is taken as the size analysis moment of the interfering target, and the corresponding camera is taken as the size analysis camera of the interfering target. The area of ​​the interfering target in the monitoring image (detection image) of its size analysis camera at the size analysis moment is obtained. The ratio of the area of ​​the region to the area of ​​the entire monitoring image (detection image) is taken as the maximum size performance value of the interfering target. A preset size ratio threshold is set. In this embodiment, the size ratio threshold is described as 0.01. If the maximum size performance value is less than the size ratio threshold, the interfering target is deleted, that is, it is no longer considered as an interfering target for collision judgment because its size is too small. All interfering targets are screened by size to obtain a number of screened interfering targets.

[0092] Furthermore, for any selected interference target, a collision risk threshold has been set during the excavator operation. This threshold is a well-known parameter in the field and will not be elaborated upon in this embodiment. If there is an element in the interference sequence of the interference target that is less than the collision risk threshold, a collision signal is transmitted to the audible and visual alarm at the corresponding moment of that element to issue a collision warning. If there is no element in the interference sequence of the interference target that is less than the collision risk threshold, it is necessary to analyze the changing trend of the interference sequence. If the elements in the interference sequence are generally decreasing and each subsequent element is less than or equal to the previous element, a collision warning is also issued for the interference target. In other cases, no collision warning is required, thus achieving safe monitoring of the excavator operation.

[0093] Thus, by initially analyzing moving targets through multi-directional vehicle-mounted cameras and combining this with 3D point cloud data from LiDAR for stereoscopic information analysis, and comprehensively considering the spatiotemporal synchronization of computer vision and LiDAR information, the degree of target influence of the moving target under the corresponding cameras at each moment is obtained. Furthermore, by combining the similarity between the overall changing trends of the target influence under multi-directional cameras, the degree of target interference is comprehensively quantified to filter interfering targets. Combined with the interference sequence reflecting the distance changes between interfering targets and excavators, safety monitoring of the excavator operation process is achieved. By combining the visual assistance and blind spot filling of multi-directional cameras with the target distance monitoring of 3D point cloud data from LiDAR, collision risk monitoring and early warning are carried out efficiently and timely, effectively ensuring the safety of excavator operation.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for monitoring the safety of excavator operations based on vision assistance and lidar, characterized in that, The method includes the following steps: The system acquires continuous frame monitoring images of the excavator during operation through vehicle-mounted cameras, obtains three-dimensional point cloud data at several moments through lidar, and simultaneously acquires continuous frame detection images from the robotic arm camera. The grayscale and area changes of each region in the continuous frame monitoring images are analyzed to obtain the target movement degree of each region. Several moving targets are selected and their movement sequences are obtained. The elements in the movement sequence are the distances between the moving targets and the vehicle-mounted camera. The monitoring images and 3D point cloud data are fused based on timestamps. The 3D representation degree is quantified according to the distribution of the moving targets in the 3D point cloud data. The 3D representation degree and the weight of the grayscale complexity factor in the monitoring images are adjusted in combination with the movement sequence to obtain the target influence degree of each moving target at each time. The differences in the degree of target influence of the same moving target under different vehicle-mounted cameras were analyzed. The grayscale and position of the moving target in the detection image of the robotic arm camera at the corresponding time were combined to obtain the comprehensive degree of influence of the moving target. Interference targets were screened and combined with the movement sequence corresponding to each vehicle-mounted camera to obtain the interference sequence of the interference targets. Based on the interfering target and its interference sequence, a collision warning is issued via an audible and visual alarm. The specific methods for obtaining the target influence degree of each moving target at each time point include: Analyze the point cloud distribution of moving targets in the spatial region of 3D point cloud data to obtain the three-dimensional representation of each moving target at each time; obtain the target movement degree of any moving target at each analysis time under any vehicle-mounted camera. Based on the mean and standard deviation of gray values ​​of all pixels in the corresponding region of the monitoring image of the moving target under the vehicle camera at any analysis time, the gray complexity factor of the moving target under the vehicle camera at that analysis time is obtained. The gray complexity factor is positively correlated with the mean and standard deviation of the gray values. No. The first moving target was captured by the vehicle-mounted camera. The degree of target influence at each analysis time point The calculation method is as follows: in, Indicates the first The first moving target was captured by the vehicle-mounted camera. The degree of target movement at each analysis time point Indicates the first The first moving target was captured by the vehicle-mounted camera. The degree of three-dimensional representation at each moment of analysis Indicates the first The first moving target was captured by the vehicle-mounted camera. Gray-level complexity factor at each analysis time point Indicates the first The moving target in the movement sequence of the vehicle-mounted camera is the [number]th ... Elements at each analysis moment Indicates the first The maximum value of all elements in the movement sequence of the moving target in the vehicle-mounted camera; Indicates the first The first moving target was captured by the vehicle-mounted camera. The weights of gray-scale complexity factors at each analysis time point; The specific methods for obtaining the three-dimensional representation of each moving target at each time point are as follows: The spatial region of any moving target in the 3D point cloud data at any analysis time under any vehicle-mounted camera is obtained. The variance of the coordinates of all points in the spatial region in three dimensions is calculated respectively to obtain the coordinate variance of the spatial region in each of the three dimensions. The ratio of the minimum value to the maximum value of the coordinate variance of the three dimensions is obtained as the degree of three-dimensional representation of the moving target at the analysis time.

2. The excavator operation safety monitoring method based on vision assistance and lidar according to claim 1, characterized in that, The specific methods for obtaining the target movement degree in each region are as follows: For any consecutive frames of monitoring images from a vehicle-mounted camera, several regions are obtained. The same region corresponding to different frames of monitoring images is obtained. Several frames of monitoring images in which the region exists are obtained from the vehicle-mounted camera and used as the monitoring images of the region in the vehicle-mounted camera. The area and centroid coordinates of the corresponding region in each monitoring image are obtained, and the average gray value of all pixels in the corresponding region is used as the gray value of the corresponding region. Based on the area difference, grayscale value difference, and centroid coordinate distance of the corresponding area in the two adjacent monitoring images, the area change, grayscale change, and movement distance of the region in the next monitoring image are obtained respectively. The diagonal length of the smallest bounding rectangle of the corresponding area in the next monitoring image is taken as the maximum movement distance of the region in the next monitoring image. No. Target movement degree in each area The calculation method is as follows: in, Indicates the first The number of monitoring images in each area Indicates the first The region in the first The frame monitors the distance the image moves. Indicates the first The region in the first The maximum movement distance of the frame monitoring image. Indicates the first The region in the first Frame monitoring of image grayscale changes This represents the maximum grayscale value. Indicates the first The region in the first The degree of area change in the frame-monitored image.

3. The excavator operation safety monitoring method based on vision assistance and lidar according to claim 2, characterized in that, The specific method for filtering to obtain several moving targets and their movement sequences includes: The region where the target movement exceeds the movement threshold is taken as the moving target; the distance between the centroid of the moving target and the vehicle camera at any given moment in any monitoring image of any vehicle camera is obtained and used as an element in the movement sequence of the moving target at the corresponding moment in that monitoring image. The time corresponding to each monitoring image of any moving target under the vehicle-mounted camera is taken as the analysis time of the moving target under the vehicle-mounted camera. The elements of each analysis time in the movement sequence of the moving target are obtained, which constitute the movement sequence of the moving target under the vehicle-mounted camera.

4. The excavator operation safety monitoring method based on vision assistance and lidar according to claim 2, characterized in that, The specific method for fusing monitoring images and 3D point cloud data based on timestamps includes: Each consecutive frame of the monitoring image from any vehicle-mounted camera corresponds to a collection time. The monitoring image from the vehicle-mounted camera at any time is fused with the 3D point cloud data at that time to obtain the spatial region of each moving target in the 3D point cloud data at that time.

5. The excavator operation safety monitoring method based on vision assistance and lidar according to claim 1, characterized in that, The specific methods for obtaining the overall impact level of the moving target include: Based on the degree of target influence of the same moving target under different vehicle cameras at the same time, we obtain several monitoring times of the moving target under each vehicle camera, as well as the similarity of the monitoring changes of the moving target. Obtain all monitoring times of any moving target from each vehicle-mounted camera, and sort them chronologically as several movement times of the target; obtain the target influence degree of the target at each movement time corresponding to the robotic arm camera, and the i-th... The combined impact of individual moving targets The calculation method is as follows: in, Indicates the first The number of movement moments of a moving target Indicates the first The first moving target The difference between the maximum and minimum values ​​of the target influence degree corresponding to each vehicle camera at each moment of movement. Indicates the first Similarity of changes in the monitoring of individual moving targets Indicates the first The first moving target The degree of target influence corresponding to the robotic arm camera at each moment of movement.

6. The excavator operation safety monitoring method based on vision assistance and lidar according to claim 5, characterized in that, The specific methods for obtaining the moving target at several monitoring times of each vehicle-mounted camera, and the similarity of the monitoring changes of the moving target, include: The same moving target is obtained under the monitoring images of different vehicle cameras; for any moving target, the target influence degree of the moving target at each time under each vehicle camera is obtained; for any vehicle camera, the time when the target influence degree of the moving target corresponding to that vehicle camera is the maximum value among the target influence degrees corresponding to each vehicle camera is taken as the monitoring time of the moving target at that vehicle camera. The moving target is captured at several monitoring times of the vehicle-mounted camera, and the target influence of the moving target at each monitoring time is arranged according to the chronological order of the monitoring times to obtain the target monitoring sequence of the moving target at the vehicle-mounted camera. Obtain the target monitoring sequence of the moving target in each vehicle camera, calculate the DTW distance of the moving target in any two vehicle camera target monitoring sequences, and use the inverse proportional normalized value of the mean DTW distance obtained between the target monitoring sequences as the monitoring change similarity of the moving target.

7. The excavator operation safety monitoring method based on vision assistance and lidar according to claim 1, characterized in that, The specific method for obtaining the interference sequence of the interference target is as follows: For any interfering target, the minimum value among the elements of the movement sequence of each vehicle-mounted camera and robotic arm camera corresponding to the interfering target at any given time is taken as the element of the interfering target's interference sequence at that time. This process is used to obtain the elements of the interfering target's interference sequence at each time, thus forming the interfering sequence of the interfering target.

8. The excavator operation safety monitoring method based on vision assistance and lidar according to claim 1, characterized in that, The specific method for providing collision warning via an audible and visual alarm based on the interfering target and its interference sequence includes: For any interfering target, the degree of target influence at each time is corresponding to each vehicle-mounted camera and robotic arm camera. The time corresponding to the maximum value of the target influence is taken as the size analysis time of the interfering target, and the corresponding camera is taken as the size analysis camera of the interfering target. Obtain the area of ​​the interfering target in the monitoring image of the size analysis camera at the time of size analysis, and take the ratio of the area of ​​the region to the area of ​​the entire monitoring image as the maximum size performance value of the interfering target; if the maximum size performance value is less than the size ratio threshold, delete the interfering target; perform size screening on all interfering targets to obtain a number of screened interfering targets; Based on the screened interference targets and their interference sequences, a collision risk threshold is used to determine the collision risk, and a collision signal is transmitted to the audible and visual alarm for collision warning.

Images