Monocular camera-based target interposition relationship extraction method

Abstract

This invention discloses a method for extracting the positional relationship between targets based on a monocular camera. The method includes: collecting and preprocessing a dataset to obtain a dataset containing excavators and workers; training a YOLOv8 target detection model; scene geometric feature analysis: estimating the horizon from a single image and solving for the camera's horizontal rotation angle through coordinate transformation; target detection analysis: detecting the excavator and worker in the image, extracting the worker's pixel height and the camera's installation pixel height based on the target detection and horizon recognition results, and solving for the camera's world height, vertical rotation angle, and focal length using the worker's average height as a reference. This invention enhances the determination of the relative positions of the excavator and worker, enabling better early warning for human-machine collision avoidance.

Description

Technical Field

[0001] This invention belongs to the field of machine vision technology, and in particular, it is a method for extracting the positional relationship between targets in a construction scene based on a monocular camera. Background Technology

[0002] The construction industry is vital to national economic development, but due to harsh construction environments and the high difficulty and risks of construction activities, the accident rate in the construction industry is relatively high. Among these accidents, collision-related accidents account for a large proportion, and human-machine collisions are gradually becoming the main cause of accidents on construction sites.

[0003] With the rapid development of digital technology, computer vision-based image processing technology has been widely applied in construction management due to its advantages such as convenient data collection, strong universality, and wide monitoring range. Vision-based semantic extraction methods can be divided into stereo vision and monocular vision, depending on the type of vision. Stereo vision can be applied to scene depth estimation and spatial measurement tasks using devices such as binocular cameras, but this method has high equipment costs and poor robustness. Monocular vision, on the other hand, uses a monocular camera, which has low equipment costs and is the most commonly used method for construction project monitoring. However, the monitoring data acquired by a monocular camera only contains two-dimensional pixel information, requiring camera calibration to recover three-dimensional scale information in spatial ranging and depth estimation tasks.

[0004] Currently, the main methods for camera calibration can be broadly categorized into: traditional camera calibration methods, active vision-based camera calibration methods, and camera self-calibration methods. Traditional camera calibration methods are complex and require specialized calibration objects. Active vision-based camera calibration methods require high-precision control of the camera's rotation or translation. In contrast, camera self-calibration methods do not require specific calibration objects or additional control equipment; they can solve for camera parameters by extracting the geometric attributes of the scene and applying the urban-rural model mapping principle. Summary of the Invention

[0005] The technical problem solved by this invention is to provide a method for extracting the positional relationship between an excavator and a worker based on a monocular camera.

[0006] The technical solution to achieve the purpose of this invention is: a method for extracting the positional relationship between targets in a construction scene based on a monocular camera, comprising the following steps:

[0007] Step 1: Collect and preprocess the dataset to obtain a dataset containing excavators and workers;

[0008] Step 2: Divide the dataset into a training set and a test set in a 7:3 ratio, and use the training set data to train the YOLOv8 object detection model;

[0009] Step 3: Scene geometric feature analysis. The NG-DSAC algorithm is used to estimate the horizon in the image, and the horizontal rotation angle of the camera is solved through coordinate transformation.

[0010] Step 4: Use the YOLOv8 algorithm to detect the excavator, bucket, and worker in the image. Based on the target detection and horizon recognition results, extract the worker's pixel height and the camera's installation pixel height. Use the worker's average height as a reference value to solve for the camera's world height, vertical rotation angle, and focal length.

[0011] Step 5: Obtain the camera projection matrix based on the camera parameters, solve for the three-dimensional spatial coordinates of the target in the two-dimensional image, and then calculate the actual spatial distance between the excavator and the worker;

[0012] Step 6: Taking the direction of the excavator bucket as the front, calculate the azimuth angle of the worker relative to the direction of the excavator bucket, and then determine whether the worker is in front of, to the side or behind the excavator.

[0013] Compared with the prior art, the significant advantages of this invention are: 1) This invention uses a monocular camera to capture and extract the positional relationship between targets, without the need for a binocular camera or sensor, resulting in a simple structure and low cost; 2) When extracting the positional relationship between targets, this invention considers not only the spatial distance between targets but also their relative orientation, enabling more accurate early warning of dangerous situations.

[0014] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0015] Figure 1 This is a flowchart of a method for extracting the positional relationship between targets in a construction scene based on a monocular camera, according to the present invention. Detailed Implementation

[0016] Combination Figure 1 The present invention provides a method for extracting the positional relationship between targets in a construction scene based on a monocular camera, comprising the following steps:

[0017] Step 1: Collect and preprocess the dataset, specifically:

[0018] Step 1-1: Collect images of construction scenes containing excavators and workers;

[0019] Steps 1-2: Use the "Create Rectangle" function of the open-source annotation tool Labelme to annotate the image, labeling it with three categories: excavator, bucket, and worker.

[0020] Step 2: Train the YOLOv8 object detection model using the preprocessed dataset, specifically as follows:

[0021] Step 2-1: Divide the dataset into a training set and a test set in a 7:3 ratio;

[0022] Step 2-2: Train the YOLOv8n model structure and weight file as the baseline network and pre-trained model. Set parameters such as batch_size, optimizer, learning rate, and number of iterations. Train the model using the training set mentioned above. Finally, obtain the trained model and use the test set to judge the accuracy.

[0023] Step 3: Estimate the horizon in the image and solve for the camera's horizontal rotation angle through coordinate transformation. Specifically:

[0024] Step 3-1: Use the NG-DSAC algorithm to extract the horizon from the image and obtain the slope of the horizon;

[0025] Step 3-2: Calculate the camera's tilt angle θ relative to the horizontal plane based on the slope of the horizon. z .

[0026] Step 4: Extract camera parameters based on object detection and horizon recognition results, specifically:

[0027] Step 4-1: Use the YOLOv8 model to detect the excavator, bucket, and worker in the image to obtain the location of the target detection box;

[0028] Step 4-2: The pixel height corresponding to the world height of the camera is the vertical height from a point on the ground in the image to the horizontal line. Set the ground point in the image as the position of the worker's feet. Based on the principle of similar triangles and linear mapping in the same plane, the actual installation height of the camera can be solved by equation (1). Taking into account the average height of Chinese adult males of about 1.726m and the thickness of construction shoes, the average height of construction workers is set to 1.74m.

[0029]

[0030] The formula for converting two-dimensional coordinates to world coordinates is shown below:

[0031] c ∝ PC (2)

[0032]

[0033] tilt the camera at angle θ X Camera f and altitude H c Substituting the solution values ​​into equations (1) and (2), we can obtain the simplified projection equation as follows:

[0034]

[0035] Among them, when the camera tilt angle θ X≠π / 2, i.e., cosθ X When ≠0, the expression for the image coordinate y is simplified as follows:

[0036]

[0037] Assume the worker's head and feet are located at positions (x, y) in the image. h y h ) and (x f y f ), whose real-world locations are (X) h Y h Z h ) and (X f Y f Z f According to equation (6), the relationship between the two is:

[0038]

[0039] Because the ground points in the image are set to the worker's foot positions, Y f =0, Y h =H person Z is a constant. f =Z h For the worker in the world coordinate system Z W The coordinates of the direction are eliminated by solving a series of equations to obtain the image coordinates y of the top of the worker's head. h The expression is as follows:

[0040]

[0041] Since the detection results of the target detection analysis layer can directly determine the worker's position in the image, i.e., y h and y f It is known that equation (8) can be transformed into equation (8) containing θ. X The estimation function of the two parameters f. As shown in equation (9):

[0042]

[0043] Since images captured on-site are always accompanied by noise, the relationship between the target detection result and the estimated value is as follows:

[0044]

[0045] Where ε represents the error generated by target detection and camera calibration.

[0046] By minimizing the predicted value and the actual detection value The parameters are optimized by summing the squared errors between the parameters, and the problem is solved using a genetic algorithm.

[0047]

[0048] This allows us to obtain the camera projection matrix, which in turn enables the conversion of two-dimensional coordinates into three-dimensional world coordinates.

[0049] Step 5: Calculate the distance between the two targets in three-dimensional space, specifically:

[0050] Step 5-1: Convert the target's 2D pixel coordinates to 3D world coordinates based on the camera projection matrix:

[0051]

[0052] Step 5-2, the three-dimensional spatial coordinates of the target on the ground are (X... W ,0,Z W The relative distance between the target and reality can be calculated as follows:

[0053]

[0054] Step 6: Determine the relative positions between targets, specifically:

[0055] Step 6-1: Taking the direction of the excavator bucket as the forward direction, calculate the direction vector from the excavator's center of rotation to the bucket:

[0056]

[0057] Where (X) e Y e Z e (X) is the coordinate of the excavator's rotation center. b Y b Z b () are the coordinates of the excavator bucket.

[0058] Step 6-2: Assume the worker's position coordinates are (X... w Y w Z w ), calculate the direction vector of the worker relative to the excavator's center of rotation.

[0059]

[0060] Step 6-3: To determine the worker's position relative to the excavator bucket, calculate the worker's direction vector relative to the excavator. The directional vector of the bucket The included angle θ between them is calculated using the following formula:

[0061]

[0062] Step 6-4: By calculating the included angle θ, it can be determined whether the worker is located "in front", "to the side", or "behind" the excavator bucket, as set as follows:

[0063] (4) Front: If 0° < θ < 45°, then the worker is in front of the excavator bucket;

[0064] (5) Lateral: If 45° < θ < 135°, then the worker is on the side of the excavator bucket;

[0065] (6) Rear: If 135° < θ < 180° ° Then the worker is behind the excavator bucket;

[0066] Example

[0067] The present invention provides a method for extracting the positional relationship between targets in a construction scene based on a monocular camera, comprising the following:

[0068] 1. Collect and preprocess the dataset:

[0069] Videos of different construction stages, scenes, and shooting angles from construction sites were collected from websites. Images containing excavators and workers were extracted from these videos. Images with too small a proportion of the subject and those that were overexposed or underexposed were manually screened and removed. Images meeting the requirements were used as the raw data for the dataset.

[0070] The "Create Rectangle" function of the open-source annotation tool Labelme was used to annotate the image, creating three categories of labels: excavator, bucket, and worker.

[0071] 2. Calculate the similarity of edge nodes and group the nodes using the Fast Newman algorithm:

[0072] First, the dataset is divided into training and test sets in a 7:3 ratio. The YOLOvgn model structure and weight file are used as the baseline network and pre-trained model for training. The batch size is set to 32, the optimizer to SGD, and the number of iterations (epochs) to 200. The model is trained using the aforementioned training set. Finally, the trained model is obtained, and its accuracy is judged using the test set.

[0073] 3. Estimate the horizon in the image and solve for the camera's horizontal rotation angle through coordinate transformation:

[0074] First, the NG-DSAC algorithm is trained using the open-source dataset Horizon Lines in the Wild (HLW). The trained network is then used to predict the horizon in the image. Assuming the expression for the horizon is ax + by + c = 0, -a / b is the slope of the horizon, and the arctangent function tan... -1Converting the slope to an angle gives us the camera's tilt angle θ relative to the horizontal plane. Z :

[0075]

[0076] 4. Extract camera parameters based on object detection and horizon recognition results:

[0077] The YOLOv8 model was used to detect excavators, buckets, and workers in an image. The output is the position and dimensions of each object detection box. The output for the excavator is (x...). e ,y e w e h e The output of the bucket is (x) b y b w b h b The worker's output is (x) w ,y w w w h w Where (x, y) are the coordinates of the center point of the detection box, w is the width of the detection box, and h is the height of the detection box.

[0078] The pixel height corresponding to the camera's world height is the vertical height from a point on the ground in the image to the horizontal line. The ground point in the image is set as the position of the worker's feet. Based on the principle of similar triangles and linear mapping in the same plane, the actual installation height of the camera can be solved by equation (2). Taking into account the average height of Chinese adult males of about 1.726m and the thickness of construction shoes, the average height of construction workers, H, is set. person =1.74m.

[0079]

[0080] Among them, H person-pixel For workers to inspect the frame height h w Let the coordinates of the worker's feet be (x, y). f y f ), can be obtained from equation (3):

[0081]

[0082] Since the expression for the horizon is ax + by + c = 0, the vertical distance from the worker's feet to the horizon can be calculated using the formula for the distance from a point to a line:

[0083]

[0084] Therefore, H can be obtained according to equation (2). c .

[0085] The formula for converting two-dimensional coordinates to world coordinates is shown below:

[0086] c ∝ PC (5)

[0087]

[0088] tilt the camera at angle θ X Camera f and altitude H c Substituting the solution values ​​into equations (5) and (6), we can obtain the simplified projection equations as follows:

[0089]

[0090] Among them, (X) W Y W Z W (x, y) represents the real-world 3D coordinates, and (x, y) represents the pixel coordinates. When the camera tilt angle θ... X ≠π / 2, i.e., cosθ X When ≠0, the expression for the image coordinate y is simplified as follows:

[0091]

[0092] Assume the worker's head and feet are located at positions (x, y) in the image. h y h ) and (x f y f ), whose real-world locations are (X) h Y h Z h ) and (X f Y f Z f According to equation (6), the relationship between the two is:

[0093]

[0094] Because the ground points in the image are set to the worker's foot positions, Y f =0, Y h =H person Z is a constant. f =Z h For the worker in the world coordinate system Z W The coordinates of the direction are eliminated by solving a series of equations to obtain the image coordinates y of the top of the worker's head. h The expression is as follows:

[0095]

[0096] Because y h =y w -h / 2 and yf =y w Given +h / 2, equation (8) can be transformed into one containing θ. X The estimation function of the two parameters f. As shown in equation (9):

[0097]

[0098] Since images captured on-site are always accompanied by noise, the relationship between the target detection result and the estimated value is as follows:

[0099]

[0100] Where ε represents the error generated by target detection and camera calibration.

[0101] By minimizing the predicted value and the actual detection value The parameters are optimized by summing the squared errors between the parameters, and the problem is solved using a genetic algorithm.

[0102]

[0103] Therefore, the horizontal tilt angle θ of the camera was calculated. Z Camera vertical tilt angle θ X Camera focal length f and camera height H c This allows us to obtain the camera projection matrix, which in turn enables the conversion of two-dimensional coordinates into three-dimensional world coordinates.

[0104] 5. Calculate the distance between two targets in three-dimensional space:

[0105] Since the excavator and the worker are on the same plane, their vertical coordinates Y in three-dimensional space are... W =0, the camera projection matrix is ​​as follows:

[0106]

[0107] The pixel coordinates (x) of the worker's foot f y f ) and the pixel coordinates (x) of the bottom of the excavator e-bottom y e-bottom Substituting these values ​​into equation (15) yields their three-dimensional spatial coordinates (X). f ,0,Z f ) and (X e-bottom ,0,Z e-bottom ). Where (x e-bottom y e-bottom The following formula is used to obtain:

[0108]

[0109] Once the three-dimensional coordinates of the excavator and the worker are obtained, their relative distance in reality can be calculated:

[0110]

[0111] 6. Determine the relative positions between targets:

[0112] Taking the direction of the excavator bucket as the forward direction, the output of the YOLOv8 model for detecting the excavator bucket is (x b y b w b h b Assuming the vertical coordinate of the bucket is also 0, substitute it into equation (15) to find the three-dimensional spatial coordinates (X). b ,0,Z b Find the direction vector of the excavator from the center of rotation to the bucket.

[0113]

[0114] Calculate the direction vector of the worker relative to the excavator's center of rotation.

[0115]

[0116] To determine the worker's position relative to the excavator bucket, calculate the worker's direction vector relative to the excavator. The directional vector of the bucket The included angle θ between them is calculated using the following formula:

[0117]

[0118] By calculating the included angle θ, it can be determined whether the worker is located "in front", "to the side" or "behind" the excavator bucket: if 0° < θ < 45°, the worker is in front of the excavator bucket; if 45° < θ < 135°, the worker is to the side of the excavator bucket; if 135° < θ < 180°, the worker is behind the excavator bucket.

[0119] This invention considers the relative distance and orientation between the excavator and the worker in three-dimensional space, and can combine these distances and orientations for hazard warning. The NG-DSAC algorithm is used to extract the horizon from the image to determine the camera's horizontal tilt angle; the YOLOv8 target detection algorithm is used to detect the excavator and worker in the image, and the camera's intrinsic and extrinsic parameters are obtained based on the image information. After obtaining the camera projection matrix, the relative position can be calculated using three-dimensional coordinates, and the direction of the excavator bucket is used as the forward direction to determine the relative orientation. Compared with existing technologies, the method of this invention adds the determination of the relative orientation between targets, enabling more accurate hazard warnings.

Claims

1. A method for extracting the positional relationship between targets in a construction scene based on a monocular camera, characterized in that, Includes the following steps: Step 1: Collect and preprocess the dataset to obtain a dataset containing excavators and workers; Step 2: Divide the dataset into a training set and a test set in a 7:3 ratio, and use the training set data to train the YOLOv8 object detection model; Step 3: Scene geometric feature analysis. The NG-DSAC algorithm is used to estimate the horizon in the image, and the horizontal rotation angle of the camera is solved through coordinate transformation. Step 4: Use the YOLOv8 algorithm to detect the excavator and worker in the image. Based on the target detection and horizon recognition results, extract the worker's pixel height and the camera's installation pixel height. Use the worker's average height as a reference value to solve for the camera's world height, camera's vertical rotation angle, and camera focal length. Step 5: Obtain the camera projection matrix based on the camera parameters, solve for the three-dimensional spatial coordinates of the target in the two-dimensional image, and then calculate the actual spatial distance between the excavator and the worker; Step 6: Taking the direction of the excavator bucket as the front, calculate the azimuth angle of the worker relative to the direction of the excavator bucket, and then determine whether the worker is in front of, to the side or behind the excavator.

2. The method for extracting the positional relationship between targets in a construction scene based on a monocular camera according to claim 1, characterized in that, Collect and preprocess the dataset, specifically: Step 1-1: Collect images of construction scenes containing excavators and workers; Steps 1-2: Use the "Create Rectangle" function of the open-source annotation tool Labelme to annotate the image, labeling it with three categories: excavator, bucket, and worker.

3. The method for extracting the positional relationship between targets in a construction scene based on a monocular camera according to claim 2, characterized in that, The YOLOv8 model for object detection is trained using a preprocessed dataset, specifically as follows: Step 2-1: Divide the dataset into a training set and a test set in a 7:3 ratio; Step 2-2: Train the YOLOv8n model structure and weight file as the baseline network and pre-trained model. Set parameters such as batch_size, optimizer, learning rate, and number of iterations. Train the model using the training set mentioned above. Finally, obtain the trained model and use the test set to judge the accuracy.

4. The method for extracting the positional relationship between targets in a construction scene based on a monocular camera according to claim 3, characterized in that, To estimate the horizon in the image, the horizontal rotation angle of the camera is calculated through coordinate transformation, specifically: Step 3-1: Use the NG-DSAC algorithm to extract the horizon from the image and obtain the slope of the horizon; Step 3-2, calculating the tilt angle θ of the camera relative to the horizontal plane from the slope of the horizon Z .

5. The method for extracting the positional relationship between targets in a construction scene based on a monocular camera according to claim 4, characterized in that, Camera parameters are extracted based on the object detection and horizon recognition results, specifically as follows: Step 4-1: Use the YOLOv8 model to detect the excavator, bucket, and worker in the image to obtain the location of the target detection box; Step 4-2: The pixel height corresponding to the world height of the camera is the vertical height from a point on the ground in the image to the horizontal line. Set the ground point in the image as the position of the worker's feet. Based on the principle of similar triangles and linear mapping in the same plane, the actual installation height of the camera can be solved by equation (1). Taking into account the average height of Chinese adult males of about 1.726m and the thickness of construction shoes, the average height of construction workers is set to 1.74m. The formula for converting two-dimensional coordinates to world coordinates is shown below: c∝PC (2) The camera tilt angle θ X , the camera f and the height H c The simplified projection equation can be obtained by substituting the solution of the camera tilt angle θ X , the camera f and the height H c into the equations (1) and (2). wherein, when the camera tilt angle θ X ≠ π / 2, i.e. cos θ X ≠ 0, the image coordinate y expression is simplified as follows: Assume the worker's head and feet are located at positions (x, y) in the image. h y h ) and (x f y f ), whose real-world locations are (X) h Y h Z h ) and (X f Y f Z f According to equation (6), the relationship between the two is: Because the ground points in the image are set to the worker's foot positions, Y f =0, Y h =H person Z is a constant. f =Z h For the worker in the world coordinate system Z W The coordinates of the direction are eliminated by solving a series of equations to obtain the image coordinates y of the top of the worker's head. h The expression is as follows: Since the detection results of the target detection analysis layer can directly determine the worker's position in the image, i.e., y h and y f It is known that equation (8) can be transformed into equation (8) containing θ. X The estimation function of the two parameters f. As shown in equation (9): Since images captured on-site are always accompanied by noise, the relationship between the target detection result and the estimated value is as follows: Where ε represents the error generated by target detection and camera calibration. By minimizing the predicted value and the actual detection value The parameters are optimized by summing the squared errors between the parameters, and the problem is solved using a genetic algorithm. This allows us to obtain the camera projection matrix, which in turn enables the conversion of two-dimensional coordinates into three-dimensional world coordinates.

6. The method for extracting the positional relationship between targets in a construction scene based on a monocular camera according to claim 5, characterized in that, The three-dimensional distance between targets is calculated as follows: Step 5-1: Convert the target's 2D pixel coordinates to 3D world coordinates based on the camera projection matrix: Step 5-2, the three-dimensional spatial coordinates of the target on the ground are (X... W ,0,Z W The relative distance between the target and reality can be calculated as follows:

7. The method for extracting the positional relationship between targets in a construction scene based on a monocular camera according to claim 6, characterized in that, Determining the relative positions between targets involves: Step 6-1: Taking the direction of the excavator bucket as the forward direction, calculate the direction vector from the excavator's center of rotation to the bucket: Where (X) e Y e Z e (X) is the coordinate of the excavator's rotation center. b Y b Z b () are the coordinates of the excavator bucket. Step 6-2: Assume the worker's position coordinates are (X... w Y w Z w ), calculate the direction vector of the worker relative to the excavator's center of rotation. Step 6-3: To determine the worker's position relative to the excavator bucket, calculate the worker's direction vector relative to the excavator. The directional vector of the bucket The included angle θ between them is calculated using the following formula: Step 6-4: By calculating the included angle θ, it can be determined whether the worker is located "in front", "to the side", or "behind" the excavator bucket, as set as follows: (1) Front: If 0° < θ < 45°, then the worker is in front of the excavator bucket; (2) Lateral: If 45° < θ < 135°, then the worker is on the side of the excavator bucket; (3) Rear: If 135° < θ < 180°, then the worker is behind the excavator bucket.

Images