Carriage three-dimensional attitude estimation system and method based on monocular vision

Abstract

The invention discloses a carriage three-dimensional attitude estimation system and method based on monocular vision, and belongs to the technical field of carriage three-dimensional attitude estimation. Comprising a server intelligent processing module which can be placed in a cab of engineering mechanical equipment, the compartment attitude data acquisition module is connected with the server intelligent processing module through data transmission; the edge end monocular sensing module can be fixedly mounted at the top end of a cockpit of engineering mechanical equipment and is connected with the server intelligent processing module through a USB (Universal Serial Bus); the method can effectively solve the problems that an operator is difficult to accurately sense the attitude of the carriage in the remote control process and the operation efficiency and safety are affected, can enhance the universality of the carriage type, improves the robustness of the attitude estimation process, and can effectively reduce the engineering cost.

Description

Technical Field

[0001] The present invention belongs to the technical field of carriage three-dimensional posture estimation, and in particular relates to a system and method for carriage three-dimensional posture estimation based on monocular vision. Background Art

[0002] In the field of construction engineering, the harsh environment of construction sites and the increasingly scarce number of professional operators have become major challenges. To address these issues, remote control of construction machinery technology has emerged, allowing operators to remotely control equipment through wired or wireless connections in a safe indoor environment, significantly improving work safety. Currently, remotely operated construction machinery usually relies on multiple cameras to provide visual information of the surrounding environment, but relying solely on the camera's perspective often has limitations. In particular, when there is a lack of three-dimensional spatial information of the truck compartment, it is difficult for the operator to accurately perceive the posture of the compartment during remote control, affecting work efficiency and safety. Therefore, obtaining 3D posture information of the truck compartment is key to improving the accuracy and safety of remote operations.

[0003] With the development of sensing technology, researchers have attempted to use different sensors to perceive the three-dimensional state of truck compartments. For example, Chinese invention patent application CN118047239A discloses a loading control method, device, electronic device, and excavator. These methods use the longitude, latitude, elevation, and attitude angle of the truck body to determine its position and attitude in the world coordinate system in real time. Based on the deviation between the real-time attitude angle and the preset attitude angle and the truck compartment, the real-time position and attitude angle are adjusted to obtain the truck compartment's accurate position and attitude. Finally, the truck compartment's dimensional information, target position, and target attitude are combined to generate the truck compartment's attitude information. However, this method requires a lot of prior information and is only applicable to fixed-type trucks. Furthermore, algorithms based on longitude and latitude often have large deviations when determining the truck's attitude.

[0004] For example, Chinese utility model patent CN221979100U discloses a collision avoidance system for electric shovels and trucks. Terminals installed on the shovel and truck respectively collect 3D coordinate and posture information from each, and transmit this data to a control terminal for real-time processing to predict and prevent collisions between the shovel and truck during operation. However, this method involves installing an RTK on the bucket, which can cause damage to the equipment during operation, and RTK is expensive.

[0005] For example, Chinese invention patent document CN115170648B discloses a method and device for determining the position and posture of a mining truck compartment based on a deep convolutional neural network and a binocular camera. This method uses a deep learning model to perform image recognition by acquiring the left-eye color image and corresponding point cloud data captured by the binocular camera on the excavator. The method then uses preset position deviation parameters to adjust the coordinates and posture angles, ultimately generating accurate position and posture information for the mining truck compartment. However, this method requires attaching key point markers to the sides of the compartment, which has significant limitations in actual engineering operations. The markers are easily damaged, and the measurement range is limited by the marker size.

[0006] In summary, existing truck cabin posture estimation methods have limited applicability, high costs, and fragile equipment.

[0007] Therefore, there is a need for a monocular vision-based three-dimensional posture estimation system and method for the carriage that can enhance the versatility of carriage types, improve the robustness of the posture estimation process, and effectively reduce engineering costs. Summary of the Invention

[0008] The purpose of the present invention is to provide a system and method for estimating the three-dimensional posture of a vehicle body based on monocular vision, which can effectively solve the problem that the operator has difficulty in accurately perceiving the posture of the vehicle body during remote control, which affects the work efficiency and safety. It can enhance the versatility of the vehicle body types, improve the robustness of the posture estimation process, and effectively reduce engineering costs.

[0009] To achieve the above object, the technical solution adopted by the present invention is:

[0010] A car body 3D posture estimation system based on monocular vision includes a server-side intelligent processing module that can be placed in the cab of engineering machinery equipment, a car body posture data acquisition module connected to the server-side intelligent processing module through data transmission, and an edge-side monocular perception module that can be fixedly installed on the top of the cab of the engineering machinery equipment and connected to the server-side intelligent processing module through USB; wherein, the server-side intelligent processing module internally integrates a car body motion structure recovery module and a car body 3D posture estimation module for performing car body motion structure recovery and real-time estimation of the car body 3D posture; the car body posture data acquisition module includes a target 3D posture data acquisition application embedded in a mobile phone device, which is used to collect car body 3D bounding box posture data and transmit it to the server-side intelligent processing module; the edge-side monocular perception module includes a monocular camera whose output end is connected to the server-side intelligent processing module through USB, which is used to collect images of the truck and the surrounding environment during actual operation, and provide image information for the car body 3D posture estimation module.

[0011] Further improvements to the technical solution of the present invention are: the car motion structure recovery module includes the structure-from-motion recovery technology SfM, a feature extractor and a feature matcher, which are used to perform car motion structure recovery on the data set collected by the car posture data acquisition module, construct the car's 3D point cloud and 2D-3D feature trajectory map; the car three-dimensional posture estimation module includes a 2D-3D matching network model, which estimates the three-dimensional posture of the truck car in the camera coordinate system in real time through car target detection, 2D-3D matching and PnP posture solution.

[0012] A method for estimating the three-dimensional posture of a vehicle compartment based on monocular vision uses an estimation system to estimate the three-dimensional posture of the vehicle compartment, including the following steps:

[0013] Step S1: Install the edge-end monocular perception module on top of the cab of the construction machinery equipment, turn on the edge-end processor power, and put it into a waiting state, waiting to establish a connection with the server-end intelligent processing module;

[0014] Step S2: The server-side intelligent processing module is placed in the cab of the engineering machinery equipment, and a connection is established with the edge-side monocular perception module via USB to receive image information from the edge-side monocular perception module;

[0015] Step S3: The carriage posture data acquisition module collects the carriage 3D bounding box posture data and transmits it to the server-side intelligent processing module;

[0016] Step S4: The carriage motion structure is restored by the carriage motion structure recovery module on the server-side intelligent processing module;

[0017] Step S5: Carriage target detection is performed by the carriage three-dimensional posture estimation module on the server-side intelligent processing module;

[0018] Step S6: Perform 2D-3D matching through the vehicle three-dimensional posture estimation module on the server-side intelligent processing module;

[0019] Step S7: Perform PnP pose calculation through the three-dimensional pose estimation module of the vehicle body on the server-side intelligent processing module.

[0020] A further improvement of the technical solution of the present invention is that: in step S3, the truck is parked on a flat bottom surface, and the three-dimensional bounding box B of the truck is accurately marked in the augmented reality environment using the target three-dimensional posture data acquisition module embedded in the mobile device. 3D The system uses AR tools to track the camera pose and adjust the center, size, and rotation angle of the bounding box to ensure that it completely covers the car. It then collects video, 3D bounding box, camera pose, and camera intrinsic parameter data for each frame around the car, and transmits the above car pose data set to the server-side intelligent processing module.

[0021] A further improvement of the technical solution of the present invention is that: in step S4, after the server-side intelligent processing module receives the car posture data set collected in step S3, it uses the car motion structure recovery module to reconstruct a sparse point cloud through the structure-from-motion recovery technology SfM, and achieves consistent alignment between different car video sequences by converting the camera posture into the 3D bounding box posture of the car. Then, a feature extractor is applied to extract 2D key points in the image, and then a feature matcher is used to find corresponding points between images to determine their positions in three-dimensional space. By bundling and adjusting BA, the positions of these feature points are optimized to construct a three-dimensional sparse point cloud of the car, in which each point is associated with a 2D key point and a 3D descriptor to form a corresponding graph with special trajectory information.

[0022] A further improvement of the technical solution of the present invention is that in step S4, the carriage motion structure recovery includes the following steps:

[0023] Step S401: Multi-sequence alignment: After inputting multiple car data sequences, the recorded camera pose is first converted into the pose of the car's 3D bounding box to align the multiple sequences. Then, the positions of these feature points are optimized by bundling adjustment BA to overcome the drift that may occur when tracking the camera pose while ensuring the B-alignment between sequences. 3D consistent alignment;

[0024] Step S402: Carriage kinematic structure recovery: First, a lightweight feature extraction algorithm is used to extract 2D key points and their corresponding descriptors from each image. Then, a lightweight feature matcher is used to find and determine the correspondence between different images, thereby forming key point correspondences on the 2D plane. Based on these 2D correspondences, triangulation technology is used to accurately determine the positions of these feature points in 3D space. The positions of these feature points are then optimized using the bundle adjustment (BA) algorithm to construct a sparse point cloud of the car.

[0025] Step S403: Constructing a 2D-3D trajectory graph: In the point cloud, each reconstructed 3D point corresponds to a set of 2D key points and descriptors, forming a correspondence graph with feature trajectory information.

[0026] A further improvement to the technical solution of the present invention is that in step S5, a lightweight and efficient object detection network is used to detect the car body. After the car body is detected, the bounding box is cropped and adjusted to a fixed size to ensure applicability in data collected by different cameras. A feature extractor is then used to obtain feature points and descriptors. Finally, these data are input into the 2D-3D matching network model to eliminate the influence of background noise on the subsequent 2D-3D matching steps.

[0027] A further improvement of the technical solution of the present invention is that: in step S6, after obtaining the sparse point cloud of the car, the detected 2D feature descriptor of the car and the 3D feature descriptor of the car sparse point cloud are used to be fed into the 2D-3D matching network model of the car three-dimensional posture estimation module for training; an aggregation-attention layer is used to adaptively select the most informative 2D features for aggregation to initialize the 3D descriptor of the car, effectively converting from 2D feature descriptors to 3D feature descriptors and retaining valid information; by introducing self-attention layers and cross-attention layers, the interaction between 2D and 3D descriptors is optimized to improve matching accuracy.

[0028] A further improvement of the technical solution of the present invention is that in step S6, the 2D-3D matching includes the following steps:

[0029] Step S601: Constructing a 2D-3D matching network model: Adaptive aggregation is performed using an aggregation-attention layer to update the 3D descriptor of the vehicle cabin. The aggregation-attention layer uses a weight matrix W to implement adaptive weight distribution for each independent feature trajectory, thereby defining an operating mechanism. In this way, the aggregation-attention layer can select and enhance important features from the 2D feature descriptor, providing accurate information for the 3D descriptor. On this basis, the self-attention layer and the cross-attention layer are integrated to process and optimize the aggregated 3D descriptor and the 2D descriptor in the query image.

[0030] Step S602: Select the loss function of the 2D-3D matching network model and train it: introduce the double SoftMax function to extract the prediction confidence from the 2D to 3D matching. First, calculate the matching score matrix between the 2D features of the query image and the 3D features of the car point cloud, and then calculate the matching confidence. Obtained through:

[0031]

[0032] Where q represents the 2D features of the query image, x represents the 3D features of the car point cloud, and S(·,·) represents the matching score matrix;

[0033] After selecting the confidence threshold T, becomes a permutation matrix Represents 2D-3D matching prediction; through The PnP algorithm with RANSAC can be used to calculate the cabin pose in the camera coordinate system, and the loss function L used is based on the confidence score output by the double SoftMax Use focal loss for calculation and optimization:

[0034]

[0035] A further improvement of the technical solution of the present invention is that: in step S7, after the 2D-3D matching network model is trained, the query image is subjected to target detection, the detected car is cropped and mapped to a fixed size and then input into the network to obtain the 2D to 3D matching prediction result, and then the PnP method enhanced by the RANSAC algorithm is used to accurately calculate the position and posture of the car in the query image in the camera coordinate system.

[0036] Due to the adoption of the above technical solution, the technical advancements achieved by the present invention are:

[0037] The monocular vision-based three-dimensional posture estimation system and method of the vehicle body of the present invention can effectively solve the problem that the operator has difficulty in accurately perceiving the posture of the vehicle body during remote control, which affects work efficiency and safety. It can enhance the versatility of vehicle body types, improve the robustness of the posture estimation process, and effectively reduce engineering costs.

[0038] This method requires only simple surround video capture of the truck cabin to build a sparse SfM model of the cabin. Using a graph attention network, the two-dimensional points of interest in the query image are matched with the three-dimensional points of interest in the SfM model, enabling effective pose estimation of the cabin. This method matches 2D and 3D feature points, making it applicable to any type of cabin and possessing universal applicability. Furthermore, the method requires only a single monocular camera for pose estimation, significantly reducing engineering costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of the carriage three-dimensional posture data collection process in the present invention;

[0040] Figure 2 This is a schematic diagram of constructing a sparse point cloud of a carriage in the present invention;

[0041] Figure 3 This is a hardware structure diagram of the vehicle three-dimensional posture estimation in the present invention;

[0042] Figure 4 This is a diagram of the network structure for estimating the three-dimensional posture of the vehicle compartment in the present invention;

[0043] Among them, 1. Carriage posture data acquisition module, 2. Carriage motion structure recovery module, 3. Edge-end monocular perception module, 4. Server-end intelligent processing module. DETAILED DESCRIPTION

[0044] The present invention is described in further detail below in conjunction with the embodiments:

[0045] Example 1

[0046] like Figure 1-Figure 3As shown, this embodiment provides a car three-dimensional posture estimation system based on monocular vision, including a server-side intelligent processing module 4, a car posture data acquisition module 1, an edge-end monocular perception module 3, a car motion structure recovery module 2 and a car three-dimensional posture estimation module.

[0047] Among them, the car posture data acquisition module 1 is connected to the server-side intelligent processing module 4 through data transmission. It includes a target three-dimensional posture data acquisition application embedded in the mobile device, which is used to collect the three-dimensional bounding box posture data of the car and transmit it to the server-side intelligent processing module 4. That is, by collecting photos of the car and recording the camera posture, camera internal parameters and the eight corner point coordinates of the 3D bounding box of each frame, the subsequent car motion structure recovery module 2 is used to restore the car's motion structure, construct the car's 3D point cloud and construct a 2D-3D trajectory map. The mobile device is an iPhone 14Pro mobile phone.

[0048] The edge-side monocular perception module 3 can be fixedly mounted on top of the construction machinery's cockpit. It includes a monocular camera, whose output is connected to the server-side intelligent processing module 4 via USB. During actual operation, it captures images of the truck and its surroundings, providing image information for the vehicle cabin's 3D pose estimation module. The monocular camera has a resolution of 1280 (H) x 720 (V) and a frame rate of 20 fps.

[0049] The server-side intelligent processing module 4 can be placed in the cab of the construction machinery. It integrates the cabin motion structure recovery module 2 and the cabin 3D pose estimation module, which are used to perform cabin motion structure recovery and real-time estimation of the cabin's 3D pose. The cabin motion structure recovery module 2 includes structure-from-motion recovery technology (SfM), a feature extractor, and a feature matcher. It is used to perform cabin motion structure recovery on the dataset collected by the cabin pose data acquisition module 1, constructing a 3D point cloud of the cabin and a 2D-3D feature trajectory map. The cabin 3D pose estimation module includes a 2D-3D matching network model, which uses cabin target detection, 2D-3D matching, and PnP pose solution to estimate the 3D pose of the truck cabin in the camera coordinate system in real time, with a certain degree of robustness against occlusion and truncation. The server-side intelligent processing module 4 is a Jetson AGX Orin NX with 16GB of RAM.

[0050] Example 2

[0051] This embodiment provides a method for estimating the three-dimensional posture of a vehicle compartment based on monocular vision, which uses the estimation system in Example 1 to estimate the three-dimensional posture of the vehicle compartment, including the following steps:

[0052] Step S1: Install the edge-end monocular perception module 3 on top of the cab of the construction machinery equipment, turn on the edge-end processor power, and put it into a waiting state, waiting to establish a connection with the server-end intelligent processing module 4;

[0053] Step S2: The server-side intelligent processing module 4 is placed in the cab of the engineering machinery equipment, and a connection is established with the edge-side monocular perception module 3 via USB to receive image information from the edge-side monocular perception module 3;

[0054] Step S3: The carriage posture data acquisition module 1 collects the carriage three-dimensional bounding box posture data and transmits it to the server-side intelligent processing module 4;

[0055] Specifically, such as Figure 1 As shown, the truck is parked on a flat bottom surface. First, the vehicle body posture data acquisition module 1 is used to accurately mark the three-dimensional bounding box B of the vehicle body in the augmented reality environment through the target three-dimensional posture data acquisition application embedded in the mobile device. 3D The AR tool tracks the camera pose and adjusts the center, size, and rotation angle of the bounding box to ensure that it completely covers the car. It then collects video, 3D bounding box, camera pose, and camera intrinsic parameter data for each frame around the car, and transmits the above car pose data set to the server-side intelligent processing module 4.

[0056] Step S4: The carriage motion structure is restored by the carriage motion structure recovery module 2 on the server-side intelligent processing module 4. The process is as follows: after the server-side intelligent processing module 4 receives the carriage posture data set collected in step S3, the carriage motion structure recovery module 2 is used to reconstruct the sparse point cloud through the structure motion recovery technology SfM. The consistency alignment between different carriage video sequences is achieved by converting the camera posture into the 3D bounding box posture of the carriage. Then, the feature extractor is applied to extract the 2D key points in the image, and the feature matcher is used to find the corresponding points between the images to determine their positions in three-dimensional space. By bundling and adjusting BA, the positions of these feature points are optimized to construct a three-dimensional sparse point cloud of the carriage. The process is as follows: Figure 2 As shown, each point is associated with a 2D keypoint and a 3D descriptor to form a correspondence map with special trajectory information;

[0057] The restoration of the carriage motion structure specifically includes the following steps:

[0058] Step S401: Multiple sequence alignment: In order to overcome the drift phenomenon that may occur when tracking the camera posture, multiple car data sequences are input. However, this operation also leads to B 3D (size, direction) inconsistency, therefore, for the B 3DConsistent alignment, after inputting multiple cabin data sequences, we first convert the recorded camera poses into the poses of the cabin's 3D bounding box to align multiple sequences, and then optimize the positions of these feature points through bundle adjustment BA;

[0059] Step S402: Carriage kinematic structure recovery: First, a lightweight feature extraction algorithm is used to extract 2D key points and their corresponding descriptors from each image. Then, a state-of-the-art lightweight feature matcher is used to find and determine the correspondences between different images, thereby forming key point correspondences on the 2D plane. Based on these 2D correspondences, triangulation techniques are used to accurately determine the positions of these feature points in 3D space. The positions of these feature points are then optimized using the bundle adjustment (BA) algorithm to construct a sparse point cloud of the car.

[0060] Step S403: Constructing a 2D-3D trajectory graph: In the point cloud, each reconstructed 3D point corresponds to a set of 2D key points and descriptors, forming a correspondence graph with feature trajectory information;

[0061] Step S5: Carriage target detection is performed through the car 3D pose estimation module on the server-side intelligent processing module 4: In order to eliminate the influence of background noise on the subsequent 2D-3D matching step, a lightweight and efficient target detection network is used to detect the car. After the car is detected, the bounding box is cropped and adjusted to a fixed size to ensure applicability in data collected by different cameras. Feature points and descriptors are then obtained using a feature extractor, and finally these data are input into the 2D-3D matching network model.

[0062] Step S6: 2D-3D matching is performed through the car body three-dimensional posture estimation module on the server-side intelligent processing module 4. The process is as follows: after obtaining the sparse point cloud of the car body, the detected 2D feature descriptor of the car body and the 3D feature descriptor of the car body sparse point cloud are used to feed them into the 2D-3D matching network model of the car body three-dimensional posture estimation module for training; in order to effectively convert from 2D feature descriptors to 3D feature descriptors and retain as much valid information as possible, an aggregation-attention layer is used to adaptively select the most informative 2D features for aggregation to initialize the 3D descriptor of the car body; by introducing the self-attention layer and the cross-attention layer, the network can optimize the interaction between 2D and 3D descriptors and improve the matching accuracy; the 2D-3D matching network hardware diagram is shown in FIG. Figure 3 As shown in the network structure diagram Figure 4 As shown;

[0063] 2D-3D matching specifically includes the following steps:

[0064] Step S601: Constructing a 2D-3D matching network model: Adaptive aggregation is performed using an aggregation-attention layer to update the 3D descriptor of the vehicle cabin. The aggregation-attention layer uses a weight matrix W to implement adaptive weight distribution for each independent feature trajectory, thereby defining an operating mechanism. In this way, the aggregation-attention layer can effectively select and enhance important features from the 2D feature descriptor to provide more accurate information for the 3D descriptor. On this basis, the self-attention layer and the cross-attention layer are integrated to process and optimize the aggregated 3D descriptor and the 2D descriptor in the query image.

[0065] The self-attention layer focuses on the interactions between 2D descriptors or 3D descriptors. By assigning differentiated weights to each node, the model can autonomously grasp the contribution of the 2D descriptors or fused 3D descriptors in the query image to the matching task. The cross-attention layer focuses on the relationship between 2D and 3D descriptors. By measuring the attention coefficients between different descriptor pairs, the model can automatically determine which interactions between 2D and 3D descriptors are more important to the final matching task.

[0066] The 2D-3D matching network architecture consists of 4 stacked attention groups;

[0067] Step S602: Select the loss function of the 2D-3D matching network model and train it: introduce the double SoftMax function to extract the prediction confidence from the 2D to 3D matching. First, calculate the matching score matrix between the 2D features of the query image and the 3D features of the car point cloud, and then calculate the matching confidence. Obtained through:

[0068]

[0069] Where q represents the 2D features of the query image, x represents the 3D features of the car point cloud, and S(·,·) represents the matching score matrix;

[0070] After selecting the confidence threshold T, becomes a permutation matrix Represents the 2D-3D matching prediction; through The PnP algorithm with RANSAC can be used to calculate the cabin pose in the camera coordinate system, and the loss function L used is based on the confidence score output by the double SoftMax Use focal loss for calculation and optimization:

[0071]

[0072] Step S7: Perform PnP to solve the pose through the three-dimensional pose estimation module of the car body on the server-side intelligent processing module 4: After the 2D-3D matching network model is trained, the query image is subjected to target detection, and the detected car body is cropped and mapped to a fixed size and then input into the network to obtain the 2D to 3D matching prediction result. Then, the PnP method enhanced by the RANSAC algorithm is used to accurately calculate the position and pose of the car body in the query image in the camera coordinate system.

[0073] Unlike previous 3D object detection methods that require point cloud annotation for training, this method only requires simple surround video capture of the truck compartment to build a sparse SfM model of the compartment. A graph attention network is then used to match the 2D points of interest in the query image with the 3D points of interest in the established SfM model, thereby enabling effective pose estimation of the compartment. This method matches 2D feature points with 3D feature points, making it applicable to any type of compartment and possessing universal applicability. Furthermore, this method requires only a single monocular camera for pose estimation, significantly reducing engineering costs.

[0074] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A vehicle compartment three-dimensional posture estimation system based on monocular vision, characterized by: The invention comprises a server-side intelligent processing module (4) that can be placed in the cab of engineering machinery equipment, a vehicle body posture data acquisition module (1) connected to the server-side intelligent processing module (4) via data transmission, and an edge-side monocular perception module (3) that can be fixedly installed on the top of the cab of the engineering machinery equipment and connected to the server-side intelligent processing module (4) via USB; wherein the server-side intelligent processing module (4) internally integrates a vehicle body motion structure recovery module (2) and a vehicle body three-dimensional posture estimation module for performing vehicle body motion structure recovery and real-time estimation of vehicle body three-dimensional posture; the vehicle body posture data acquisition module (1) includes a target three-dimensional posture data acquisition application embedded in a mobile phone device, for performing vehicle body three-dimensional boundary box posture data acquisition and transmitting the data to the server-side intelligent processing module (4); the edge-side monocular perception module (3) includes a monocular camera whose output end is connected to the server-side intelligent processing module (4) via USB, for acquiring images of the truck and the surrounding environment during actual operation, and providing image information for the vehicle body three-dimensional posture estimation module.

2. The monocular vision-based three-dimensional posture estimation system for a vehicle compartment according to claim 1, characterized in that: The vehicle body motion structure recovery module (2) includes a structure-from-motion recovery technology (SfM), a feature extractor, and a feature matcher, and is used to perform vehicle body motion structure recovery on the data set collected by the vehicle body posture data acquisition module (1), and to construct a 3D point cloud and a 2D-3D feature trajectory map of the vehicle body; the vehicle body three-dimensional posture estimation module includes a 2D-3D matching network model, which estimates the three-dimensional posture of the truck body in the camera coordinate system in real time through vehicle body target detection, 2D-3D matching, and PnP posture solution.

3. A method for estimating the three-dimensional posture of a vehicle compartment based on monocular vision, characterized by: Using the estimation system of any one of claims 1 or 2 to estimate the three-dimensional posture of the vehicle compartment comprises the following steps: Step S1: Install the edge-end monocular perception module (3) on the top of the cab of the engineering machinery equipment, turn on the power of the edge-end processor, and put it into a waiting state, waiting to establish a connection with the server-end intelligent processing module (4); Step S2: placing the server-side intelligent processing module (4) in the cab of the engineering machinery equipment, and establishing a connection with the edge-side monocular perception module (3) via a USB to receive image information from the edge-side monocular perception module (3); Step S3: The carriage posture data acquisition module (1) acquires the carriage three-dimensional bounding box posture data and transmits it to the server-side intelligent processing module (4); Step S4: Performing carriage motion structure recovery through the carriage motion structure recovery module (2) on the server-side intelligent processing module (4); Step S5: Carriage target detection is performed by the carriage three-dimensional posture estimation module on the server-side intelligent processing module (4); Step S6: Perform 2D-3D matching through the vehicle three-dimensional posture estimation module on the server-side intelligent processing module (4); Step S7: Perform PnP pose calculation through the three-dimensional pose estimation module of the vehicle body on the server-side intelligent processing module (4).

4. The method for estimating a three-dimensional posture of a vehicle compartment based on monocular vision according to claim 3, characterized in that: In step S3, the truck is parked on a flat bottom surface. First, the vehicle body posture data acquisition module (1) is used to accurately mark the three-dimensional bounding box B of the vehicle body in the augmented reality environment through the target three-dimensional posture data acquisition application embedded in the mobile device. 3D The AR tool is used to track the camera posture, adjust the center, size and rotation angle of the bounding box to ensure that it completely covers the car, and then collect video, 3D bounding box, camera posture and camera intrinsic parameter data of each frame around the car, and transmit the above car posture data set to the server-side intelligent processing module (4).

5. The method for estimating the three-dimensional posture of a vehicle compartment based on monocular vision according to claim 4, characterized in that: In step S4, after the server-side intelligent processing module (4) receives the car posture data set collected in step S3, it uses the car motion structure recovery module (2) to reconstruct a sparse point cloud using the structure-from-motion recovery technology SfM. By converting the camera posture into the 3D bounding box posture of the car, the consistent alignment between different car video sequences is achieved. Then, the feature extractor is applied to extract the 2D key points in the image, and the feature matcher is used to find the corresponding points between the images to determine their positions in the three-dimensional space. By bundling and adjusting BA, the positions of these feature points are optimized to construct a three-dimensional sparse point cloud of the car, in which each point is associated with a 2D key point and a 3D descriptor to form a corresponding map with special trajectory information.

6. The method for estimating a three-dimensional posture of a vehicle compartment based on monocular vision according to claim 5, characterized in that: In step S4, the carriage motion structure recovery includes the following steps: Step S401: Multi-sequence alignment: After inputting multiple car data sequences, the recorded camera pose is first converted into the pose of the car's 3D bounding box to align the multiple sequences. Then, the positions of these feature points are optimized by bundling adjustment BA to overcome the drift that may occur when tracking the camera pose while ensuring the B-alignment between sequences. 3D consistent alignment; Step S402: Carriage kinematic structure recovery: First, a lightweight feature extraction algorithm is used to extract 2D key points and their corresponding descriptors from each image. Then, a lightweight feature matcher is used to find and determine the correspondence between different images, thereby forming key point correspondences on the 2D plane. Based on these 2D correspondences, triangulation technology is used to accurately determine the positions of these feature points in 3D space. The positions of these feature points are then optimized using the bundle adjustment (BA) algorithm to construct a sparse point cloud of the car. Step S403: Constructing a 2D-3D trajectory graph: In the point cloud, each reconstructed 3D point corresponds to a set of 2D key points and descriptors, forming a correspondence graph with feature trajectory information.

7. The method for estimating a three-dimensional posture of a vehicle compartment based on monocular vision according to claim 6, characterized in that: In step S5, a lightweight and efficient object detection network is used to detect the car. After the car is detected, the bounding box is cropped and resized to a fixed size to ensure applicability in data collected by different cameras. A feature extractor is then used to obtain feature points and descriptors. Finally, this data is input into the 2D-3D matching network model to eliminate the impact of background noise on the subsequent 2D-3D matching steps.

8. The method for estimating a three-dimensional posture of a vehicle compartment based on monocular vision according to claim 7, characterized in that: In step S6, after obtaining the sparse point cloud of the car, the detected 2D feature descriptor of the car and the 3D feature descriptor of the car sparse point cloud are fed into the 2D-3D matching network model of the car three-dimensional posture estimation module for training; an aggregation-attention layer is used to adaptively select the most informative 2D features for aggregation to initialize the 3D descriptor of the car, effectively converting the 2D feature descriptor to the 3D feature descriptor and retaining the valid information; by introducing the self-attention layer and the cross-attention layer, the interaction between the 2D and 3D descriptors is optimized to improve the matching accuracy.

9. The method for estimating a three-dimensional posture of a vehicle compartment based on monocular vision according to claim 8, characterized in that: In step S6, 2D-3D matching includes the following steps: Step S601: Constructing a 2D-3D matching network model: Adaptive aggregation is performed using an aggregation-attention layer to update the 3D descriptor of the vehicle cabin. The aggregation-attention layer uses a weight matrix W to implement adaptive weight distribution for each independent feature trajectory, thereby defining an operating mechanism. In this way, the aggregation-attention layer can select and enhance important features from the 2D feature descriptor, providing accurate information for the 3D descriptor. On this basis, the self-attention layer and the cross-attention layer are integrated to process and optimize the aggregated 3D descriptor and the 2D descriptor in the query image. Step S602: Select the loss function of the 2D-3D matching network model and train it: introduce the double SoftMax function to extract the prediction confidence from the 2D to 3D matching. First, calculate the matching score matrix between the 2D features of the query image and the 3D features of the car point cloud, and then calculate the matching confidence. Obtained through: Where q represents the 2D features of the query image, x represents the 3D features of the car point cloud, and S(·,·) represents the matching score matrix; After selecting the confidence threshold T, becomes a permutation matrix Represents 2D-3D matching prediction; through The PnP algorithm with RANSAC can be used to calculate the cabin pose in the camera coordinate system, and the loss function L used is based on the confidence score output by the double SoftMax Use focal loss for calculation and optimization:

10. The method for estimating a three-dimensional posture of a vehicle compartment based on monocular vision according to claim 9, characterized in that: In step S7, after the 2D-3D matching network model is trained, the query image is subjected to target detection. The detected car is cropped and mapped to a fixed size before being input into the network to obtain the 2D to 3D matching prediction result. The PnP method enhanced by the RANSAC algorithm is then used to accurately calculate the position and posture of the car in the query image in the camera coordinate system.

Images