Monocular vision-based three-dimensional modeling method and system for complex scenes

Abstract

The application discloses a complex scene three-dimensional modeling method and system based on monocular vision. The method comprises the following steps: acquiring video streams shot by multiple cameras and pose data detected by a six-axis sensor; image sampling is performed on the video streams; a marker is used to assist in extracting and screening matched feature points from the sampled images, and the depth of the feature points is calculated according to camera parameters to determine the three-dimensional position, thereby forming a point cloud diagram around the tower crane; the point cloud diagrams around the tower crane are combined through three-dimensional affine transformation to obtain a combination result; the combination result is fitted; the fitting result is processed according to the types of large planes, small planes and discrete points to obtain a three-dimensional reconstruction model around the tower crane; and the three-dimensional reconstruction model around the tower crane is sent to a terminal to set a tower crane cargo terminal point and a running path based on the three-dimensional reconstruction model around the tower crane. The method can reduce the cost of three-dimensional reconstruction, is easy to deploy and can effectively improve the safety and accuracy of tower crane operation.

Description

TECHNICAL FIELD

[0001] The present application relates to a three-dimensional modeling method, more particularly to a three-dimensional modeling method and system for complex scenes based on monocular vision. BACKGROUND

[0002] With the increasing scale of construction projects, tower cranes, as indispensable material handling equipment in construction sites, significantly improve construction efficiency while facing unprecedented challenges in safety and accuracy of operation. In the traditional mode, the operation of tower cranes highly depends on the experience and skill level of the driver, which makes the operation process extremely susceptible to changes in the external environment and the state of the operator, leading to frequent safety hazards. Especially in high-density and variable construction site environments, relying solely on manual operation cannot ensure that each movement meets the best safety standards and path optimization.

[0003] In order to solve the above problems and further promote the automation and intelligentization of tower crane operation, three-dimensional reconstruction technology is introduced into the application scenario of tower cranes, aiming to provide accurate spatial perception capabilities to support automatic obstacle avoidance and path planning functions. However, traditional three-dimensional reconstruction methods, such as laser radar scanning, although can achieve very high measurement accuracy, but due to high cost and complex deployment requirements, its popularization and use on ordinary construction sites are greatly limited.

[0004] Therefore, it is necessary to design a new method to reduce the cost of three-dimensional reconstruction, easy to deploy and effectively improve the safety and accuracy of tower crane operation. SUMMARY

[0005] The present application aims to overcome the defects of the prior art and provide a three-dimensional modeling method and system for complex scenes based on monocular vision.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solution: a three-dimensional modeling method for complex scenes based on monocular vision, comprising:

[0007] Obtaining video streams captured by multiple cameras and pose data detected by a six-axis sensor;

[0008] Image sampling is performed on the video stream to obtain a sampled image;

[0009] The sampled image is extracted and filtered for matching feature points with the aid of markers, and the depth of the feature points is calculated based on camera parameters to determine the three-dimensional position and form a point cloud map around the tower crane, wherein the point cloud map around the tower crane includes point cloud maps from multiple camera perspectives;

[0010] The point cloud maps around the tower crane are merged by three-dimensional affine transformation to obtain a merging result;

[0011] fitting a plane to the merging result to obtain a fitting result;

[0012] processing the fitting result according to the types of large planes, small planes and discrete points to obtain a three-dimensional reconstruction model around the tower crane;

[0013] sending the three-dimensional reconstruction model around the tower crane to a terminal to set a tower crane cargo destination and a running path based on the three-dimensional reconstruction model around the tower crane by the terminal.

[0014] Further technical solutions are as follows: the image sampling from the video stream to obtain a sampling image comprises:

[0015] extracting image data from the video stream in a sampling manner, saving each image as an independent file, and attaching a collection time and angle information of a tower crane jib or a camera in space at a corresponding moment to each image to obtain a sampling image; wherein the collection time is calculated according to a video stream start time plus an actual sampling time, and the angle information of the tower crane jib or the camera in space at the corresponding moment is an angle position of the tower crane jib calculated synchronously by a six-axis sensor.

[0016] Further technical solutions are as follows: the feature point is extracted and screened by using a marker, and the depth of the feature point is calculated according to camera parameters to determine a three-dimensional position, to form a point cloud map around the tower crane, comprising:

[0017] setting a marker around the tower crane;

[0018] identifying images containing the marker in the sampling images, and determining adjacent images according to the identified images to obtain a plurality of images;

[0019] extracting features from the plurality of images to obtain feature points;

[0020] calculating a Hamming distance of the feature points corresponding to the plurality of images, and determining initial matching feature points according to the Hamming distance;

[0021] screening final matching feature point pairs from the initial matching feature points;

[0022] determining a rotation and translation matrix between cameras, and calculating actual positions of the feature points by using a camera model formula combined with position information of the final matching feature point pairs;

[0023] correcting the actual positions of the feature points by using actual coordinates of the marker to obtain feature point position information around the tower crane, and forming a point cloud map around the tower crane.

[0024] A further technical solution is that the Hamming distance of the feature points corresponding to the plurality of images is calculated, and initial matching feature points are determined according to the Hamming distance, including:

[0025] The Hamming distance of the feature points corresponding to the plurality of images is calculated, and feature points with the smallest Hamming distance and smaller than a set threshold are screened to obtain a matching point pair;

[0026] The difference vector of the pixel coordinates between the matching point pairs is calculated, and the RANSAC algorithm is applied to screen the matching pairs combined with the difference vector to obtain initial matching feature points.

[0027] A further technical solution is that the rotation and translation matrices between the cameras are determined, and the actual positions of the feature points are calculated using a camera model formula combined with the position information of the final matching feature point pairs, including:

[0028] According to the pose data and the known tower arm rotation angle, the rotation matrix and the translation matrix between the two camera views are determined;

[0029] The depth of the final matching feature point pairs and the position angle of the final matching feature point pairs in the actual space are calculated using a camera model formula combined with the epipolar geometry constraint relationship to obtain the actual positions of the feature points.

[0030] A further technical solution is that the point cloud map around the tower crane is merged through three-dimensional affine transformation to obtain a merging result, including:

[0031] An affine transformation matrix is calculated using the feature position difference of the marker in the views of different cameras;

[0032] The affine transformation matrix is applied to the feature points corresponding to the camera coordinate system close to the cab, and the feature points corresponding to the camera coordinate system are converted to another camera coordinate system to obtain a merging result.

[0033] A further technical solution is that the fitting result is processed according to the types of large planes, small planes, and discrete points to obtain a three-dimensional reconstruction model around the tower crane, including:

[0034] For the discrete points in the fitting result, a clustering algorithm is used to identify and remove noise points, and a minimum cuboid is used to represent the aggregated discrete points; for the large planes in the fitting result, image mapping technology is used to enhance the color information and density of the point cloud, and a pre-trained convolutional neural network model is used to optimize the three-dimensional structure and texture of the object; for the small planes in the fitting result, the clustering algorithm is applied again, and a minimum cuboid is used to represent the complex object composed of small planes to obtain a three-dimensional reconstruction model around the tower crane.

[0035] Further technical solutions are as follows: the discrete points in the fitting result are identified and removed by a clustering algorithm, and the aggregated discrete points are represented by a minimum cuboid, including:

[0036] The clustering algorithm is applied to the discrete points in the fitting result, the aggregated discrete points are regarded as a feature set of a complex object, a minimum cuboid capable of completely containing the feature set of the complex object is used to represent the irregular object, and the discrete points existing alone are removed.

[0037] Further technical solutions are as follows: the large plane in the fitting result is enhanced by image mapping technology to enhance the color information and density of the point cloud, and a pre-trained convolutional neural network model is used to optimize the three-dimensional structure and texture of the object, including:

[0038] Color information is interpolated into the point cloud based on the corresponding relationship by using image mapping technology;

[0039] A pre-trained convolutional neural network model is used to identify and fill the three-dimensional structure and texture of the object in a specific scene.

[0040] The application also provides a monocular vision-based three-dimensional modeling system for a complex scene, including:

[0041] An acquisition unit is configured to acquire video streams captured by multiple cameras and pose data detected by a six-axis sensor;

[0042] An image sampling unit is configured to sample images from the video streams to obtain sampled images;

[0043] A point cloud map generation unit is configured to extract and filter matched feature points from the sampled images with the aid of markers, and calculate the depth of the feature points based on camera parameters to determine three-dimensional positions and form a point cloud map around a tower crane, wherein the point cloud map around the tower crane includes point cloud maps from multiple camera perspectives;

[0044] A merging unit is configured to merge the point cloud maps around the tower crane by three-dimensional affine transformation to obtain a merging result;

[0045] A fitting unit is configured to perform plane fitting on the merging result to obtain a fitting result;

[0046] A processing unit is configured to process the fitting result according to the types of large planes, small planes, and discrete points to obtain a three-dimensional reconstruction model around the tower crane;

[0047] A sending unit is configured to send the three-dimensional reconstruction model around the tower crane to a terminal to set a tower crane cargo terminal point and a running path based on the three-dimensional reconstruction model around the tower crane.

[0048] Compared with the prior art, the present application has the beneficial effects that: the present application integrates the video stream shot by multiple cameras and the pose data obtained by a six-axis sensor, uses a marker to assist in extracting matching feature points and calculating their three-dimensional positions to form a point cloud diagram, and then combines the multi-view point cloud diagrams through three-dimensional affine transformation, processes the fitting plane, and classifies them into large planes, small planes and discrete points to construct a three-dimensional reconstruction model around the tower crane, and finally sends the model to a terminal to set the cargo terminal and the running path; this method reduces the dependence on expensive 3D scanning equipment, simplifies the data acquisition and processing process, is easy to deploy and implement, and improves the safety and accuracy of tower crane operation, achieving double improvement in cost effectiveness and operation efficiency.

[0049] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0051] Figure 1 The flowchart of the complex scene three-dimensional modeling method based on monocular vision provided by the embodiment of the present application is shown in the figure.

[0052] Figure 2 The installation diagram of the camera and the six-axis sensor provided by the embodiment of the present application is shown in the figure.

[0053] Figure 3 The schematic diagram of the sampling image provided by the embodiment of the present application is shown in the figure.

[0054] Figure 4 The schematic diagram of the matching point provided by the embodiment of the present application is shown in the figure.

[0055] Figure 5 The schematic diagram of the actual position of the feature point provided by the embodiment of the present application is shown in the figure.

[0056] Figure 6 The schematic diagram of the camera view provided by the embodiment of the present application is shown in the figure.

[0057] Figure 7 The schematic diagram of the shape in the point cloud diagram and the shape in the image provided by the embodiment of the present application is shown in the figure.

[0058] Figure 8 The schematic diagram of the complex object processing provided by the embodiment of the present application is shown in the figure.

[0059] Figure 9A schematic block diagram of a computer device according to an embodiment of the present application is shown in FIG. 1.

[0060] Figure 10 A schematic block diagram of a computer device according to an embodiment of the present application is shown in FIG. 1. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of the present application.

[0062] It should be understood that the terms "comprise" and "include" as used in the specification and the appended claims indicate the presence of the described features, integers, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0063] It should also be understood that the terms used in the present application specification are only for the purpose of describing particular embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0064] It should be further understood that the term "and / or" as used in the present application specification and the appended claims means one or more of the associated listed items as well as all possible combinations of the items and includes the combinations.

[0065] Please refer to Figure 1 , Figure 1A schematic flowchart of the monocular vision-based complex scene three-dimensional modeling method provided by the embodiment of the present application. The monocular vision-based complex scene three-dimensional modeling method is applied to a server which interacts with a camera, a tower crane controller, a six-axis sensor, etc. to integrate multi-camera video streams and six-axis sensor data, uses a marker to assist feature point extraction and screening, and calculates feature point depth in combination with camera parameters to form a point cloud map around the tower crane. Three-dimensional affine transformation is used to merge point cloud maps under different perspectives, and the merging result is subjected to plane fitting processing to finally generate a three-dimensional reconstruction model around the tower crane including large planes, small planes, and discrete points. This process not only reduces the high-cost hardware dependence required by traditional three-dimensional reconstruction, but also simplifies the deployment process. Through image mapping technology and convolutional neural network optimization of the color information density of the large plane and the three-dimensional structure texture of the object, the model accuracy and visual effect are improved, thereby effectively improving the safety of tower crane operation and the accuracy of path planning. In addition, by clustering and removing noise points and using the method of using the smallest cuboid to represent the object feature set, the accuracy and practicality of the three-dimensional reconstruction model are further enhanced.

[0066] Specifically, the method of the embodiment is based on the SfM (Structure from Motion) theory for three-dimensional modeling. By placing the cameras in an optimized position, it is ensured that a wide area near and far can be covered. Rotating the tower crane's boom for one revolution allows the two cameras to capture omnidirectional visual images around the tower crane. Subsequently, image processing algorithms are applied to analyze the collected images and calculate the depth information of the feature points in the images, thereby constructing a detailed three-dimensional view. This method takes advantage of dynamically obtaining multi-view images and effectively reconstructs the three-dimensional model of the tower crane operating environment.

[0067] Figure 1 is a flowchart of the monocular vision-based complex scene three-dimensional modeling method provided by the embodiment of the present application. As shown in Figure 1 , the method includes the following steps S110 to S170.

[0068] S110, acquire video streams captured by multiple cameras and pose data detected by a six-axis sensor.

[0069] In this embodiment, the video stream refers to a sequence of images containing timestamp and spatial angle information continuously captured by two monocular cameras during the rotation of the tower crane boom, which is used to provide comprehensive visual data of the environment around the tower crane.

[0070] The pose data refers to the position and attitude (including pitch angle, yaw angle, and roll angle) of the tower crane boom / camera in space detected and recorded by the six-axis sensor in real time, which provides key information for accurately calculating the three-dimensional coordinates of the feature points in the images.

[0071] Referring to Figure 2 The method of the present embodiment consists of two monocular cameras, a six-axis sensor, and a high-performance computer. The two monocular cameras are installed at the end of the tower crane jib and near the cab to ensure that a wide range of visual images can be captured; the six-axis sensor is installed on the jib to provide spatial position and attitude information of the camera; the high-performance computer is responsible for processing the collected data and reconstructing accurate three-dimensional point cloud images.

[0072] Continuous image sequences captured by two monocular cameras installed at the end of the tower crane jib and near the cab. These two cameras are set to ensure that they can cover a sufficient distance range and provide comprehensive visual information of the environment around the tower crane. Specifically, the camera at the end of the jib is mainly responsible for capturing distant scenes, while the camera near the cab focuses on the near area. These cameras dynamically capture images around them as the tower crane jib rotates, forming a continuous video stream. In order to improve the accuracy and completeness of three-dimensional reconstruction, the system adjusts the sampling time interval according to the angle of rotation of the tower crane jib, ensuring that there are enough and evenly distributed images at different angles for subsequent processing.

[0073] Information about camera position and attitude provided by the six-axis sensor installed on the tower crane jib. The six-axis sensor (usually including an accelerometer and a gyroscope) can detect and record the angular changes of the tower crane jib / camera in space in real time, including key parameters such as pitch angle, yaw angle, and roll angle. This set of data is crucial for determining the position of the camera relative to the world coordinate system, as it allows the algorithm to accurately calculate the three-dimensional coordinates of each feature point in the actual space. In addition, each picture collected is attached with its corresponding collection time and the spatial angle information of the tower crane jib or camera at that moment, so as to facilitate subsequent accurate time synchronization and spatial positioning processing of the video stream. This method of combining video stream and pose data not only improves the accuracy of three-dimensional reconstruction, but also makes the entire system easier to deploy and operate.

[0074] S120, image sampling is performed on the video stream to obtain sampled images.

[0075] In the present embodiment, a sampled image refers to image data extracted from a video stream in a sampling manner, with each image saved as an independent file and attached with collection time and angle information of the tower crane jib or camera in space.

[0076] In short, a sampled image is a single-frame image file extracted from a video stream with time and spatial angle information annotations.

[0077] Specifically, image data is extracted from the video stream in a sampling manner, and each image is saved as an independent file, and the acquisition time and the angle information of the tower crane jib or camera in space at the corresponding moment are attached to each image to obtain a sampling image; wherein the acquisition time is calculated according to the video stream start time plus the actual sampling time, and the angle information of the tower crane jib or camera in space at the corresponding moment is the angle position of the tower crane jib calculated synchronously by the six-axis sensor.

[0078] Please refer to Figure 3 , the data acquisition is divided into two parts: camera data acquisition and six-axis sensor data acquisition. Camera data acquisition acquires images by sampling from the video stream, and records the acquisition time and the angle of the tower crane jib of each image; the six-axis sensor provides the angle information of the tower crane jib in space in real time.

[0079] The sampling time interval is determined by the angle of the tower crane jib, that is, as the rotation angle of the jib changes, the sampling time interval will also be adjusted accordingly. Figure 7 Five sampling moments (1s, 2s, 4s, 6s, 7s) and the corresponding tower crane jib angles (0.5, 1.0, 1.5, 2.0, 2.5) are shown. This shows that as the rotation angle of the jib increases, the sampling time interval is not uniformly distributed, but is adjusted according to actual needs.

[0080] Sampling time = video stream start time + sampling time interval. For example, if the video stream starts at 0s, the sampling time of the first picture is 0s + 1s = 1s, the second picture is 0s + 2s = 2s, and so on.

[0081] Each picture is attached with the angle of the tower crane jib at the time of acquisition, which is provided by the synchronously calculated six-axis sensor. For example, the first picture collected at 1s corresponds to an angle of 0.5, the second picture collected at 2s corresponds to an angle of 1.0, and so on.

[0082] The six-axis sensor is installed on the tower crane jib and can detect and record the spatial position and attitude of the jib (including pitch angle, yaw angle and roll angle) in real time. These angle information is used to determine the specific position of the tower crane jib at the time of each image acquisition, thereby assisting three-dimensional modeling.

[0083] Images are extracted from the video stream at uneven time intervals, and the acquisition time and corresponding jib angle of each image are recorded. The spatial angle of the tower crane jib is monitored and recorded in real time, providing accurate angle information for each image. Each image is associated with its acquisition time and jib angle to form a complete data set for subsequent three-dimensional reconstruction and analysis.

[0084] In this way, the accuracy and integrity of the data can be ensured, and a reliable basis can be provided for subsequent three-dimensional modeling.

[0085] S130, marker-assisted extraction is performed on the sampled images, and matched feature points are screened and the depth of the feature points is calculated according to the camera parameters to determine the three-dimensional position, thereby forming a point cloud map around the tower crane, wherein the point cloud map around the tower crane includes point cloud maps under multiple camera perspectives.

[0086] In this embodiment, the point cloud map around the tower crane refers to a data set representing the three-dimensional structure of the tower crane and its surrounding environment, which is generated by processing multi-perspective images and combining marker correction.

[0087] In an embodiment, the step S130 described above can include steps S131-S137.

[0088] S131, a marker is arranged around the tower crane.

[0089] In this embodiment, in order to improve the accuracy of reconstruction, one or more specific markers are placed around the tower crane as reference points to facilitate subsequent image processing and feature matching.

[0090] S132, images containing markers in the sampled images are identified, and adjacent images are determined according to the identified images to obtain a plurality of images.

[0091] In this embodiment, the plurality of images refers to images containing markers and their adjacent images, which are determined and selected by identifying markers in the sampled images, thereby forming an image set for subsequent processing.

[0092] The images containing markers are identified from the sampled images using a convolutional neural network, and adjacent images are determined based on these images to obtain a plurality of images for further analysis.

[0093] S133, feature extraction is performed on the plurality of images to obtain feature points.

[0094] In this embodiment, the feature points refer to key points or feature vectors that can uniquely identify or describe the content of the image, which are extracted by analyzing and processing the plurality of images containing markers and their adjacent images.

[0095] ORB (Oriented FAST and Rotated BRIEF) feature extraction is performed on the selected plurality of images to identify key feature points in each image. These feature points will be used for matching and three-dimensional reconstruction.

[0096] Specifically, in order to improve the accuracy of the three-dimensional reconstruction of the tower crane surrounding environment, the scheme proposes to set a marker in the tower crane working area. The image segment containing the marker is identified from the image collected by the camera through the convolutional neural network. Two images containing the marker and taken at adjacent time points are selected for ORB feature extraction. The ORB algorithm can efficiently detect and describe the key points and their corresponding descriptors in the image.

[0097] In step S134, the Hamming distances of the feature points corresponding to the multiple images are calculated, and initial matching feature points are determined according to the Hamming distances.

[0098] In this embodiment, the initial matching feature points refer to the feature point pairs with the smallest Hamming distances, which are selected by comparing the Hamming distances of the feature points between the multiple images.

[0099] In an embodiment, the step S134 described above can include steps S1341-S1342.

[0100] In step S1341, the Hamming distances of the feature points corresponding to the multiple images are calculated, and the feature points with the smallest Hamming distances and smaller than a set threshold are screened to obtain matching point pairs.

[0101] In this embodiment, the matching point pairs refer to the combinations of feature points with the smallest Hamming distances and smaller than a set threshold, which are screened by calculating the Hamming distances of the feature points between the multiple images, and are determined as the matching point pairs.

[0102] In step S1342, the difference vectors of the pixel coordinates between the matching point pairs are calculated, and the RANSAC algorithm is applied to screen the matching pairs combined with the difference vectors to obtain the initial matching feature points.

[0103] In this embodiment, for the ORB feature points in the two images, the Hamming distances between them are calculated one by one, and the point pairs with the smallest Hamming distances and smaller than a preset threshold are identified as the preliminary matching points. However, this matching method based on Hamming distance may introduce false matching points. Therefore, further processing of these matching points is needed to improve the matching accuracy.

[0104] For the preliminary matching point set obtained above, the pixel coordinate difference of each pair of matching points in their respective images is calculated to form a so-called difference vector. For example, as shown in Figure 4 if the coordinates of a pair of matching points in the first image are and the coordinates in the second image are then the difference vector of this pair of matching points is Since the correct matching points are in the majority, it means that most of the difference vectors should have similar direction and size. With the RANSAC (Random Sample Consensus) algorithm, those vectors representing the correct matching relationship can be filtered out from the numerous difference vectors, thus eliminating the false matches.

[0105] S135, screening the final matching feature point pairs from the initial matching feature points.

[0106] In the embodiment, the final matching feature point pairs refer to the most accurate and reliable feature point correspondence relationship determined from the initial matching feature points through further screening conditions or algorithm optimization.

[0107] S136, determining the rotation and translation matrix between the cameras, and calculating the actual position of the feature points using the camera model formula combined with the position information of the final matching feature point pairs;

[0108] In an embodiment, the above step S136 can include steps S1361-S1362.

[0109] S1361, determining the rotation matrix and translation matrix between the two camera perspectives according to the pose data and the known tower arm rotation angle;

[0110] S1362, calculating the depth of the final matching feature point pairs and the position angle of the final matching feature point pairs in the actual space using the camera model formula combined with the epipolar geometry constraint relationship.

[0111] As shown in Figure 5 , O1 and O2 are camera positions, P is the actual point position, and are the positions of point P under two perspective images.

[0112] As shown in Figure 6 , considering the actual operation scene, the two images used for matching are perspective images captured by the same camera at different angles, and the rotation angle θ between the two perspectives is known. According to the camera imaging model, the following equations can be established:

[0113] For the first perspective: ;

[0114] For the second perspective: ;

[0115] Where K represents the camera internal parameter matrix, R is the rotation matrix between the two perspectives, t is the translation vector, and and represent the distance of the object to the two camera positions, respectively.

[0116] Given the rotation angle θ, the corresponding rotation matrix and translation matrix can be constructed. Based on the epipolar geometry constraint, combined with the normalized coordinates of the feature points in the two images and , we have . By solving this system of equations, we can obtain the depth information of the feature points and their actual position angles relative to the camera, and further determine the polar coordinate representation of the feature points in three-dimensional space.

[0117] Specifically, because the camera view angle changes with the rotation of the large arm, according to the rotation angle , we can obtain the rotation matrix R= ; and the translation matrix t= .

[0118] Let the normalized coordinates of p1 and p2 be x1 and x2, then according to the epipolar geometry constraint: ;

[0119] According to the above relationship, the depth of the feature point can be obtained. According to the camera coordinates and the coordinates of the feature points in the image, the angle of the feature point in the actual position can be obtained. At this time, the polar coordinates of the feature point in the actual position have been obtained.

[0120] S137, the actual coordinates of the feature points are corrected using the actual coordinates of the markers to obtain the position information of the feature points around the tower crane and form a point cloud map around the tower crane.

[0121] Since the real-world coordinates of the markers used are known, any possible systematic errors can be evaluated and corrected by comparing the calculated results with the actual coordinates. This process helps to improve the accuracy of the entire reconstruction system.

[0122] First, a marker with a clear geometric shape and size is selected and placed in a fixed and known position within the tower crane work area. This marker should have easily identifiable features to facilitate accurate detection by the convolutional neural network from the images.

[0123] The work area containing the marker is photographed multiple times by the camera installed on the large arm of the tower crane. These images will be used for feature extraction, matching, and subsequent three-dimensional reconstruction. Based on the method described earlier, the position of the feature points corresponding to the marker in the actual space can be calculated under each view.

[0124] For each reconstructed feature point (i.e., the mapping of the marker under different views), we can compare its calculated spatial coordinates with the actual known coordinates of the marker. This comparison not only includes the difference in position, but also may involve the analysis of other parameters such as angle, distance, etc.

[0125] By comparing the deviations between the calculated results and the actual coordinates, the system error of the entire reconstruction system can be quantitatively evaluated. This step is crucial because it reveals potential problems or limitations in the algorithm, such as inaccurate camera parameter estimation, feature matching errors, etc.

[0126] According to the evaluated system error, appropriate measures are taken for correction. For example:

[0127] If it is found that the reconstruction results generally deviate from the actual positions, it may be due to inaccurate measurement of camera internal parameters (such as focal length, principal point position). At this time, re-calibration of the camera can be performed to improve it.

[0128] If a specific type of mismatch frequently occurs, consider optimizing the feature extraction and matching algorithm, or adjusting the relevant threshold to reduce such errors.

[0129] Combine multi-source data (such as GPS, IMU sensor information) to further improve positioning accuracy.

[0130] After completing the above corrections, the known coordinate markers are used again for testing to verify the improvement effect. This process is a cyclic iterative process until the reconstruction results meet the predetermined accuracy requirements.

[0131] In this embodiment, the pose data and tower arm rotation angle are combined to determine the rotation and translation relationship between the two camera views, providing a basis for subsequent calculations; using the camera model formula and polar geometric constraint relationship, the actual spatial position and depth of the final matched feature point pair are calculated. Then, the actual coordinates of the markers are used to correct the calculation results, improve the reconstruction accuracy, and finally form a point cloud map of the tower crane surroundings.

[0132] Through this series of steps, the three-dimensional structure of the tower crane surroundings can be effectively reconstructed, providing safety guidance for operators and avoiding potential collision risks. This method combines the advantages of visual information, sensor data, and mathematical models, improving the accuracy and reliability of the reconstruction process. In addition, the formed point cloud map provides a clear operating boundary for the tower crane, helping to improve the safety and efficiency of the operation.

[0133] S140, merging the point cloud map around the tower crane through a three-dimensional affine transformation to obtain a merging result.

[0134] In this embodiment, the merging result refers to aligning and integrating the point cloud data obtained by different cameras through the application of a three-dimensional affine transformation matrix, thereby generating a unified three-dimensional point cloud map that accurately reflects the overall structure of the tower crane and its surroundings.

[0135] In the three-dimensional reconstruction of the tower crane environment, a binocular or multi-view camera system can obtain more comprehensive spatial information. However, due to the physical position difference and the angle difference between the two cameras, directly merging the two sets of point cloud data may cause problems of mismatching or poor overlap.

[0136] In an embodiment, the step S140 described above can include steps S141-S142.

[0137] S141, using the feature position difference of the marker in the different camera angles to calculate the affine transformation matrix.

[0138] Feature point calculation is performed on all sampled images to obtain the feature point coordinates of the tower crane environment, i.e., to generate a point cloud map of the region. In short, the positions identified by these feature points define the safety boundary that should be avoided during tower crane operation. When we merge the point cloud data captured by two different angle cameras, we can construct a three-dimensional point cloud map of the entire tower crane scene.

[0139] However, due to the angle difference and the physical position difference between the two cameras, the point cloud obtained by direct merging may have the problem of incomplete overlap. There are two specific reasons for this: on the one hand, the reconstruction feature point profile is deformed due to the angle difference between the two cameras; on the other hand, there is a slight translation error due to the difference in physical position between the two cameras. In order to solve these problems, three-dimensional affine transformation technology can be used to solve them at one time.

[0140] In this embodiment, the actual coordinates of the known markers are used as a reference to compare the corresponding feature point positions under each camera angle. According to these differences, a system of equations can be established to solve the affine transformation parameters. The affine transformation matrix usually includes rotation and translation components, which are used to adjust the spatial relationship between different angles.

[0141] The matrix form is as follows: where xyz are rotation and scaling coefficients, and abc are translation coefficients. These parameters need to be determined according to the actual data. The method of determination is based on the pre-determined markers. Because the shape and position of the markers are known, and the markers are easily identifiable objects, the coefficients of the transformation matrix are determined by comparing the features and actual shape and position of the markers in the two three-dimensional angles.

[0142] S142, applying the affine transformation matrix to the feature points corresponding to the camera coordinate system near the cab to convert the feature points corresponding to the camera coordinate system to another camera coordinate system to obtain the merging result.

[0143] In this embodiment, once the precise affine transformation matrix is obtained, it can be applied to all feature point coordinates from one of the cameras. This step involves performing a matrix multiplication operation for each feature point, thereby converting it from the original camera coordinate system to the target camera coordinate system.

[0144] After completing the coordinate conversion, the two sets of point cloud data can now be merged into a unified set of point clouds. This merging process not only includes geometric alignment, but also may involve the integration of color or other attribute information to generate the final three-dimensional point cloud model.

[0145] Finally, check if the merged point cloud meets the expected effect, i.e. all feature points accurately reflect the true structure of the tower crane and its surrounding environment. If any deviation or misplacement is found, it may be necessary to return to adjust the affine transformation parameters until a satisfactory merging result is obtained.

[0146] Through the above steps, the effective merging of multi-view point cloud data is achieved, improving the safety and efficiency of tower crane operations, and also providing more accurate basic data support for subsequent operations.

[0147] S150, fitting a plane to the merging result to obtain a fitting result.

[0148] In this embodiment, the fitting result refers to the related parameters of the best fitting plane calculated by the least squares method.

[0149] The principle of least squares method in statistics is used to find a best fitting plane, so that the sum of squared distances (error) from all points in the point cloud to the plane is minimized.

[0150] S160, processing the fitting result according to the types of large planes, small planes and discrete points to obtain a three-dimensional reconstruction model around the tower crane.

[0151] In this embodiment, the three-dimensional reconstruction model around the tower crane refers to processing different types of point cloud data (including large planes, small planes and discrete points) in the fitting result through specific algorithms and technical means, thereby generating an accurate, detailed and visually realistic three-dimensional environment model. This model not only reflects the basic structure and positional relationship of objects around the tower crane, but also provides rich detailed information such as color, texture, etc., which helps to achieve more accurate obstacle avoidance and path planning

[0152] Specifically, for the discrete points in the fitting result, noise points are identified and removed by a clustering algorithm, and the aggregated discrete points are represented by a minimum cuboid; for the large planes in the fitting result, the color information and density of the point cloud are enhanced by image mapping technology, and the three-dimensional structure and texture of the object are optimized using a pre-trained convolutional neural network model; for the small planes in the fitting result, the clustering algorithm is applied again, and a minimum cuboid is used to represent the complex object composed of small planes, to obtain the three-dimensional reconstruction model around the tower crane.

[0153] Among them, for the discrete points in the fitting result, the clustering algorithm is applied to regard the aggregated discrete points as a feature set of a complex object, and a minimum cuboid capable of completely containing the feature set of the complex object is used to represent the irregular object, and the discrete points existing alone are removed.

[0154] The large plane in the fitting result is enhanced by image mapping technology to enhance the color information and density of the point cloud, and the three-dimensional structure and texture of the object are optimized using a pre-trained convolutional neural network model, comprising:

[0155] Using image mapping technology, the color information is interpolated to the point cloud based on the correspondence;

[0156] Using a pre-trained convolutional neural network model to identify and fill the three-dimensional structure and texture of the object in a specific scene.

[0157] In this embodiment, when modeling a complex scene, the fitting process usually presents three different situations: the first is large and obvious objects such as buildings and cars, which form large planes after fitting; the second is small objects such as sundries, which produce small planes after fitting; the third is discrete points that cannot form planes, which may appear on small irregular objects or due to image noise. For these three situations, the processing strategy is as follows:

[0158] First, the discrete points are processed. By analyzing the spatial coordinates of these points, a clustering algorithm is used to regard the aggregated discrete points as a feature point set of a complex object, and a minimum cuboid capable of completely containing these points is used to represent this irregular complex object. The discrete points existing alone are regarded as image noise and are removed.

[0159] At this time, as shown in Figure 7 and Figure 8 , there are two types of planes in the entire point cloud: planes formed by large regular objects and planes formed by small irregular objects. Next, specific operations are performed on these two types of planes respectively.

[0160] The large plane needs to perform the following two steps:

[0161] In view of the fact that the point cloud obtained by a monocular camera has a significantly lower density than a laser radar, especially in terms of the amount of information (texture) on large planes. In order to approach a real three-dimensional reconstruction map, it is necessary to map the point cloud map with the actual captured image. Based on the correspondence between the feature points in the point cloud and the image feature points, the color information in the image is mapped into the point cloud using an interpolation method, thereby improving the density of the point cloud and further improving the visual effect of the reconstruction.

[0162] In order to make the reconstruction more realistic, a recognition algorithm based on a convolutional neural network can be used to optimize the three-dimensional reconstruction. Considering that the types of objects in the tower crane scene are relatively limited, common object types (such as buildings, engineering vehicles, etc.) can be pre-added to the recognition algorithm library, and at the same time, the three-dimensional structure and texture information corresponding to these objects are added. On the basis that the two-dimensional image and the three-dimensional image have been mapped with each other, the recognition algorithm is applied to the two-dimensional image. If it is recognized as belonging to the object in the library, the corresponding three-dimensional structure and texture are used to fill the corresponding part in the three-dimensional reconstruction.

[0163] For the segmented small planes, because they represent a variety of combined components of complex objects, and the most critical thing in modeling around the tower crane is to obtain obstacle avoidance information of the tower crane. Therefore, the cuboid obtained by fitting is used as the contour boundary of the complex object. Similarly, according to the coordinates of the small planes, a clustering algorithm is used again, and small planes close to each other are regarded as part of a complex object. Then, a minimum cuboid that can completely contain these small planes is used to replace it, thereby realizing the three-dimensional reconstruction map of irregular complex objects.

[0164] In summary, this processing method ensures that key information is extracted from three-dimensional point cloud data of a complex scene, generating an environment model that is not only accurate but also has high visual quality, improving the practicality of the model and providing important safety support for tower crane operation.

[0165] Through the fine processing of different types of point cloud data in the fitting result, the final three-dimensional reconstruction model around the tower crane not only accurately depicts the position and shape of large regular objects, but also effectively expresses the existence of small complex objects and their approximate outline, while improving the visual reality and practicality of the entire scene. This process is crucial for improving the safety and efficiency of tower crane operation.

[0166] S170, sending the three-dimensional reconstruction model around the tower crane to a terminal to set a tower crane cargo destination and a running path based on the three-dimensional reconstruction model around the tower crane.

[0167] In this embodiment, after constructing the complete 3D reconstruction model around the tower crane, the next step is to further optimize and utilize this model to improve operational efficiency and safety. Specifically, based on the distance between each complex sub-model, sub-models that are close to each other or connected are merged into a larger model. This process is mainly to identify a single group of objects, thereby simplifying the model structure and facilitating subsequent processing. If these combined objects are not in the pre-defined 3D graphics library, in order to facilitate observation and differentiation, these newly combined models can be given unique colors. This approach not only helps improve visual recognition, but also helps operators quickly locate and understand the positional relationship of different objects.

[0168] After completing the above steps, the 3D reconstruction model around the tower crane is constructed. This accurate and intuitive model provides a powerful tool for operators to plan and execute tower crane operation tasks. The following are specific application scenarios of this model:

[0169] End point coordinate selection: Operators can directly select the destination of the goods to be transported by the tower crane on the 3D model and obtain the precise coordinates of that point. This coordinate information is then transmitted to the intelligent control system of the tower crane as the basis for automatic route planning.

[0170] Automatic path planning: After receiving the end point coordinates, the intelligent control terminal of the tower crane automatically calculates an optimal running path based on the environmental information provided by the 3D reconstruction model. This path takes into account the positions of all obstacles and the physical limitations of the tower crane itself, ensuring safety and efficiency of operation.

[0171] Manual path drawing: In addition to automatic path planning, operators can also manually draw specific running paths that the tower crane needs to follow on the 3D model. This method is particularly suitable for situations that require additional flexibility or have special requirements for specific operations. The drawn path is also sent to the intelligent control terminal of the tower crane for implementation.

[0172] Through this process, the operation of the tower crane becomes more intelligent and efficient, while greatly improving the safety of the operation. Operators can more accurately understand the surrounding environment and make more informed decisions, while the tower crane can safely and efficiently complete various transportation tasks based on detailed environmental models and specific task requirements. This combination of automation and human intervention provides a new perspective and technical support for modern tower crane operation.

[0173] The above-mentioned monocular vision-based three-dimensional modeling method for complex scenes integrates video streams captured by multiple cameras and pose data obtained by a six-axis sensor, uses a marker to assist in extracting matching feature points and calculating their three-dimensional positions to form a point cloud map, and then merges multi-view point cloud maps through three-dimensional affine transformation, fits planes, and classifies them into large planes, small planes, and discrete points to construct a three-dimensional reconstruction model around the tower crane. Finally, the model is sent to the terminal to set the cargo destination and operation path. This method reduces the dependence on expensive 3D scanning equipment, simplifies the data collection and processing process, is easy to deploy and implement, and improves the safety and accuracy of tower crane operation, achieving cost-effectiveness and operational efficiency.

[0174] Figure 9 is a schematic block diagram of a monocular vision-based three-dimensional modeling system 300 for complex scenes provided by an embodiment of the present application. As shown in Figure 9 corresponding to the above-mentioned monocular vision-based three-dimensional modeling method for complex scenes, the present application also provides a monocular vision-based three-dimensional modeling system 300 for complex scenes. The monocular vision-based three-dimensional modeling system 300 for complex scenes includes units for performing the above-mentioned monocular vision-based three-dimensional modeling method for complex scenes, and the system can be configured in a server. Specifically, please refer to Figure 9 , the monocular vision-based three-dimensional modeling system 300 for complex scenes includes an acquisition unit 301, an image sampling unit 302, a point cloud map generation unit 303, a merging unit 304, a fitting unit 305, a processing unit 306, and a sending unit 307.

[0175] The acquisition unit 301 is configured to acquire video streams captured by multiple cameras and pose data detected by a six-axis sensor. The image sampling unit 302 is configured to perform image sampling on the video streams to obtain sampled images. The point cloud map generation unit 303 is configured to extract and filter matching feature points from the sampled images with the assistance of a marker, and calculate the depth of the feature points based on camera parameters to determine their three-dimensional positions, thereby forming a point cloud map around the tower crane. The point cloud map around the tower crane includes point cloud maps from multiple camera perspectives. The merging unit 304 is configured to merge the point cloud map around the tower crane through three-dimensional affine transformation to obtain a merging result. The fitting unit 305 is configured to fit planes to the merging result to obtain a fitting result. The processing unit 306 is configured to process the fitting result according to the types of large planes, small planes, and discrete points to obtain a three-dimensional reconstruction model around the tower crane. The sending unit 307 is configured to send the three-dimensional reconstruction model around the tower crane to a terminal to set the cargo destination and operation path of the tower crane based on the three-dimensional reconstruction model around the tower crane.

[0176] In an embodiment, the image sampling unit 302 is configured to extract image data from the video stream in a sampling manner, save each image as an independent file, and attach the collection time and the angle information of the tower crane jib or the camera in the space at the corresponding moment to each image to obtain a sampling image; wherein the collection time is calculated according to the video stream start time plus the actual sampling time, and the angle information of the tower crane jib or the camera in the space at the corresponding moment is the angle position of the tower crane jib calculated by the six-axis sensor synchronously.

[0177] In an embodiment, the point cloud map generation unit 303 comprises:

[0178] The setting subunit is configured to set a marker around the tower crane; the identification subunit is configured to identify images containing the marker in the sampling images, and determine adjacent images according to the identified images to obtain a plurality of images; the extraction subunit is configured to perform feature extraction on the plurality of images to obtain feature points; the calculation subunit is configured to calculate the Hamming distance of the feature points corresponding to the plurality of images, and determine initial matching feature points according to the Hamming distance; the screening subunit is configured to screen final matching feature point pairs from the initial matching feature points; the position determination subunit is configured to determine the rotation and translation matrixes between the cameras, and calculate the actual positions of the feature points using a camera model formula combined with the position information of the final matching feature point pairs; and the correction subunit is configured to correct the actual positions of the feature points using the actual coordinates of the marker to obtain the feature point position information around the tower crane, and form a point cloud map around the tower crane.

[0179] In an embodiment, the calculation subunit comprises:

[0180] The first calculation module is configured to calculate the Hamming distance of the feature points corresponding to the plurality of images, screen feature points with the smallest Hamming distance and smaller than a set threshold to obtain matching point pairs; and the second calculation module is configured to calculate the difference vector of the pixel coordinates between the matching point pairs, and screen the matching pairs using a RANSAC algorithm combined with the difference vector to obtain initial matching feature points.

[0181] In an embodiment, the position determination subunit comprises:

[0182] The matrix determination module is configured to determine the rotation matrix and the translation matrix between the visual angles of the two cameras according to the pose data and the known tower arm rotation angle; and the angle calculation module is configured to calculate the depth of the final matching feature point pairs and the position angle of the final matching feature point pairs in the actual space using a camera model formula combined with the epipolar geometry constraint relationship to obtain the actual positions of the feature points.

[0183] In an embodiment, the merging unit 304 comprises:

[0184] The transformation matrix calculation subunit is configured to calculate an affine transformation matrix using the feature position difference of the marker in different camera perspectives; and the conversion subunit is configured to apply the affine transformation matrix to the feature points corresponding to the camera coordinate system close to the cab, and convert the feature points corresponding to the camera coordinate system to another camera coordinate system to obtain a merging result.

[0185] In an embodiment, the processing unit 306 is configured to identify and remove noise points from the discrete points in the fitting result by using a clustering algorithm, and represent the aggregated discrete points by using a minimum cuboid; enhance the color information and density of the point cloud by using an image mapping technique, and optimize the three-dimensional structure and texture of the object by using a pre-trained convolutional neural network model for the large plane in the fitting result; and apply the clustering algorithm again, and represent the complex object composed of the small plane by using a minimum cuboid to obtain a three-dimensional reconstruction model around the tower crane.

[0186] In an embodiment, the processing unit 306 is configured to regard the aggregated discrete points as a feature set of a complex object by using a clustering algorithm, and represent the irregular object by using a minimum cuboid capable of completely containing the feature set of the complex object, and remove the discrete points existing alone for the discrete points in the fitting result.

[0187] In an embodiment, the processing unit 306 is configured to interpolate the color information to the point cloud based on the corresponding relationship by using an image mapping technique, and identify and fill the three-dimensional structure and texture of the object in a specific scene by using a pre-trained convolutional neural network model.

[0188] It should be noted that the specific implementation process of the above-mentioned monocular vision-based complex scene three-dimensional modeling system 300 and each unit can be clearly understood by those skilled in the art, and can refer to the corresponding description in the foregoing method embodiments. For the convenience and brevity of description, it will not be repeated here.

[0189] The monocular vision-based complex scene three-dimensional modeling system 300 can be implemented in the form of a computer program, which can run on a computer device as shown in the computer device. Figure 10

[0190] Please refer to Figure 10 , Figure 10 is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device 500 can be a server, wherein the server can be a stand-alone server or a server cluster composed of multiple servers.

[0191] Please refer to Figure 10 ​The computer device 500 comprises a processor 502, a memory and a network interface 505 connected through a system bus 501, wherein the memory can comprise a non-volatile storage medium 503 and an internal memory 504.

[0192] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 comprises program instructions which, when executed, can cause the processor 502 to perform a monocular vision-based three-dimensional modeling method for a complex scene.

[0193] The processor 502 is configured to provide computing and control capabilities to support the operation of the entire computer device 500.

[0194] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503, which, when executed by the processor 502, can cause the processor 502 to perform a monocular vision-based three-dimensional modeling method for a complex scene.

[0195] The network interface 505 is configured to perform network communication with other devices. Those skilled in the art can understand that the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device 500 to which the scheme of the present application is applied. The specific computer device 500 can comprise more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. Figure 10

[0196] The processor 502 is configured to run the computer program 5032 stored in the memory to implement the following steps:

[0197] Obtain video streams captured by a plurality of cameras and pose data detected by a six-axis sensor; perform image sampling on the video streams to obtain sampled images; extract and filter matched feature points using a marker for the sampled images, and calculate the depth of the feature points according to camera parameters to determine three-dimensional positions and form a point cloud map around a tower crane, wherein the point cloud map around the tower crane comprises point cloud maps from multiple camera perspectives; merge the point cloud maps around the tower crane through three-dimensional affine transformation to obtain a merging result; perform plane fitting on the merging result to obtain a fitting result; process the fitting result according to the types of large planes, small planes and discrete points to obtain a three-dimensional reconstruction model around the tower crane; and send the three-dimensional reconstruction model around the tower crane to a terminal to set a tower crane cargo terminal point and a running path based on the three-dimensional reconstruction model around the tower crane.

[0198] ​In an embodiment, the processor 502, in implementing the step of image sampling the video stream to obtain sampled images, implements the following steps:

[0199] extracting image data from the video stream in a sampling manner, and saving each image as an independent file, and attaching the collection time and the angle information of the tower crane jib or the camera in the space at the corresponding moment to each image to obtain a sampled image; wherein the collection time is calculated according to the video stream start time plus the actual sampling time, and the angle information of the tower crane jib or the camera in the space at the corresponding moment is the angle position of the tower crane jib calculated synchronously by the six-axis sensor.

[0200] In an embodiment, the processor 502, in implementing the step of extracting and screening matched feature points with the aid of markers for the sampled images, and calculating the depth of the feature points according to the camera parameters to determine the three-dimensional position, and forming a point cloud map around the tower crane, implements the following steps:

[0201] setting markers around the tower crane; identifying images containing markers in the sampled images, and determining adjacent images according to the identified images to obtain a plurality of images; extracting features from the plurality of images to obtain feature points; calculating the Hamming distance of the feature points corresponding to the plurality of images, and determining initial matching feature points according to the Hamming distance; screening final matching feature point pairs from the initial matching feature points; determining the rotation and translation matrix between cameras, and calculating the actual position of the feature points using the camera model formula combined with the position information of the final matching feature point pairs; correcting the actual position of the feature points using the actual coordinates of the markers to obtain the position information of the feature points around the tower crane, and forming a point cloud map around the tower crane.

[0202] In an embodiment, the processor 502, in implementing the step of calculating the Hamming distance of the feature points corresponding to the plurality of images, and determining initial matching feature points according to the Hamming distance, implements the following steps:

[0203] calculating the Hamming distance of the feature points corresponding to the plurality of images, and screening feature points with the smallest Hamming distance and smaller than a set threshold to obtain matching point pairs; calculating the difference vector of the pixel coordinates between the matching point pairs, and applying the RANSAC algorithm combined with the difference vector to screen the matching pairs to obtain initial matching feature points.

[0204] In an embodiment, the processor 502, in implementing the step of determining the rotation and translation matrix between cameras, and calculating the actual position of the feature points using the camera model formula combined with the position information of the final matching feature point pairs, implements the following steps:

[0205] According to the pose data and a known tower arm rotation angle, a rotation matrix and a translation matrix between two camera perspectives are determined; a depth of the final matched feature point pair and a position angle of the final matched feature point pair in an actual space are calculated by using a camera model formula combined with an epipolar geometry constraint relationship, so as to obtain an actual position of the feature point.

[0206] In an embodiment, when implementing the step of merging the point cloud around the tower crane by the three-dimensional affine transformation to obtain a merging result, the processor 502 specifically implements the following steps:

[0207] An affine transformation matrix is calculated using a feature position difference of the marker in different camera perspectives; the affine transformation matrix is applied to feature points corresponding to a camera coordinate system close to the cab, so as to convert the feature points corresponding to the camera coordinate system to another camera coordinate system to obtain the merging result.

[0208] In an embodiment, when implementing the step of processing the fitting result according to types of large planes, small planes and discrete points to obtain a three-dimensional reconstruction model around the tower crane, the processor 502 specifically implements the following steps:

[0209] For the discrete points in the fitting result, a clustering algorithm is used to identify and remove noise points, and a minimum cuboid is used to represent the aggregated discrete points; for the large planes in the fitting result, an image mapping technique is used to enhance color information and density of the point cloud, and a pre-trained convolutional neural network model is used to optimize three-dimensional structure and texture of the object; for the small planes in the fitting result, the clustering algorithm is applied again, and a minimum cuboid is used to represent a complex object composed of the small planes, so as to obtain the three-dimensional reconstruction model around the tower crane.

[0210] In an embodiment, when implementing the step of processing the discrete points in the fitting result by a clustering algorithm to identify and remove noise points, and using a minimum cuboid to represent the aggregated discrete points, the processor 502 specifically implements the following steps:

[0211] For the discrete points in the fitting result, a clustering algorithm is used to treat the aggregated discrete points as a feature set of a complex object, and a minimum cuboid capable of completely containing the feature set of the complex object is used to represent the irregular object, and the discrete points existing alone are removed.

[0212] In an embodiment, when implementing the step of processing the large planes in the fitting result by an image mapping technique to enhance color information and density of the point cloud, and using a pre-trained convolutional neural network model to optimize three-dimensional structure and texture of the object, the processor 502 specifically implements the following steps:

[0213] Color information is interpolated to the point cloud based on the correspondence relation by using image mapping technology; and a pre-trained convolutional neural network model is used to identify and fill the three-dimensional structure and texture of objects in a specific scene.

[0214] It should be understood that, in the embodiments of the present application, the processor 502 can be a central processing unit 306 (CPU), and the processor 502 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0215] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the above-mentioned embodiments.

[0216] Therefore, the present application also provides a storage medium. The storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program is executed by a processor to make the processor perform the following steps:

[0217] Obtain video streams captured by multiple cameras and pose data detected by a six-axis sensor; perform image sampling on the video streams to obtain sample images; extract and screen matched feature points by using a marker for assistance on the sample images, and calculate depths of the feature points according to camera parameters to determine three-dimensional positions and form a point cloud map around a tower crane, wherein the point cloud map around the tower crane includes point cloud maps under multiple camera perspectives; merge the point cloud maps around the tower crane by three-dimensional affine transformation to obtain a merging result; perform plane fitting on the merging result to obtain a fitting result; process the fitting result according to types of large planes, small planes and discrete points to obtain a three-dimensional reconstruction model around the tower crane; and send the three-dimensional reconstruction model around the tower crane to a terminal to set a tower crane cargo terminal point and a running path based on the three-dimensional reconstruction model around the tower crane.

[0218] In an embodiment, the processor, when executing the computer program to implement the image sampling of the video stream to obtain the sampled images, implements the following steps:

[0219] extracting image data from the video stream in a sampling manner, and saving each image as an independent file, and attaching the collection time and the angle information of the tower crane jib or the camera in the space at the corresponding moment to each image to obtain the sampled images; wherein the collection time is calculated according to the video stream start time plus the actual sampling time, and the angle information of the tower crane jib or the camera in the space at the corresponding moment is the angle position of the tower crane jib calculated synchronously by the six-axis sensor.

[0220] In an embodiment, the processor, when executing the computer program to implement the marker-assisted extraction of the sampled images and the screening of the matched feature points, and the calculation of the depth of the feature points according to the camera parameters to determine the three-dimensional positions and form the point cloud map around the tower crane, implements the following steps:

[0221] setting a marker around the tower crane; identifying the images containing the marker in the sampled images, and determining the adjacent images according to the identified images to obtain a plurality of images; performing feature extraction on the plurality of images to obtain feature points; calculating the Hamming distances of the feature points corresponding to the plurality of images, and determining the initial matched feature points according to the Hamming distances; screening the final matched feature point pairs from the initial matched feature points; determining the rotation and translation matrices between the cameras, and calculating the actual positions of the feature points using the camera model formula combined with the position information of the final matched feature point pairs; correcting the actual positions of the feature points using the actual coordinates of the marker to obtain the position information of the feature points around the tower crane, and forming the point cloud map around the tower crane.

[0222] In an embodiment, the processor, when executing the computer program to implement the calculation of the Hamming distances of the feature points corresponding to the plurality of images, and the determination of the initial matched feature points according to the Hamming distances, implements the following steps:

[0223] calculating the Hamming distances of the feature points corresponding to the plurality of images, and screening the feature points with the smallest Hamming distances and smaller than a set threshold to obtain matched point pairs; calculating the difference vectors of the pixel coordinates between the matched point pairs, and applying the RANSAC algorithm combined with the difference vectors to screen the matched pairs to obtain the initial matched feature points.

[0224] In an embodiment, the processor, when executing the computer program to implement the determination of the rotation and translation matrices between the cameras, and the calculation of the actual positions of the feature points using the camera model formula combined with the position information of the final matched feature point pairs, implements the following steps:

[0225] According to the pose data and the known tower arm rotation angle, a rotation matrix and a translation matrix between two camera perspectives are determined; and by using a camera model formula combined with an epipolar geometry constraint relationship, depth of the final matched feature point pair and a position angle of the final matched feature point pair in an actual space are calculated to obtain an actual position of the feature point.

[0226] In an embodiment, when the processor implements the step of merging the point cloud around the tower crane by three-dimensional affine transformation to obtain a merging result by executing the computer program, the following steps are implemented:

[0227] An affine transformation matrix is calculated using a feature position difference of the marker in different camera perspectives; the affine transformation matrix is applied to feature points corresponding to a camera coordinate system close to the driver's cabin to convert the feature points corresponding to the camera coordinate system to another camera coordinate system to obtain the merging result.

[0228] In an embodiment, when the processor implements the step of processing the fitting result according to types of large planes, small planes and discrete points to obtain a three-dimensional reconstruction model around the tower crane by executing the computer program, the following steps are implemented:

[0229] For the discrete points in the fitting result, a clustering algorithm is used to identify and remove noise points, and a minimum cuboid is used to represent the aggregated discrete points; for the large planes in the fitting result, an image mapping technique is used to enhance color information and density of the point cloud, and a pre-trained convolutional neural network model is used to optimize three-dimensional structure and texture of the object; for the small planes in the fitting result, the clustering algorithm is applied again, and a minimum cuboid is used to represent a complex object composed of the small planes to obtain the three-dimensional reconstruction model around the tower crane.

[0230] In an embodiment, when the processor implements the step of processing the discrete points in the fitting result by using a clustering algorithm to identify and remove noise points, and using a minimum cuboid to represent the aggregated discrete points by executing the computer program, the following steps are implemented:

[0231] For the discrete points in the fitting result, a clustering algorithm is used to consider the aggregated discrete points as a feature set of a complex object, and a minimum cuboid capable of completely containing the feature set of the complex object is used to represent the irregular object, and the discrete points existing alone are removed.

[0232] In an embodiment, when the processor implements the step of processing the large planes in the fitting result by using an image mapping technique to enhance color information and density of the point cloud, and using a pre-trained convolutional neural network model to optimize three-dimensional structure and texture of the object by executing the computer program, the following steps are implemented:

[0233] Color information is interpolated to the point cloud based on the correspondence using image mapping techniques; a pre-trained convolutional neural network model is used to identify and fill in the three-dimensional structure and texture of objects within a particular scene.

[0234] The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, and the like, which are various computer-readable storage media that can store program codes.

[0235] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software, or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0236] In several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and actual implementation can have another division manner. For example, multiple units or components can be combined or integrated into another system, or some features can be omitted or not executed.

[0237] The steps in the method embodiments of the present application can be adjusted, combined, and reduced in sequence according to actual needs. The units in the system embodiments of the present application can be combined, divided, and reduced according to actual needs. In addition, each functional unit in each embodiment of the present application can be integrated in a processing unit 306, or each unit can exist physically, or two or more units can be integrated in one unit.

[0238] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a storage medium. Based on such understanding, the technical solutions of the present application essentially or say the parts that make contributions to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0239] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A monocular vision-based three-dimensional modeling method for complex scenes, characterized in that, The method comprises: acquiring video streams captured by multiple cameras and pose data detected by a six-axis sensor; image sampling on the video streams to obtain sampled images; marker-assisted extraction and screening of matched feature points on the sampled images, and calculation of depths of the feature points according to camera parameters to determine three-dimensional positions and form a point cloud map around a tower crane, wherein the point cloud map around the tower crane comprises point cloud maps under multiple camera perspectives; merging the point cloud maps around the tower crane through three-dimensional affine transformation to obtain a merging result; plane fitting on the merging result to obtain a fitting result; processing of the fitting result according to types of large planes, small planes and discrete points to obtain a three-dimensional reconstruction model around the tower crane; sending the three-dimensional reconstruction model around the tower crane to a terminal to set a tower crane cargo terminal point and a running path based on the three-dimensional reconstruction model around the tower crane.

2. The monocular vision based 3D modeling method of claim 1, wherein, The image sampling on the video streams to obtain the sampled images comprises: extracting image data from the video streams in a sampling manner, saving each image as an independent file, and attaching acquisition time and angle information of a tower crane jib or a camera in space at a corresponding moment to each image to obtain the sampled images; wherein the acquisition time is calculated according to a video stream start time plus an actual sampling time, and the angle information of the tower crane jib or the camera in space at the corresponding moment is an angle position of the tower crane jib calculated synchronously through the six-axis sensor. 3.The monocular vision based 3D modeling method of complex scene according to claim 1, wherein, The marker-assisted extraction and screening of matched feature points on the sampled images, and the calculation of depths of the feature points according to camera parameters to determine three-dimensional positions and form a point cloud map around a tower crane, comprises: setting markers around the tower crane; identifying images containing the markers in the sampled images, and determining adjacent images according to the identified images to obtain multiple images; feature extraction on the multiple images to obtain feature points; calculation of Hamming distances of the feature points corresponding to the multiple images, and determination of initial matching feature points according to the Hamming distances; screening of final matching feature point pairs from the initial matching feature points; determination of rotation and translation matrices between cameras, and calculation of actual positions of the feature points using a camera model formula combined with position information of the final matching feature point pairs; correction of the actual positions of the feature points using actual coordinates of the markers to obtain feature point position information around the tower crane and form a point cloud map around the tower crane.

4. The monocular vision based 3D modeling method of claim 3, wherein, The calculation of Hamming distances of the feature points corresponding to the multiple images, and the determination of initial matching feature points according to the Hamming distances, comprises: calculation of Hamming distances of the feature points corresponding to the multiple images, screening of feature points with minimum Hamming distances and less than a set threshold to obtain matching point pairs; calculation of difference vectors of pixel coordinates between the matching point pairs, and screening of the matching pairs using a RANSAC algorithm combined with the difference vectors to obtain initial matching feature points.

5. The monocular vision based 3D modeling method of claim 3, wherein, The determination of rotation and translation matrices between cameras, and the calculation of actual positions of the feature points using a camera model formula combined with position information of the final matching feature point pairs, comprises: According to the pose data and a known tower arm rotation angle, a rotation matrix and a translation matrix between two camera perspectives are determined; By using a camera model formula combined with an epipolar geometry constraint relationship, depth of the final matching feature point pair and a position angle of the final matching feature point pair in an actual space are calculated to obtain an actual position of the feature point. 6.The monocular vision based 3D modeling method of complex scene according to claim 1, wherein, The merging the point cloud maps around the tower crane by the three-dimensional affine transformation to obtain a merging result, comprising: An affine transformation matrix is calculated using a feature position difference of the marker in different camera perspectives; The affine transformation matrix is applied to feature points corresponding to a camera coordinate system close to the cab to convert the feature points corresponding to the camera coordinate system to another camera coordinate system to obtain the merging result. 7.The monocular vision based 3D modeling method of complex scene according to claim 1, wherein, The fitting result is processed according to types of large planes, small planes and discrete points to obtain a three-dimensional reconstruction model around the tower crane, comprising: For the discrete points in the fitting result, noise points are identified and removed by a clustering algorithm, and the aggregated discrete points are represented by a minimum cuboid; for the large planes in the fitting result, color information and density of the point cloud are enhanced by an image mapping technology, and a three-dimensional structure and texture of an object are optimized by using a pre-trained convolutional neural network model; for the small planes in the fitting result, a clustering algorithm is applied again, and a complex object composed of the small planes is represented by a minimum cuboid to obtain the three-dimensional reconstruction model around the tower crane.

8. The monocular vision-based three-dimensional modeling method of claim 7, wherein, For the discrete points in the fitting result, noise points are identified and removed by a clustering algorithm, and the aggregated discrete points are represented by a minimum cuboid, comprising: For the discrete points in the fitting result, the aggregated discrete points are regarded as a feature set of a complex object by applying a clustering algorithm, and an irregular object is represented by a minimum cuboid capable of completely containing the feature set of the complex object, and the discrete points existing alone are removed. 9.The monocular vision based 3D modeling method of complex scene according to claim 7, wherein, For the large planes in the fitting result, color information and density of the point cloud are enhanced by an image mapping technology, and a three-dimensional structure and texture of an object are optimized by using a pre-trained convolutional neural network model, comprising: Color information is interpolated to the point cloud based on a corresponding relationship by using an image mapping technology; A three-dimensional structure and texture of an object in a specific scene are identified and filled by using a pre-trained convolutional neural network model.

10. A monocular vision based 3D modeling system for complex scenes, characterized in that, Comprising: An acquisition unit is configured to acquire video streams captured by multiple cameras and pose data detected by a six-axis sensor; An image sampling unit is configured to perform image sampling on the video streams to obtain sampling images; A point cloud map generation unit is configured to extract and filter matching feature points by using a marker, and calculate depths of the feature points according to camera parameters to determine three-dimensional positions, thereby forming point cloud maps around a tower crane, wherein the point cloud maps around the tower crane include point cloud maps in multiple camera perspectives; A merging unit is configured to merge the point cloud maps around the tower crane by three-dimensional affine transformation to obtain a merging result; A fitting unit is configured to perform plane fitting on the merging result to obtain a fitting result; A processing unit is configured to process the fitting result according to types of large planes, small planes and discrete points to obtain a three-dimensional reconstruction model around the tower crane. The sending unit is configured to send the three-dimensional reconstruction model of the surroundings of the tower crane to a terminal, so that the terminal sets a cargo destination and a running path of the tower crane based on the three-dimensional reconstruction model of the surroundings of the tower crane.

Images