Multi-mode-based tower crane construction process object pose detection method and device

Abstract

The invention relates to a multi-modal-based tower crane construction process object pose detection method and device, and belongs to the technical field of multi-modal data fusion, and the method comprises the steps: obtaining the spatial position information of goods hoisted by a tower crane in real time through an RTK positioning system; based on a label detection function of a monocular camera, obtaining space attitude and position information of the hoisted goods in a camera coordinate system; unifying the geographic coordinate system information and the camera coordinate system information; the space position information of the cargo hoisted by the tower crane and the position information in the camera coordinate system are fused to generate object space position information of the hoisted cargo; and according to the spatial position information of the object, a 3D animation model of the hoisted goods and 6D position and posture information under a preset coordinate system are generated, and the moving track and posture change of the goods in the tower crane construction process are displayed in real time. According to the invention, high-precision and stable acquisition of the position information and the posture of the cargo is realized, and the hoisting process of the cargo is dynamically displayed in a visual mode of 3D animation.

Description

Technical Field

[0001] The present invention relates to a method and a device for detecting the position and posture of an object in a tower crane construction process based on multimodality, and specifically to a method and a device for detecting the position and posture of an object in a tower crane construction process based on RTK positioning and monocular camera posture detection technology, and belongs to the technical field of multimodal data fusion. Background Art

[0002] In large-scale construction projects, infrastructure construction and other fields, tower cranes, as an important construction equipment, are widely used for the lifting and transportation of heavy materials. In the process of tower crane lifting goods, the spatial position and posture information of the goods are important parameters for monitoring and operation of the construction site, which are directly related to the safety, accuracy and efficiency of the construction. However, the construction site environment is complex, and it is extremely difficult for operators to accurately grasp the spatial position and posture information of the hoisted goods in real time when the tower crane is operating. There are many interference factors on site, and conventional detection methods are difficult to provide comprehensive and accurate data. This makes the operator lack of effective information support when lifting goods, and it is difficult to accurately control the operation of the tower crane, which is easy to cause safety accidents such as collisions and falling goods, posing serious safety hazards to the construction.

[0003] Due to the wide range of operations and limited field of view, there are many technical challenges in accurately and in real time obtaining the spatial position and attitude information of the goods. At present, the commonly used positioning technologies include RTK (Real-Time Kinematic) technology and visual inspection technology. RTK technology is based on the high-precision global navigation satellite system (GNSS), which can provide spatial position information such as latitude, longitude and altitude. It has high positioning accuracy and is suitable for scenes where labels are not visible; but its disadvantage is that RTK positioning information lacks a description of the cargo posture, and the reception of satellite signals is limited by bad weather, resulting in inaccurate and fluctuating position information, which is difficult to meet the needs of all-round position and real-time accurate information during dynamic lifting. The visual inspection technology based on monocular cameras can provide the spatial position and attitude information (such as rotation angle) of the cargo by identifying the labels on the cargo; this method relies on the visibility of the labels and has high stability and accuracy in situations where the labels are visible, but when the cargo is blocked or exceeds the camera's field of view, the detection effect is greatly reduced, and continuous monitoring cannot be achieved.

[0004] With the rapid development of the construction industry, the scale of construction is expanding, and the construction requirements are increasing. High-rise and super-high-rise buildings are constantly emerging, and the construction process is becoming more and more complicated, which puts forward more urgent demands for the visualization and precise monitoring of the tower crane construction process. The traditional tower crane operation method can no longer meet the requirements of modern construction, and innovative technical solutions are urgently needed to solve these problems. Therefore, the present invention provides a method and device for detecting the position and posture of objects in the tower crane construction process based on multi-modality. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a method and device for detecting the position and posture of objects in the tower crane construction process based on multimodality, which can achieve high-precision and stable acquisition of cargo position information and posture, and dynamically display the cargo lifting process through the visualization of 3D animation.

[0006] The technical solution adopted by the present invention to solve the technical problem is: In a first aspect, an embodiment of the present invention provides a method for detecting an object posture during a tower crane construction process based on multimodality, comprising the following steps: Step S1, obtaining the spatial position information of the cargo hoisted by the tower crane in real time through the RTK positioning system, wherein the spatial position information includes the latitude, longitude and altitude of the cargo, and is used to determine the precise position of the cargo in the geographic coordinate system; Step S2, obtaining the spatial posture and position information of the hoisted cargo in the camera coordinate system based on the tag detection function of the monocular camera; Step S3, coordinate system 1 of the acquired geographic coordinate system information and camera coordinate system information; Step S4, in the unified coordinate system, the spatial position information of the cargo hoisted by the tower crane obtained in real time by the RTK positioning system and the position information of the cargo hoisted in the camera coordinate system are integrated to generate the object spatial position information of the cargo hoisted; Step S5, based on the object spatial position information of the hoisted cargo, generates a 3D animation model of the hoisted cargo and 6D position and posture information in a preset coordinate system, and displays the movement trajectory and posture changes of the cargo during the tower crane construction process in real time.

[0007] As a possible implementation of this embodiment, the real-time acquisition of spatial position information of cargo hoisted by a tower crane includes: When the cargo tag is beyond the camera's field of view, the RTK positioning system is used to obtain the spatial position information of the cargo hoisted by the tower crane; Output cargo location data in the form of latitude, longitude and altitude.

[0008] As a possible implementation of this embodiment, the tag detection function based on the monocular camera obtains the spatial posture and position information of the hoisted cargo in the camera coordinate system, including: When the cargo tag is within the camera's field of view, the cargo's spatial position and posture information is obtained through the monocular camera's tag detection function, and the cargo's position is represented by the spatial position and posture information of the camera coordinate system.

[0009] As a possible implementation of this embodiment, the step of performing coordinate system integration of the acquired geographic coordinate system information and the camera coordinate system information includes: Geographic coordinate system information conversion: The geographic coordinate system information of the goods hoisted by the tower crane is obtained in real time through the RTK positioning system, and the coordinates of the WGS84 geographic coordinate system are converted into the coordinates of the UTM projection or Gauss-Krüger projection to obtain the projection coordinates of the goods. The difference between the projection coordinates of the tower crane base and the obtained projection coordinates of the goods is used to convert the projection coordinates of the goods into coordinate points in the spatial coordinate system with the tower crane base as the origin, the vertical height as the Z axis, the Y axis facing the observer, and the right side of the Y axis as the X axis, where the altitude is directly converted to the height of the Z axis; Camera coordinate system information conversion: The angle between the monocular camera and the crane arm is used to convert the measured spatial position information of the cargo into spatial coordinates with the tower crane base as the origin, and the position information of the cargo under the camera is converted.

[0010] As a possible implementation of this embodiment, in the process of converting geographic coordinate system information, the process of converting the geographic coordinate system information of the goods from the coordinates of the WGS84 geographic coordinate system to the UTM projection coordinates is as follows: Determine the UTM projection zone based on longitude; Convert longitude and latitude to UTM coordinates: First convert latitude φ and longitude λ from degrees to radians; then use the semi-major axis defined by the geodetic datum (WGS84) a and flattening f Ellipsoid parameters, calculate the radius of curvature N ; Then use the Transverse Mercator projection formula to convert the longitude and latitude to X and Y in UTM coordinates.

[0011] As a possible implementation of this embodiment, the camera coordinate system information conversion includes: Define the camera coordinate system of the monocular camera: the line of sight of the lens is the negative direction of the Z axis, the direction directly below the lens is the positive direction of the Y axis, and the direction directly to the right of the lens is the positive direction of the X axis; Assume the coordinates in the camera coordinate system are (Xc, Yc), and the converted 3D interface coordinates are (X, Y): , , Among them, θ is the angle of the camera.

[0012] As a possible implementation of this embodiment, the spatial position information of the cargo hoisted by the tower crane is obtained in real time through the RTK positioning system in the unified coordinate system, and the position information of the cargo hoisted in the camera coordinate system is obtained and fused to generate the object spatial position information of the cargo hoisted, including: Based on the spatial position information of the cargo hoisted by the tower crane obtained in real time through the RTK positioning system and the past data of the position information of the hoisted cargo in the camera coordinate system, compare whether the position data has data fluctuations exceeding the threshold. If the threshold is exceeded, the other type of position information whose data fluctuations do not exceed the threshold is directly used as the position information of the cargo hoisted by the tower crane in space; if both types of position information do not exceed the threshold of data fluctuations, the two types of position information are summed and the average is taken as the final spatial position information of the object. If the data fluctuations of both types of position information exceed the threshold, the position information is ignored. If continuous data fluctuations occur, the sending device may prompt an error.

[0013] As a possible implementation of this embodiment, the 3D animation model of the hoisted cargo and the 6D position and posture information in a preset coordinate system are generated according to the object spatial position information of the hoisted cargo, and the movement trajectory and posture changes of the cargo during the tower crane construction process are displayed in real time, including: Utilize the collected cargo posture information and convert it into the coordinate system in the visualization scene; The open3D rendering technology is used to visualize the relative position information of the tower crane and the hoisted cargo, the posture information of the hoisted cargo, and the rotation information of the tower crane.

[0014] In a second aspect, an embodiment of the present invention provides a multi-modal tower crane construction process object posture detection device, comprising: A geographic coordinate acquisition module is used to obtain the spatial position information of the cargo hoisted by the tower crane in real time through the RTK positioning system. The spatial position information includes the latitude, longitude and altitude of the cargo, which is used to determine the precise position of the cargo in the geographic coordinate system; The camera coordinate acquisition module is used to obtain the spatial posture and position information of the hoisted cargo in the camera coordinate system based on the label detection function of the monocular camera; A coordinate system 1 module is used to coordinate the acquired geographic coordinate system information and the camera coordinate system information; The position information fusion module is used to fuse the spatial position information of the cargo hoisted by the tower crane obtained in real time through the RTK positioning system and the position information of the cargo hoisted in the camera coordinate system in a unified coordinate system to generate the object spatial position information of the cargo hoisted; The animation model generation module is used to generate a 3D animation model of the hoisted cargo and 6D position and posture information in a preset coordinate system according to the spatial position information of the hoisted cargo object, and to display the movement trajectory and posture changes of the cargo during the tower crane construction process in real time.

[0015] As a possible implementation of this embodiment, the coordinate system module includes: A geographic coordinate system information conversion module is used to obtain the geographic coordinate system information of the goods hoisted by the tower crane in real time through the RTK positioning system, convert the coordinates of the WGS84 geographic coordinate system into the coordinates of the UTM projection or the Gauss-Krüger projection, obtain the projection coordinates of the goods, and use the difference between the projection coordinates of the tower crane base and the obtained projection coordinates of the goods to convert the projection coordinates of the goods into coordinate points in the spatial coordinate system with the tower crane base as the origin and the vertical height as the Z axis, facing the observer as the Y axis, and the right side of the Y axis as the X axis, wherein the altitude is directly converted into the height of the Z axis; The camera coordinate system information conversion module is used to convert the measured spatial posture information of the cargo into spatial coordinates with the tower crane base as the origin by using the angle between the monocular camera and the crane arm, and to convert the position information of the cargo under the camera.

[0016] As a possible implementation of this embodiment, the coordinate system module may include: The projection coordinate acquisition module is used to convert the geographic coordinate system information of the goods hoisted by the tower crane obtained in real time through the RTK positioning system, and convert the longitude and latitude information into a plane coordinate system through the UTM projection coordinate system or the Gauss-Krüger projection coordinate system to obtain the projection coordinates of the goods; The projection coordinate conversion module is used to calculate the difference between the projection coordinates of the tower crane base and the projection coordinates of the cargo, and convert the projection coordinates of the cargo into a three-dimensional coordinate system with the tower crane base as the origin; The spatial posture conversion module is used to use the installation angle between the monocular camera and the tower crane to convert the spatial posture information of the cargo in the camera coordinate system into position data in a three-dimensional coordinate system with the tower crane base as the origin, so as to unify the camera coordinate system information with the geographic coordinate system information.

[0017] The beneficial effects of the technical solution of the embodiment of the present invention are as follows: A method for detecting the posture of an object in a tower crane construction process based on multimodality according to the technical solution of an embodiment of the present invention comprises the following steps: step S1, obtaining the spatial position information of the cargo hoisted by the tower crane in real time through an RTK positioning system, wherein the spatial position information includes the latitude, longitude and altitude of the cargo, and is used to determine the precise position of the cargo in the geographic coordinate system; step S2, obtaining the spatial posture and position information of the hoisted cargo in the camera coordinate system based on the tag detection function of the monocular camera; step S3, integrating the acquired geographic coordinate system information with the camera coordinate system information; step S4, integrating the spatial position information of the cargo hoisted by the tower crane obtained in real time through the RTK positioning system and the position information of the cargo hoisted in the camera coordinate system in a unified coordinate system to generate the object spatial position information of the cargo hoisted; step S5, generating a 3D animation model of the cargo hoisted and the 6D position and posture information in a preset coordinate system according to the object spatial position information of the cargo hoisted, and displaying the movement trajectory and posture changes of the cargo during the tower crane construction process in real time. The present invention utilizes the advantages of fusing multi-source data to achieve high-precision and stable acquisition of cargo position information and posture, and dynamically displays the cargo lifting process through 3D animation visualization, which has important practical value and engineering significance.

[0018] The present invention realizes high-precision visualization of the spatial position and posture of hoisted cargo by combining the RTK positioning system with a monocular camera. When the cargo exceeds the camera's field of view, it automatically switches to the RTK system data to ensure real-time and continuous display of position information. Through position information fusion, coordinate system unification and 3D rendering technology, construction personnel can obtain accurate cargo position and motion trajectory, which helps to improve the safety and operational efficiency of tower crane construction.

[0019] The present invention unifies the spatial position information represented in different systems into a visual coordinate system and performs visual display, which is used to display the spatial position and posture of the hoisted cargo in real time during the tower crane construction process, and realizes the precise positioning and 3D visual display of the hoisted cargo. Compared with the prior art, the present invention has the following characteristics: 1. Real-time visualization: By combining RTK positioning and a monocular camera, the present invention realizes the real-time visualization of the spatial position and posture information of the hoisted cargo during the construction process, which helps to improve the visualization effect of the construction and make the construction process more intuitive.

[0020] 2. Accurate spatial positioning: The RTK system of the present invention provides accurate geographic location data, while the monocular camera provides accurate and stable cargo spatial position and posture information. The combination of the two enables high-precision and stable acquisition of spatial position information. Even if the cargo is beyond the camera's field of view, RTK can still provide stable positioning support.

[0021] 3. Automatic switching of data sources: The present invention can ensure the continuity and accuracy of data and improve the stability and reliability of the construction process by automatically switching the data source when the goods enter or exceed the camera's field of view.

[0022] 4. Visual coordinate system conversion: Based on a specific coordinate conversion method, the present invention converts the cargo location information from different coordinate systems into a unified 3D coordinate system, adapts to a variety of complex construction scenarios, and provides a basis for multi-angle construction monitoring.

[0023] 5. Fusion of two types of position information: The spatial position information of the crane hoisted cargo obtained in real time through the RTK positioning system and the position information of the hoisted cargo in the camera coordinate system are integrated. The final position information has the characteristics of precise geographic location information measured by RTK, and also takes into account the continuous and stable characteristics of visual measurement, making the final object position information more accurate and stable.

[0024] 6. Improve construction safety and efficiency: The present invention uses visual display to help construction personnel better understand the real-time position and posture of the goods during the lifting process, thereby reducing operational difficulty, reducing potential risks, and improving construction efficiency and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of a method for detecting the position and posture of an object in a tower crane construction process based on multi-modality according to an exemplary embodiment; Figure 2 This is a display diagram of object posture detection in a tower crane construction process based on multi-modality according to an exemplary embodiment; Figure 3 It is a schematic structural diagram of a multi-modal object posture detection device in a tower crane construction process according to an exemplary embodiment. DETAILED DESCRIPTION

[0026] In order to more clearly illustrate the technical features of the solution of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0027] There is a potential solution in the prior art, which is to fuse multi-source sensor information. By combining the advantages of RTK technology and monocular camera detection technology, more accurate spatial position and posture information can be provided when the tag is visible, and RTK positioning information can be used to supplement the tag when it is not visible to ensure the continuity and accuracy of the data. In addition, in order to improve the intuitiveness and controllability of on-site operations, the three-dimensional animation visualization technology is used to dynamically display the cargo hoisting process, which can provide real-time visualization assistance for construction personnel and improve construction efficiency and safety. Therefore, the present invention proposes a tower crane cargo hoisting process visualization method based on RTK positioning and monocular camera posture detection technology, which integrates the advantages of multi-source data, realizes high-precision and stable acquisition of cargo position information and posture, and dynamically displays the cargo hoisting process through 3D animation visualization, which has important practical value and engineering significance.

[0028] like Figure 1 As shown, an embodiment of the present invention provides a method for detecting an object posture in a tower crane construction process based on multimodality, comprising the following steps: Step S1, obtaining the spatial position information of the cargo hoisted by the tower crane in real time through the RTK positioning system, wherein the spatial position information includes the latitude, longitude and altitude of the cargo, and is used to determine the precise position of the cargo in the geographic coordinate system; Step S2, obtaining the spatial posture and position information of the hoisted cargo in the camera coordinate system based on the tag detection function of the monocular camera; Step S3, coordinate system 1 of the acquired geographic coordinate system information and camera coordinate system information; Step S4, in the unified coordinate system, the spatial position information of the cargo hoisted by the tower crane obtained in real time by the RTK positioning system and the position information of the cargo hoisted in the camera coordinate system are integrated to generate the object spatial position information of the cargo hoisted; Step S5, based on the object spatial position information of the hoisted cargo, generates a 3D animation model of the hoisted cargo and 6D position and posture information in a preset coordinate system, and displays the movement trajectory and posture changes of the cargo during the tower crane construction process in real time.

[0029] As a possible implementation of this embodiment, the real-time acquisition of spatial position information of cargo hoisted by a tower crane includes: When the cargo tag is beyond the camera's field of view, the RTK positioning system is used to obtain the spatial position information of the cargo hoisted by the tower crane; Output cargo location data in the form of latitude, longitude and altitude.

[0030] As a possible implementation of this embodiment, the tag detection function based on the monocular camera obtains the spatial posture and position information of the hoisted cargo in the camera coordinate system, including: When the cargo tag is within the camera's field of view, the cargo's spatial position and posture information is obtained through the monocular camera's tag detection function, and the cargo's position is represented by the spatial position and posture information of the camera coordinate system.

[0031] As a possible implementation of this embodiment, the step of performing coordinate system integration of the acquired geographic coordinate system information and the camera coordinate system information includes: Geographic coordinate system information conversion: The geographic coordinate system information of the goods hoisted by the tower crane is obtained in real time through the RTK positioning system, and the coordinates of the WGS84 geographic coordinate system are converted into the coordinates of the UTM projection or Gauss-Krüger projection to obtain the projection coordinates of the goods. The difference between the projection coordinates of the tower crane base and the obtained projection coordinates of the goods is used to convert the projection coordinates of the goods into coordinate points in the spatial coordinate system with the tower crane base as the origin, the vertical height as the Z axis, the Y axis facing the observer, and the right side of the Y axis as the X axis, where the altitude is directly converted to the height of the Z axis; Camera coordinate system information conversion: The angle between the monocular camera and the crane arm is used to convert the measured spatial position information of the cargo into spatial coordinates with the tower crane base as the origin, and the position information of the cargo under the camera is converted.

[0032] As a possible implementation of this embodiment, in the process of converting geographic coordinate system information, the process of converting the geographic coordinate system information of the goods from the coordinates of the WGS84 geographic coordinate system to the UTM projection coordinates is as follows: Determine the UTM projection zone based on longitude; Convert longitude and latitude to UTM coordinates: First convert latitude φ and longitude λ from degrees to radians; then use the semi-major axis defined by the geodetic datum (WGS84) a and flattening f Ellipsoid parameters, calculate the radius of curvature N ; Then use the Transverse Mercator projection formula to convert the longitude and latitude to X and Y in UTM coordinates.

[0033] The process of converting latitude and longitude to UTM plane coordinates is as follows: 1) Convert latitude and longitude to radians: Convert latitude (φ) and longitude (λ) from degrees to radians: ; 2) Calculate auxiliary parameters: use the major semi-axis of the ellipsoid parameters defined by the geodetic datum (WGS84) a and flattening f Calculate the radius of curvature N : , Where e is the first eccentricity of the ellipsoid, which can be calculated by the following formula: ; 3) Calculate X and Y coordinates: Use the Transverse Mercator projection formula to convert the longitude and latitude to X (easting) and Y (northing) in UTM coordinates: , where k0 is the scale factor (usually 0.9996), M is the meridian arc length, and λ0 is the central longitude of the projection zone.

[0034] Then, the difference between the projection coordinates of the crane base and the projection coordinates of the cargo is used to convert the projection coordinates of the cargo into (X, Y) coordinates with the crane base as the origin and the altitude as the Z coordinate.

[0035] As a possible implementation of this embodiment, the camera coordinate system information conversion includes: Define the camera coordinate system of the monocular camera: the line of sight of the lens is the negative direction of the Z axis, the direction directly below the lens is the positive direction of the Y axis, and the direction directly to the right of the lens is the positive direction of the X axis; Assume the coordinates in the camera coordinate system are (Xc, Yc), and the converted 3D interface coordinates are (X, Y): , , Where θ is the angle of the camera.

[0036] As a possible implementation of this embodiment, the spatial position information of the cargo hoisted by the tower crane is obtained in real time through the RTK positioning system in the unified coordinate system, and the position information of the cargo hoisted in the camera coordinate system is obtained and fused to generate the object spatial position information of the cargo hoisted, including: Based on the spatial position information of the cargo hoisted by the tower crane obtained in real time through the RTK positioning system and the past data of the position information of the hoisted cargo in the camera coordinate system, compare whether the position data has data fluctuations exceeding the threshold. If the threshold is exceeded, the other type of position information whose data fluctuations do not exceed the threshold is directly used as the position information of the cargo hoisted by the tower crane in space; if both types of position information do not exceed the threshold of data fluctuations, the two types of position information are summed and the average is taken as the final spatial position information of the object. If the data fluctuations of both types of position information exceed the threshold, the position information is ignored. If continuous data fluctuations occur, the sending device may prompt an error.

[0037] As a possible implementation of this embodiment, the 3D animation model of the hoisted cargo and the 6D position and posture information in a preset coordinate system are generated according to the object spatial position information of the hoisted cargo, and the movement trajectory and posture changes of the cargo during the tower crane construction process are displayed in real time, including: Utilize the collected cargo posture information and convert it into the coordinate system in the visualization scene; The relative position information of the tower crane and the hoisted cargo, the posture information of the hoisted cargo, and the rotation information of the tower crane are visualized using open3D rendering technology. Figure 2 As shown, Figure 2 In the figure, blue is the Z axis, red is the Y axis, and green is the X axis.

[0038] The visualization process of the tower crane construction process of the present invention is as follows: 1) When lifting goods on the ground, you need to install the RTK module and label plate at the corresponding position where the goods need to be lifted.

[0039] 2) After the ground workers install the RTK and label plate, the entire tower crane cargo visualization system can be started. When the system is started, it will automatically render the tower crane model in the canvas according to the height of the tower crane, and then the visualization system will start the cargo position information measurement module in the background (the measurement module will automatically convert the absolute position information measured by the RTK module and the position information in the camera coordinate system measured by the vision module into visualization coordinate information with the tower crane as the coordinate system when receiving the information data).

[0040] 3) After the measurement module is started, the measurement module will first detect whether the RTK measurement module and the visual measurement module have detected the corresponding location information. When neither of them detects the information, an error message will be fed back to the system operator. When only the RTK information is detected and no label information is detected, it means that the cargo does not appear in the camera's visual image. At this time, the visualization system only visualizes the real-time location information of the cargo based on the RTK information. When RTK and label information are detected, the system will integrate the RTK information and label information to comprehensively display the real-time location information of the hoisted cargo. When the system only receives label information, it means that the RTK measurement module cannot operate normally, and the error information of the RTK measurement module will be fed back to the visualization system.

[0041] 4) When the measurement modules are operating normally, the visualization system will render the location of the goods on the visualization screen in real time according to the real-time location of the goods. At the same time, when the posture information is received, the posture information of the goods will be rendered. When the posture information of the goods is not received, only the location information of the goods will be rendered.

[0042] like Figure 3 As shown, an embodiment of the present invention provides a multi-modal tower crane construction process object posture detection device, comprising: A geographic coordinate acquisition module is used to obtain the spatial position information of the cargo hoisted by the tower crane in real time through the RTK positioning system. The spatial position information includes the latitude, longitude and altitude of the cargo, which is used to determine the precise position of the cargo in the geographic coordinate system; The camera coordinate acquisition module is used to obtain the spatial posture and position information of the hoisted cargo in the camera coordinate system based on the label detection function of the monocular camera; A coordinate system 1 module is used to coordinate the acquired geographic coordinate system information and the camera coordinate system information; The position information fusion module is used to fuse the spatial position information of the cargo hoisted by the tower crane obtained in real time through the RTK positioning system and the position information of the cargo hoisted in the camera coordinate system in a unified coordinate system to generate the object spatial position information of the cargo hoisted; The animation model generation module is used to generate a 3D animation model of the hoisted cargo and 6D position and posture information in a preset coordinate system according to the spatial position information of the hoisted cargo object, and to display the movement trajectory and posture changes of the cargo during the tower crane construction process in real time.

[0043] As a possible implementation of this embodiment, the coordinate system module includes: A geographic coordinate system information conversion module is used to obtain the geographic coordinate system information of the goods hoisted by the tower crane in real time through the RTK positioning system, convert the coordinates of the WGS84 geographic coordinate system into the coordinates of the UTM projection or the Gauss-Krüger projection, obtain the projection coordinates of the goods, and use the difference between the projection coordinates of the tower crane base and the obtained projection coordinates of the goods to convert the projection coordinates of the goods into coordinate points in the spatial coordinate system with the tower crane base as the origin and the vertical height as the Z axis, facing the observer as the Y axis, and the right side of the Y axis as the X axis, wherein the altitude is directly converted into the height of the Z axis; The camera coordinate system information conversion module is used to convert the measured spatial posture information of the cargo into spatial coordinates with the tower crane base as the origin by using the angle between the monocular camera and the crane arm, and to convert the position information of the cargo under the camera.

[0044] When the coordinate system module 1 includes a geographic coordinate system information conversion module and a camera coordinate system information conversion module, the object posture detection process of the tower crane construction process based on multi-modality is as follows: 1. Obtain the spatial location information of the hoisted cargo: Use the RTK positioning system to obtain the latitude, longitude and altitude information of the cargo to provide the geographic location information of the cargo; 2. Obtain the posture and position information of the goods: Through the label detection function of the monocular camera, the spatial position and posture information of the goods in the camera coordinate system is obtained, indicating the precise position of the goods relative to the camera; 3. Coordinate system unification: The position information obtained by the RTK system and the monocular camera is converted and unified to ensure that the data of both can be expressed in the same 3D space coordinate system, and the tower crane base is used as the origin for positioning; 4. Automatically switch the location information source: When the goods enter or exceed the camera's field of view, the system will automatically switch the data source according to demand to ensure continuous and accurate cargo location information; 5. Fusion of two types of position information: When two types of position information are obtained at the same time, the position information after the two coordinate systems are transformed is fused to finally obtain the position information of an object in space. The fusion of the two types of position information makes the final position information have the characteristics of accurate geographic location information measured by RTK, and also takes into account the characteristics of continuous stability of visual measurement, making the final object position information more accurate and stable; 5.3D animation generation and display: Based on the unified cargo space location information, 3D animation is generated using Open3D rendering technology to provide real-time visualization of the movement trajectory of the hoisted cargo in space.

[0045] As a possible implementation of this embodiment, the coordinate system module may include: The projection coordinate acquisition module is used to convert the geographic coordinate system information of the goods hoisted by the tower crane obtained in real time through the RTK positioning system, and convert the longitude and latitude information into a plane coordinate system through the UTM projection coordinate system or the Gauss-Krüger projection coordinate system to obtain the projection coordinates of the goods; The projection coordinate conversion module is used to calculate the difference between the projection coordinates of the tower crane base and the projection coordinates of the cargo, and convert the projection coordinates of the cargo into a three-dimensional coordinate system with the tower crane base as the origin; The spatial posture conversion module is used to use the installation angle between the monocular camera and the tower crane to convert the spatial posture information of the cargo in the camera coordinate system into position data in a three-dimensional coordinate system with the tower crane base as the origin, so as to unify the camera coordinate system information with the geographic coordinate system information.

[0046] When the coordinate system module includes a projection coordinate acquisition module, a projection coordinate conversion module and a space posture conversion module, the object posture detection process of the tower crane construction process based on multi-modality is as follows: 1. Obtain the spatial location information of the goods: Use the RTK (real-time kinematic positioning) system to obtain the geographical location information of the hoisted cargo, and the data format is longitude, latitude and altitude; The RTK positioning system provides the location information of the cargo when it is beyond the field of view of the monocular camera, thus maintaining continuity.

[0047] 2. Obtain the spatial posture and position information of the goods: When the cargo tag is within the camera's field of view, the cargo's spatial posture (rotation angle) and position are acquired through the tag detection function of the monocular camera; The monocular camera obtains position information in the camera coordinate system (i.e. the direction of the lens is the negative direction of the Z axis, the bottom is the positive direction of the Y axis, and the right is the positive direction of the X axis); the data in the camera coordinate system contains the precise spatial position and posture of the cargo relative to the camera.

[0048] 3. Unified coordinate system: Convert the geographic coordinates (latitude, longitude and altitude in WGS84 format) obtained by the RTK positioning system, and convert the longitude and latitude information into a plane coordinate system through the UTM or Gauss-Krüger projection coordinate system; By calculating the difference between the projection coordinates of the crane base and the projection coordinates of the cargo, the projection coordinates of the cargo are converted into a three-dimensional coordinate system with the crane base as the origin (the Z axis is the vertical height, the Y axis is the front of the crane, and the X axis is the right side of the crane); The coordinate information of the monocular camera is converted into position data in the tower crane base coordinate system by utilizing the installation angle between the camera and the tower crane, so as to achieve unification with the RTK positioning system data.

[0049] 4. Automatic switching of location information sources: When the cargo enters or leaves the camera's field of view, the system automatically switches the location information source. That is, when the cargo is within the camera's field of view, the camera data is used, and when it leaves the field of view, the data is switched to the RTK positioning system to maintain the continuity and stability of the location tracking. The switching mechanism uses sensors to detect the location of the goods in real time in order to determine the source of the data.

[0050] 5. Fusion of two types of location information: When two types of position information are obtained at the same time, the system will fuse the position information after the two coordinate systems are converted to finally obtain the position information of an object in space. The fusion of the two types of position information makes the final position information have the characteristics of accurate geographic location information measured by RTK, and also takes into account the continuous and stable characteristics of visual measurement, making the final object position information more accurate and stable.

[0051] 6. Generate 3D animation for visual display: Using Open3D rendering technology, the unified cargo location data is converted into a 3D animation model; The animated model displays the real-time movement trajectory and posture changes of the hoisted cargo in space, allowing operators to intuitively monitor the hoisting process and improve the safety and efficiency of construction.

[0052] The details of the projection coordinate transformation performed by the present invention (applicable to RTK data) are as follows: First, determine the UTM projection zone where the cargo is located and convert the longitude and latitude data into plane coordinates; Convert latitude and longitude to radians and calculate relevant intermediate variables using the ellipsoid parameters of the geodetic datum (WGS84); Use the Transverse Mercator projection formula to convert the longitude and latitude data into X and Y coordinates in the UTM coordinate system; Through the difference in projection coordinates between the tower crane base and the cargo, the plane coordinate data is further converted into spatial coordinate system coordinates (X, Y, Z) with the tower crane base as the origin.

[0053] The process of spatial posture information conversion of the present invention is as follows: For the cargo posture information, take the cargo center as a reference and record its rotation angles around the X-axis, Y-axis, and Z-axis; The posture information in the camera coordinate system is directly used without further conversion, ensuring that the real posture of the goods is displayed in the visualization model.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for detecting the position and posture of an object in a tower crane construction process based on multi-modality, characterized in that: The steps include: Step S1, obtaining the spatial position information of the cargo hoisted by the tower crane in real time through the RTK positioning system, wherein the spatial position information includes the latitude, longitude and altitude of the cargo, and is used to determine the precise position of the cargo in the geographic coordinate system; Step S2, obtaining the spatial posture and position information of the hoisted cargo in the camera coordinate system based on the tag detection function of the monocular camera; Step S3, coordinate system 1 of the acquired geographic coordinate system information and camera coordinate system information; Step S4, in the unified coordinate system, the spatial position information of the cargo hoisted by the tower crane obtained in real time by the RTK positioning system and the position information of the cargo hoisted in the camera coordinate system are integrated to generate the object spatial position information of the cargo hoisted; Step S5, based on the object spatial position information of the hoisted cargo, generates a 3D animation model of the hoisted cargo and 6D position and posture information in a preset coordinate system, and displays the movement trajectory and posture changes of the cargo during the tower crane construction process in real time.

2. The method for detecting object posture during tower crane construction based on multi-modality according to claim 1 is characterized in that: The real-time acquisition of spatial position information of cargo hoisted by a tower crane includes: When the cargo tag is beyond the camera's field of view, the RTK positioning system is used to obtain the spatial position information of the cargo hoisted by the tower crane; Output cargo location data in the form of latitude, longitude and altitude; The label detection function based on the monocular camera obtains the spatial posture and position information of the hoisted cargo in the camera coordinate system, including: When the cargo tag is within the camera's field of view, the cargo's spatial position and posture information is obtained through the monocular camera's tag detection function, and the cargo's position is represented by the spatial position and posture information of the camera coordinate system.

3. The method for detecting object posture during tower crane construction based on multi-modality according to claim 1 is characterized in that: The step of performing coordinate system 1 on the acquired geographic coordinate system information and the camera coordinate system information includes: Geographic coordinate system information conversion: The geographic coordinate system information of the goods hoisted by the tower crane is obtained in real time through the RTK positioning system, and the coordinates of the WGS84 geographic coordinate system are converted into the coordinates of the UTM projection or Gauss-Krüger projection to obtain the projection coordinates of the goods. The difference between the projection coordinates of the tower crane base and the obtained projection coordinates of the goods is used to convert the projection coordinates of the goods into coordinate points in the spatial coordinate system with the tower crane base as the origin, the vertical height as the Z axis, the Y axis facing the observer, and the right side of the Y axis as the X axis, where the altitude is directly converted to the height of the Z axis; Camera coordinate system information conversion: The angle between the monocular camera and the crane arm is used to convert the measured spatial position information of the cargo into spatial coordinates with the tower crane base as the origin, and the position information of the cargo under the camera is converted.

4. The method for detecting object posture during tower crane construction based on multi-modality according to claim 3 is characterized in that: In the process of geographic coordinate system information conversion, the process of converting the geographic coordinate system information of the goods from the coordinates of the WGS84 geographic coordinate system to the UTM projection coordinates is as follows: Determine the UTM projection zone based on longitude; Convert longitude and latitude to UTM coordinates: First convert latitude φ and longitude λ from degrees to radians; then use the major semi-axis defined by WGS84 a and flattening f Ellipsoid parameters, calculate the radius of curvature N ; Then use the Transverse Mercator projection formula to convert the longitude and latitude to X and Y in UTM coordinates.

5. The method for detecting object posture during tower crane construction based on multi-modality according to claim 3 is characterized in that: The camera coordinate system information conversion includes: Define the camera coordinate system of the monocular camera: the line of sight of the lens is the negative direction of the Z axis, the direction directly below the lens is the positive direction of the Y axis, and the direction directly to the right of the lens is the positive direction of the X axis; Assume the coordinates in the camera coordinate system are (Xc, Yc), and the converted 3D interface coordinates are (X, Y): ， ， Among them, θ is the angle of the camera.

6. The method for detecting object posture during tower crane construction based on multi-modality according to claim 1 is characterized in that: The spatial position information of the cargo hoisted by the tower crane is obtained in real time by the RTK positioning system in the unified coordinate system, and the position information of the cargo hoisted in the camera coordinate system is obtained and fused to generate the object spatial position information of the cargo hoisted, including: Based on the spatial position information of the cargo hoisted by the tower crane obtained in real time through the RTK positioning system and the past data of the position information of the hoisted cargo in the camera coordinate system, compare whether the position data has data fluctuations exceeding the threshold. If the threshold is exceeded, the other type of position information whose data fluctuations do not exceed the threshold is directly used as the position information of the cargo hoisted by the tower crane in space; if both types of position information do not exceed the threshold of data fluctuations, the two types of position information are summed and the average is taken as the final spatial position information of the object. If the data fluctuations of both types of position information exceed the threshold, the position information is ignored. If continuous data fluctuations occur, the sending device may prompt an error.

7. The method for detecting object posture during tower crane construction based on multi-modality according to any one of claims 1 to 6, characterized in that: According to the spatial position information of the hoisted cargo, a 3D animation model of the hoisted cargo and 6D position and posture information in a preset coordinate system are generated to display the movement trajectory and posture changes of the cargo during the tower crane construction process in real time, including: Utilize the collected cargo posture information and convert it into the coordinate system in the visualization scene; The open3D rendering technology is used to visualize the relative position information of the tower crane and the hoisted cargo, the posture information of the hoisted cargo, and the rotation information of the tower crane.

8. A multi-modal object posture detection device for tower crane construction process, characterized in that: include: A geographic coordinate acquisition module is used to obtain the spatial position information of the cargo hoisted by the tower crane in real time through the RTK positioning system. The spatial position information includes the latitude, longitude and altitude of the cargo, which is used to determine the precise position of the cargo in the geographic coordinate system; The camera coordinate acquisition module is used to obtain the spatial posture and position information of the hoisted cargo in the camera coordinate system based on the label detection function of the monocular camera; A coordinate system 1 module is used to coordinate the acquired geographic coordinate system information and the camera coordinate system information; The position information fusion module is used to fuse the spatial position information of the cargo hoisted by the tower crane obtained in real time through the RTK positioning system and the position information of the cargo hoisted in the camera coordinate system in a unified coordinate system to generate the object spatial position information of the cargo hoisted; The animation model generation module is used to generate a 3D animation model of the hoisted cargo and 6D position and posture information in a preset coordinate system according to the spatial position information of the hoisted cargo object, and to display the movement trajectory and posture changes of the cargo during the tower crane construction process in real time.

9. The multi-modal tower crane construction process object posture detection device according to claim 8, characterized in that: The coordinate system module includes: A geographic coordinate system information conversion module is used to obtain the geographic coordinate system information of the goods hoisted by the tower crane in real time through the RTK positioning system, convert the coordinates of the WGS84 geographic coordinate system into the coordinates of the UTM projection or the Gauss-Krüger projection, obtain the projection coordinates of the goods, and use the difference between the projection coordinates of the tower crane base and the obtained projection coordinates of the goods to convert the projection coordinates of the goods into coordinate points in the spatial coordinate system with the tower crane base as the origin and the vertical height as the Z axis, facing the observer as the Y axis, and the right side of the Y axis as the X axis, wherein the altitude is directly converted into the height of the Z axis; The camera coordinate system information conversion module is used to convert the measured spatial posture information of the cargo into spatial coordinates with the tower crane base as the origin by using the angle between the monocular camera and the crane arm, and to convert the position information of the cargo under the camera.

10. The multi-modal tower crane construction process object posture detection device according to claim 8, characterized in that: The coordinate system module includes: The projection coordinate acquisition module is used to convert the geographic coordinate system information of the goods hoisted by the tower crane obtained in real time through the RTK positioning system, and convert the longitude and latitude information into a plane coordinate system through the UTM projection coordinate system or the Gauss-Krüger projection coordinate system to obtain the projection coordinates of the goods; The projection coordinate conversion module is used to calculate the difference between the projection coordinates of the tower crane base and the projection coordinates of the cargo, and convert the projection coordinates of the cargo into a three-dimensional coordinate system with the tower crane base as the origin; The spatial posture conversion module is used to use the installation angle between the monocular camera and the tower crane to convert the spatial posture information of the cargo in the camera coordinate system into position data in a three-dimensional coordinate system with the tower crane base as the origin, so as to unify the camera coordinate system information with the geographic coordinate system information.

Images