Pose estimation method, device and system for modular integrated building

Abstract

The invention belongs to the technical field of modular integrated building automatic assembly, and provides a pose estimation method, device and system for modular integrated building assembly connection points, and the method comprises the steps: obtaining an RGB image and a depth image, collected by a visual perception camera, of a to-be-assembled building structure in a working area; fusing the RGB image and the depth image by using a neural network structure of a cross attention mechanism to obtain characterization features used for describing environment change of the working area; matching the depth image with a CAD (Computer Aided Design) model of the visible structural component by adopting a pose estimation algorithm under the condition that the assembly connection point is determined to be shielded based on the characterization characteristics, so as to obtain first pose information of the visible structural component relative to a camera coordinate system of the visual perception camera; and determining second pose information of the shielded assembly connection point relative to the camera coordinate system according to a pre-calibrated transformation matrix and the first pose information.

Description

Technical Field

[0001] This application belongs to the field of modular integrated building automated assembly technology, and more specifically, it relates to a pose estimation method, device and system for modular integrated buildings. Background Technology

[0002] Modular Integrated Construction (MiC) has become a core development direction of building industrialization due to its advantages such as high prefabrication, fast construction efficiency, and green environmental protection. In the automatic assembly process of modular integrated construction, the assembly connection points (such as bolt holes and tenons) of the modules to be assembled need to be precisely aligned with the target structure, and the position estimation of the assembly connection points is the core technical bottleneck to achieve automatic alignment.

[0003] Existing pose estimation schemes mostly rely on single-modal vision data. However, when assembly connection points are obstructed by hoisting ropes or adjacent modules, the pose of the assembly connection points cannot be directly solved. It is difficult to accurately describe changes in the working area environment (such as the assembly target being obstructed), resulting in large pose estimation errors in obstructed scenarios. Manual adjustment is required, which seriously restricts the efficiency and accuracy of automatic assembly of modular buildings. Summary of the Invention

[0004] The purpose of this application is to provide a pose estimation method, device and system for modular integrated buildings, which aims to solve the technical problem in the prior art that when the assembly connection point is occluded, the pose estimation error of the assembly connection point is large, which seriously restricts the efficiency and accuracy of automatic assembly of modular buildings.

[0005] To achieve the above objectives, according to a first aspect of this application, a pose estimation method for modular integrated buildings is provided, the method comprising: Acquire RGB and depth images of the building structure to be assembled in the working area captured by a visual perception camera, wherein the building structure to be assembled includes visible structural components and assembly connection points are set on the visible structural components; A neural network structure utilizing a cross-attention mechanism fuses the RGB image and the depth image to obtain representational features for describing environmental changes in the working area; When it is determined that the assembly connection point is occluded based on the characterization features, a pose estimation algorithm is used to match the depth image with the CAD model of the visible structural component to obtain the first pose information of the visible structural component relative to the camera coordinate system of the visual perception camera. Based on the pre-calibrated transformation matrix and the first pose information, the second pose information of the occluded assembly connection point relative to the camera coordinate system is determined, wherein the transformation matrix is ​​used to characterize the fixed pose of the assembly connection point in the local coordinate system of the visible structural component.

[0006] In one possible implementation, the first pose information includes a rotation matrix and a translation vector. The step of determining the second pose information of the occluded assembly connection point relative to the camera coordinate system based on a pre-calibrated transformation matrix and the first pose information includes: The transformation matrix, which is pre-calibrated based on the rigid connection relationship between the visible structural component and the assembly connection point, is invoked. The transformation matrix includes the rotation and translation components of the assembly connection point in the local coordinate system of the visible structural component. The local coordinate system is a coordinate system established with the visible structural component as the reference. Perform matrix multiplication on the rotation matrix and the rotation components to obtain the rotation parameters of the assembly connection point relative to the camera coordinate system; The rotation matrix and the translation component are multiplied to obtain the equivalent translation amount of the assembly connection point from the local coordinate system of the visible structural component to the camera coordinate system. The equivalent translation amount and the translation vector are added together to obtain the translation parameters of the assembly connection point relative to the camera coordinate system. Based on the rotation parameters and the translation parameters, the second pose information of the obscured assembly connection point relative to the camera coordinate system is determined.

[0007] In one possible implementation, the method further includes: The second pose information of the assembly connection point relative to the camera coordinate system is continuously acquired to form a pose sequence that is continuous over time. The pose information in the pose sequence is smoothed using a moving average filter to obtain a smoothed pose sequence. Based on the smoothed pose sequence, the pose difference between adjacent time steps is calculated; Based on the relationship between the pose difference and the preset difference threshold, the assembly robot plans the assembly actions for the building structure to be assembled. The preset difference threshold is set according to the assembly safety requirements of the building structure to be assembled and the pose change range of the assembly connection point of the building structure to be assembled under normal movement.

[0008] In one possible implementation, planning the assembly robot's assembly actions for the building structure to be assembled based on the relationship between the pose difference and a preset difference threshold includes: If the pose difference exceeds the preset difference threshold, the assembly robot is scheduled to stop performing assembly actions on the building structure to be assembled, wherein the pose difference exceeding the preset difference threshold indicates that there is an abnormal jump in the smoothed pose sequence. If the pose difference does not exceed the preset difference threshold, the assembly robot is planned to continue performing assembly actions on the building structure to be assembled, wherein the pose difference not exceeding the preset difference threshold indicates that there are no abnormal jumps in the smoothed pose sequence.

[0009] In one possible implementation, the neural network structure utilizing a cross-attention mechanism fuses the RGB image and the depth image to obtain representational features for describing environmental changes in the working area, including: The RGB image and the depth image are preprocessed, wherein the preprocessing is used to uniformly adjust the RGB image and the depth image to a preset size and normalize them respectively; The two parallel convolutional neural network branches in the neural network structure of the cross-attention mechanism extract the features of the preprocessed RGB image and depth image respectively, and obtain RGB feature vectors and depth feature vectors of the same dimension. The RGB feature vector and the depth feature vector are input into the cross-attention module in the neural network structure of the cross-attention mechanism. The RGB feature vector is used as the query vector and the depth feature vector is used as the key vector and value vector. The attention weight of the RGB feature to the depth feature is calculated to obtain the attention-weighted depth feature. The depth feature vector is used as the query vector and the RGB feature vector is used as the key vector and value vector. The attention weight of the depth feature to the RGB feature is calculated to obtain the attention-weighted RGB feature. The fused feature vector obtained by element-wise addition of the attention-weighted RGB features and the attention-weighted depth features is used as a representation feature to describe the environmental changes in the working area.

[0010] In one possible implementation, the step of matching the depth image with the CAD model of the visible structural component using a pose estimation algorithm to obtain the first pose information of the visible structural component relative to the camera coordinate system of the visual perception camera includes: Based on the depth image, extract the spatial distance information corresponding to the visible structural components; Geometric features are extracted from the CAD model of the visible structural components to generate CAD model data that matches the scale of the spatial distance information. The geometric features include edge contours, surface normals, and key vertex coordinates. An initial correspondence is established between the spatial distance information and the CAD model data using a coarse matching algorithm based on geometric features, thereby obtaining the initial pose of the visible structural component relative to the camera coordinate system; Based on the initial pose, the rotation matrix and translation vector are iteratively optimized by minimizing the Euclidean distance error between the sampling points in the spatial distance information and the corresponding points in the CAD model data until the error converges to a preset accuracy threshold. The converged rotation matrix and translation vector are used as the first pose information of the visible structure component relative to the camera coordinate system.

[0011] In one possible implementation, the method further includes: The representation features are input into a pre-defined fully connected layer neural network structure, wherein the fully connected layer contains at least two hidden layers and one output layer; The neural network structure of the fully connected layer is used to perform feature compression and classification mapping on the representation features, and output the occlusion probability value of the assembly connection point; If the occlusion probability value is greater than or equal to the occlusion determination threshold, then the assembly connection point is determined to be occluded.

[0012] In one possible implementation, the method further includes: If the assembly connection point is determined to be unobstructed based on the characterization features, the pose estimation algorithm is used to match the depth image with the CAD model of the assembly connection point to obtain the second pose information of the assembly connection point relative to the camera coordinate system.

[0013] According to a second aspect of this application, a pose estimation system for modular integrated buildings is provided, the system comprising: A visual perception camera is used to acquire RGB and depth images of the building structure to be assembled in the work area, wherein the building structure to be assembled includes visible structural components and assembly connection points are set on the visible structural components. An embedded AI computing unit, connected to the visual perception camera, is used to run the pose estimation method as described in any one of the claims to obtain second pose information of the assembly connection point relative to the camera coordinate system of the visual perception camera. The robot execution unit is connected to the embedded AI computing unit and uses an underactuated cable parallel assembly robot to plan the assembly robot's assembly actions for the building structure to be assembled based on the pose information.

[0014] According to a third aspect of this application, a pose estimation device for modular integrated buildings is provided, the device comprising: The acquisition unit is used to acquire RGB images and depth images of the building structure to be assembled in the working area captured by the visual perception camera, wherein the building structure to be assembled includes visible structural components and assembly connection points are set on the visible structural components. The fusion unit is used to fuse the RGB image and the depth image using a neural network structure with a cross-attention mechanism to obtain representational features for describing environmental changes in the working area; The matching unit is used to match the depth image with the CAD model of the visible structural component using a pose estimation algorithm when the assembly connection point is determined to be occluded based on the characterization features, so as to obtain the first pose information of the visible structural component relative to the camera coordinate system of the visual perception camera. The determining unit is used to determine the second pose information of the occluded assembly connection point relative to the camera coordinate system based on the pre-calibrated transformation matrix and the first pose information, wherein the transformation matrix is ​​used to characterize the fixed pose of the assembly connection point in the local coordinate system of the visible structural component.

[0015] According to a fourth aspect of this application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device causes the electronic device to perform the method as described in any one of the claims.

[0016] According to a fifth aspect of this application, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements the method as described in any one of the claims.

[0017] According to a sixth aspect of this application, a computer program product is provided that, when run on an electronic device, causes the electronic device to perform the method described in any one of the first aspects above.

[0018] The beneficial effects of the embodiments in this application compared with the prior art are: This application provides a pose estimation method for modular integrated buildings. It acquires RGB and depth images of the building structure to be assembled within a working area captured by a visual perception camera. The building structure includes visible structural components and assembly connection points set on these components. A neural network structure with a cross-attention mechanism is used to fuse the RGB and depth images to obtain representational features describing environmental changes in the working area. When the assembly connection points are determined to be occluded based on these representational features, a pose estimation algorithm is used to match the depth image with the CAD model of the visible structural components to obtain the first pose information of the visible structural components relative to the camera coordinate system of the visual perception camera. Based on a pre-calibrated transformation matrix and the first pose information, the second pose information of the occluded assembly connection points relative to the camera coordinate system is determined. The transformation matrix represents the fixed pose of the assembly connection points in the local coordinate system of the visible structural components. This method solves the technical problem in the prior art where the pose estimation error of the assembly connection points is large when they are occluded, severely restricting the efficiency and accuracy of automatic assembly of modular buildings.

[0019] It is understandable that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a pose estimation method for modular integrated buildings provided in an embodiment of this application; Figure 2 This is a flowchart illustrating an optional pose estimation method for modular integrated buildings provided in an embodiment of this application. Figure 3 This is a flowchart illustrating an optional pose estimation method for modular integrated buildings provided in an embodiment of this application. Figure 4 This is a flowchart illustrating an optional pose estimation method for modular integrated buildings provided in an embodiment of this application. Figure 5 This is a schematic diagram of a pose estimation system for modular integrated buildings provided in an embodiment of this application; Figure 6This is a schematic diagram of a pose estimation device for modular integrated buildings provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0023] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0024] It should also be understood that, in the description of this application, unless otherwise stated, the " / " used in the specification and appended claims indicates that the related objects are in an "or" relationship. For example, A / B can mean A or B. The "and / or" in this application is merely a description of the relationship between the related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0025] Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, but are only used for distinguishing descriptions, and the terms "first" and "second" do not necessarily imply that they are different, nor should they be construed as indicating or implying relative importance.

[0026] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0027] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0028] First, some terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0029] Modular Integrated Construction (MiC) is a new type of building that uses prefabricated modular units in a factory as its core and achieves efficient construction throughout the entire building lifecycle through standardized design, industrialized production, and assembly-based construction.

[0030] The above is a brief introduction to the terms used in the embodiments of this application, and will not be repeated below.

[0031] This application provides an example of a pose estimation method for modular integrated buildings. Please refer to [reference needed]. Figure 1 As shown, Figure 1 A schematic flowchart of a pose estimation method for modular integrated buildings provided in this application is shown. This is an example, not a limitation, and the method can be applied to or run in electronic devices. The method includes: S101, acquire the RGB image and depth image of the building structure to be assembled in the working area captured by the visual perception camera.

[0032] The building structure to be assembled includes visible structural components and assembly connection points set on the visible structural components.

[0033] S102 utilizes a neural network structure with a cross-attention mechanism to fuse RGB images and depth images, obtaining representational features to describe environmental changes in the work area.

[0034] S103, when the assembly connection point is determined to be occluded based on the representation features, a pose estimation algorithm is used to match the depth image with the CAD model of the visible structural component to obtain the first pose information of the visible structural component relative to the camera coordinate system of the visual perception camera.

[0035] S104, based on the pre-calibrated transformation matrix and the first pose information, determine the second pose information of the occluded assembly connection point relative to the camera coordinate system.

[0036] The transformation matrix is ​​used to characterize the fixed pose of the assembly connection point in the local coordinate system of the visible structural component.

[0037] In some embodiments, modular integrated buildings are constructed by breaking down a building into multiple prefabricated assembly units with independent structural functions or complete usability, i.e., building structures to be assembled (such as wall modules, floor modules, bathroom integrated modules, etc.). The structural construction, pipeline pre-embedding, equipment installation and interior decoration of the modules are completed in the factory. The prefabricated module units are then transported to the project site and quickly assembled into a complete building using reliable connection technologies (such as bolted connections, grouting connections, etc.). At the same time, the building integrates the functions of multiple systems such as building structure, electromechanical equipment, and decoration, realizing a construction mode of manufacturing modules in the factory and assembling the building on site.

[0038] This application provides a pose estimation method for modular integrated buildings, which aims to solve the problem that the pose of the assembly connection points of the building structure to be assembled is difficult to estimate directly due to occlusion during the automatic assembly process of modular integrated buildings. By integrating visual data and coordinate transformation logic, the method can realize accurate pose calculation of assembly connection points in occluded scenarios. The following describes the method in detail with specific scenarios.

[0039] First, a visual perception camera (e.g., a color-depth RGB-D camera) is used to acquire image data of the work area. This work area refers to the assembly work space surrounding the building structure to be assembled. The scope of this work area is based on the fact that the visual perception camera can effectively acquire data and cover the key areas of assembly and docking. It typically includes the movement path of the building structure to be assembled, the docking target area, and a surrounding range of 1-5 meters (the specific dimensions are adjusted according to the module size of the modular integrated building and the camera's field of view parameters).

[0040] It should be understood that the building structure to be assembled is an independent assembly unit in modular integrated architecture, such as prefabricated wall modules, floor slab modules, or beam and column modules. Specifically, the building structure to be assembled includes visible structural components and assembly connection points. Among them, visible structural components are local structures in the building structure to be assembled that have stable geometric features and are not easily obscured during the assembly process, such as the top steel frame of the wall module or the edge protrusion of the floor slab module; assembly connection points are key interfaces set on the visible structural components to realize the docking of modules, such as bolt holes, tenons, or grouting ports, and the spatial position of the assembly connection points is fixedly associated with the visible structural components.

[0041] In some embodiments, the visual perception camera simultaneously acquires RGB and depth images of the working area. The RGB image has a resolution of 1280×720 and includes color and texture information. The depth image has the same resolution as the RGB image and obtains the three-dimensional spatial distance information of pixels through infrared ranging. Simultaneously, the acquired RGB and depth images must completely contain the visible structural components and assembly connection points of the building structure to be assembled.

[0042] Subsequently, information from RGB and depth images is fused using a neural network structure to extract representational features that can describe changes in the working area environment (including the spatial state of assembly connection points and visible structural components, occlusion relationships, etc.). Specifically, the RGB and depth images are first preprocessed to uniformly adjust them to a size of 512×512 and eliminate scale differences through normalization. Then, two parallel convolutional branches based on an improved ResNet-18 are used to extract RGB feature vectors and depth feature vectors, each with a dimension of 1×256. Subsequently, a cross-attention module performs bidirectional fusion: using RGB features as the query and depth features as the key and value, and using depth features as the query and RGB features as the key and value, attention weights are calculated to obtain weighted features of the two classes. Finally, the weighted features are summed element-wise to obtain a one-dimensional vector of length 256, which is the representational feature of changes in the working area environment, effectively reflecting the state of assembly connection points, such as whether they are occluded.

[0043] Furthermore, the representational features are input into a fully connected classification network, which outputs the occlusion probability corresponding to the representational features. When it is determined that the assembly connection point is occluded based on the occlusion probability corresponding to the representational features, the pose estimation algorithm is used to match the depth image with the CAD model of the visible structural component to solve the first pose information (including rotation matrix and translation vector) of the visible structural component relative to the camera coordinate system of the visual perception camera.

[0044] In some embodiments, this application further calibrates a transformation matrix in advance based on the rigid connection relationship between the visible structural component and the assembly connection point. The pre-calibrated transformation matrix is ​​used to characterize the fixed pose of the assembly connection point in the local coordinate system of the visible structural component. When the assembly connection point is occluded, and the first pose information of the visible structural component relative to the camera coordinate system of the visual perception camera is determined, a second pose information of the occluded assembly connection point relative to the camera coordinate system is determined based on the pre-calibrated transformation matrix and the first pose information. This second pose information can be directly used to guide the automated assembly operation of modular integrated buildings to achieve precise docking.

[0045] In summary, by fusing RGB and depth image information to determine occlusion of assembly connection points, and by combining the first pose information of the visible structural components relative to the camera coordinate system of the visual perception camera with a pre-calibrated transformation matrix, the second pose information of the assembly connection points relative to the camera coordinate system can still be accurately estimated even when the assembly connection points are occluded, effectively improving the robustness and accuracy of automatic assembly of modular buildings.

[0046] In one possible implementation, the first pose information includes a rotation matrix and a translation vector, such as... Figure 2 As shown, based on the pre-calibrated transformation matrix and the first pose information, the second pose information of the occluded assembly connection point relative to the camera coordinate system is determined, including: S201 invokes a transformation matrix that has been pre-calibrated based on the rigid connection relationship between visible structural components and assembly connection points.

[0047] The transformation matrix includes the rotation and translation components of the assembly connection points in the local coordinate system of the visible structural component. The local coordinate system is a coordinate system established with the visible structural component as the reference.

[0048] S202, perform matrix multiplication on the rotation matrix and rotation components to obtain the rotation parameters of the assembly connection point relative to the camera coordinate system.

[0049] S203, multiply the rotation matrix and the translation component to obtain the equivalent translation amount of the assembly connection point from the local coordinate system of the visible structural component to the camera coordinate system.

[0050] S204, perform vector addition on the equivalent translation amount and the translation vector to obtain the translation parameters of the assembly connection point relative to the camera coordinate system.

[0051] S205, based on the rotation and translation parameters, determines the second pose information of the obscured assembly connection point relative to the camera coordinate system.

[0052] In this embodiment, firstly, a pre-calibrated transformation matrix is ​​invoked. This transformation matrix is ​​obtained offline based on the rigid connection between the visible structural component and the assembly connection point (the two are prefabricated integrated structures with fixed relative poses): a local coordinate system is established with the geometric feature points of the visible structural component (such as the endpoints of the central axis and corner vertices) as the reference (X-axis along the length direction of the component, Y-axis along the width direction, and Z-axis perpendicular to the surface of the component); a 3D laser scanner is used to collect the 3D coordinate point set of the visible structural component and the assembly connection point, and the pose parameters of the assembly connection point in this local coordinate system are calculated by the Iterative Closest Point (ICP) rigid registration algorithm to obtain the transformation matrix.

[0053] In some embodiments, the transformation matrix is ​​a 4×4 homogeneous matrix containing a 3×3 rotation component R0 (characterizing the rotation relationship of the assembly connection point relative to the local coordinate system of the visible structural component) and a 3×1 translation component t0 (characterizing the position coordinates of the assembly connection point in the local coordinate system, in meters).

[0054] Secondly, the rotation parameters of the assembly connection point relative to the camera coordinate system are calculated. The rotation matrix R1 (3×3 matrix) in the first pose information represents the rotation state of the visible structural component relative to the camera coordinate system. A matrix multiplication operation (R2=R1×R0) is performed between this matrix and the rotation component R0 in the transformation matrix to obtain the rotation parameter R2 of the assembly connection point relative to the camera coordinate system. The physical meaning of this operation is to transform the rotation relationship (R0) of the assembly connection point in the local coordinate system to the camera coordinate system through the rotation state (R1) of the visible structural component, reflecting the overall spatial attitude of the assembly connection point.

[0055] Next, the translation parameters of the assembly connection point relative to the camera coordinate system are calculated. This can be achieved in two steps: First, the rotation matrix R1 of the visible structural component is multiplied by the translation component t0 in the transformation matrix (R1×t0) to obtain the equivalent translation (3×1 vector) of the assembly connection point in the local coordinate system of the visible structural component after rotation transformation to the camera coordinate system. This process eliminates the influence of the rotation of the visible structural component on the translation coordinates of the assembly connection point. Second, the equivalent translation is added to the translation vector t1 (3×1 vector, representing the position of the visible structural component relative to the camera coordinate system) in the first pose information (t2=R1×t0+t1) to obtain the translation parameter t2 of the assembly connection point relative to the camera coordinate system. This parameter reflects the absolute position of the assembly connection point in the camera coordinate system.

[0056] Finally, the rotation parameter R2 and the translation parameter t2 are combined to form the second pose information of the assembly connection point relative to the camera coordinate system (represented as a 4×4 homogeneous transformation matrix: T2=[[R2,t2],[0,1]]). This second pose information can be directly output to the robot execution unit to guide the posture adjustment and docking actions of the building structure to be assembled, ensuring that high-precision assembly can still be achieved even if the assembly connection point is obscured.

[0057] One possible implementation is, such as Figure 3 As shown, the method also includes: S301 continuously acquires the second pose information of the assembly connection point relative to the camera coordinate system, forming a pose sequence that is continuous over time.

[0058] S302 uses a moving average filter to smooth the pose information in the pose sequence, resulting in a smoothed pose sequence.

[0059] S303 calculates the pose difference between adjacent time steps based on the smoothed pose sequence.

[0060] S304, based on the relationship between the pose difference and the preset difference threshold, plan the assembly robot's assembly actions for the building structure to be assembled.

[0061] In this embodiment, to further improve the stability and accuracy of automated assembly of modular integrated buildings, the attitude estimation method provided in this application embodiment also includes dynamic analysis and assembly action planning steps based on pose sequences.

[0062] Images are continuously acquired using a visual perception camera at a preset frequency (10-30Hz, i.e., one frame of data is acquired every 0.033-0.1 seconds, depending on the assembly speed and accuracy requirements), and the aforementioned steps are executed. The second pose information of the assembly connection point relative to the camera coordinate system is output in real time, forming a pose sequence that is continuous over time. Each element in this pose sequence contains rotation parameters (represented by Euler angles or rotation matrices) and translation parameters (three-dimensional coordinate vectors, in meters), reflecting the instantaneous spatial state of the assembly connection point during the movement.

[0063] A moving average filter is used to smooth the pose sequence. For example, the filter window size is set to 3-5 consecutive sampling points (the window size is adjusted according to the sensor noise characteristics; a larger window is used when the noise is high). A moving average is calculated (i.e., smoothing) for the pose parameters (including the x, y, and z components of the translation vector and the α, β, and γ components of the Euler angles of rotation) at each time step in the pose sequence. For example, for the translation component x, the smoothed value at time t is x_smooth(t) = [x(t-2) + x(t-1) + x(t)] / 3 (when the window size is 3). Similarly, for the Euler angles of rotation α, the smoothed value at time t is α_smooth(t) = [α(t-2) + α(t-1) + α(t)] / 3 (when the window size is 3). This smoothing process filters out high-frequency jitter caused by sensor measurement noise and image matching errors, while preserving the trend characteristics of pose changes, resulting in a smoothed pose sequence.

[0064] In some embodiments, based on the smoothed pose sequence, the pose difference between two adjacent sampling times (e.g., time t and time t-1) is calculated. Specifically, this includes: calculating the Euclidean distance between the translation parameters at time t and time t-1, i.e., Δd = sqrt((x(t) - x(t-1))² + (y(t) - y(t-1))² + (z(t) - z(t-1))²), in millimeters, reflecting the spatial movement of the assembly connection point; converting the rotation matrices at time t and time t-1 into axis-angle representations (rotation axis u and rotation angle θ), and calculating the difference in rotation angles between the two times, Δθ = |θ(t) - θ(t-1)|, in degrees, reflecting the rotation amplitude of the assembly connection point. The combined translation difference Δd and rotation difference Δθ are then used to evaluate the motion stability of the assembly connection point.

[0065] In some embodiments, the preset difference threshold can be set based on the assembly accuracy requirements of modular buildings, such as a translation threshold Δd. threshold =2mm, rotation threshold Δθ threshold =0.5°, and dynamically plan the motion of the assembly robot (such as a cable hoisting robot or a multi-axis robotic arm) based on the relationship between the pose difference and the threshold: When Δd≤Δd threshold And Δθ≤Δθ threshold When the assembly connection point is determined to be stable, the assembly robot is planned to operate in a preset high-speed assembly mode (e.g., translational speed 50mm / s, rotational speed 1° / s) to quickly approach the docking target; when Δd>Δd threshold Or Δθ>Δθ thresholdWhen the position fluctuates slightly (but not exceeding twice the threshold), the assembly robot is switched to a low-speed precision mode (e.g., translation speed 10mm / s, rotation speed 0.2° / s) to reduce the impact of motion inertia on accuracy; when Δd > 2 × Δd threshold Or Δθ > 2 × Δθ threshold If the pose fluctuation is abnormal (possibly due to mechanical vibration or environmental interference), the assembly robot will pause its current action and maintain its posture. Once the pose difference at three consecutive sampling times returns to within the threshold, the assembly will resume in low-speed mode.

[0066] By smoothing the pose sequence and analyzing dynamic differences, adaptive planning of assembly actions is achieved. This ensures assembly efficiency while avoiding docking deviations caused by pose fluctuations. It also ensures that the assembly connection points (such as bolt holes and bolts) of the building structures to be assembled (such as wall modules and floor modules) achieve sub-millimeter precision docking, further improving the robustness and accuracy of the modular integrated building automatic assembly system.

[0067] In one possible implementation, the assembly robot's assembly actions for the building structure to be assembled are planned based on the relationship between the pose difference and a preset difference threshold, including: If the pose difference exceeds the preset difference threshold, the assembly robot will stop performing the assembly action on the building structure to be assembled. The pose difference exceeding the preset difference threshold indicates that there is an abnormal jump in the smoothed pose sequence. If the pose difference does not exceed the preset difference threshold, the assembly robot will continue to perform assembly actions on the building structure to be assembled. The fact that the pose difference does not exceed the preset difference threshold indicates that there are no abnormal jumps in the smoothed pose sequence.

[0068] In this embodiment, the assembly robot's assembly actions are planned based on the relationship between the pose difference and the preset difference threshold. Specifically, the assembly process is controlled and its continuity is ensured through pose stability judgment.

[0069] In one possible implementation, the preset difference threshold is set based on the assembly safety requirements of the building structure to be assembled and the range of pose changes of the assembly connection points under normal movement. For example, the translation difference threshold is set to 2mm (i.e., the Euclidean distance between translation parameters at adjacent moments exceeds 2mm and is considered abnormal), and the rotation difference threshold is set to 1° (i.e., the difference in rotation angle between adjacent moments exceeds 1° and is considered abnormal). Furthermore, the logic must be met that if either the translation difference or the rotation difference exceeds the corresponding threshold, it is considered that the pose difference exceeds the standard.

[0070] When the pose difference calculated based on the smoothed pose sequence does not exceed the preset difference threshold, the motion state of the assembly connection point conforms to the expected pattern. That is, the building structure to be assembled is translated or rotated at a stable speed under the drive of the robot (such as moving towards the docking target at a constant speed), and the pose change originates from normal assembly motion rather than abnormal interference. At this time, a continue execution command is sent to the assembly robot (such as a cable parallel robot or a multi-axis robotic arm). The assembly robot continues to drive the building structure to be assembled according to the preset trajectory planning parameters (such as current speed and acceleration), ensuring the continuity of the assembly process.

[0071] When the pose difference exceeds a preset difference threshold, an abnormal jump occurs in the smoothed pose sequence. This jump is usually caused by unexpected factors, such as: shooting deviation caused by vibration of the visual perception camera, mismatch during the depth image matching process, and loosening of the rigid connection between visible structural components and assembly connection points (such as welding defects in prefabricated structures). This leads to a significant deviation between the actual pose and the calculated pose of the assembly connection points. If assembly continues, it may cause module collisions or docking failures. At this time, a stop execution command is sent to the assembly robot. After receiving the stop execution command, the robot immediately stops all driving actions (such as cutting off the power output of the cable drive motor and locking the robotic arm joints) and maintains the current spatial posture of the building structure to be assembled (achieved through a brake or self-locking structure). At the same time, an abnormal alarm is triggered (such as audible and visual prompts or system log recording) so that the cause of the abnormality can be investigated manually later to ensure the safety of the assembly process.

[0072] In summary, by comparing the pose difference with the preset threshold, it is possible to ensure the continuity of normal assembly while responding to abnormal states and stopping the action in a timely manner, effectively reducing the failure risk of automated assembly of modular integrated buildings.

[0073] In one possible implementation, a neural network structure utilizing a cross-attention mechanism fuses RGB and depth images to obtain representational features for describing environmental changes in the work area, including: The RGB image and depth image are preprocessed, wherein the preprocessing is used to uniformly adjust the RGB image and depth image to a preset size and normalize them respectively; The cross-attention mechanism uses two parallel convolutional neural network branches to extract features from the preprocessed RGB and depth images, respectively, to obtain RGB and depth feature vectors of the same dimension. The RGB feature vector and the depth feature vector are input into the cross-attention module in the neural network structure of the cross-attention mechanism. The RGB feature vector is used as the query vector and the depth feature vector is used as the key vector and value vector. The attention weight of the RGB feature on the depth feature is calculated to obtain the attention-weighted depth feature. The depth feature vector is used as the query vector and the RGB feature vector is used as the key vector and value vector. The attention weight of the depth feature on the RGB feature is calculated to obtain the attention-weighted RGB feature. The fused feature vector obtained by element-wise addition of the attention-weighted RGB features and the attention-weighted depth features is used as a representation feature to describe environmental changes in the working area.

[0074] In this embodiment, the process of fusing RGB images and depth images using a neural network structure with a cross-attention mechanism to obtain representational features aims to strengthen the correlation features between the two types of images through bidirectional information interaction, and accurately capture environmental changes in the working area (such as the visibility of assembly connection points, the spatial state of structural components, etc.).

[0075] First, the RGB and depth images are preprocessed to unify the data format for neural network input: the acquired RGB image (original resolution 1280×720) and depth image (same resolution) are adjusted to a preset size (512×512) using a bicubic interpolation algorithm to ensure spatial alignment. The adjusted images are then normalized: the RGB image is linearly mapped based on the pixel value range (0-255), normalizing the pixel value of each channel to the [0,1] interval to eliminate the influence of differences in illumination intensity; the depth image is normalized based on the actual depth range of the working area (e.g., 0-8 meters, preset according to the assembly scene), mapping the depth value to the [0,1] interval to avoid interference from different distance scales on feature extraction. After preprocessing the RGB and depth images, the size and numerical range of the two types of images are consistent, laying the foundation for subsequent feature fusion.

[0076] Secondly, basic features are extracted through parallel convolutional neural network branches. The neural network structure of the cross-attention mechanism contains two identical convolutional neural network branches (both based on an improved ResNet-34 network), used to extract features from RGB and depth images respectively: The RGB branch performs convolution (3×3 convolutional kernel, stride 2), batch normalization, ReLU activation, and four residual block operations on the preprocessed RGB image sequentially, preserving semantic information such as color distribution and surface texture (e.g., prefabricated textures on component surfaces) through multi-scale feature extraction, ultimately outputting a 64×64×128 RGB feature map (height 64, width 64, number of channels 128); The depth branch performs the same network operations on the preprocessed depth image, focusing on extracting geometric information such as spatial distance and contour morphology (e.g., depth abrupt changes at component edges), outputting a depth feature map (64×64×128) with dimensions completely identical to the RGB feature map. The identical dimensions of the two types of feature vectors ensure the feasibility of subsequent attention calculations. The RGB feature map and the depth feature map are compressed into 1×128 RGB feature vectors and depth feature vectors respectively through pooling operations.

[0077] Next, bidirectional feature interaction is achieved through a cross-attention module, which is the core of feature fusion. The connection between RGB features and deep features is established through bidirectional attention calculation: in the first direction, the RGB feature vector is used as the query vector, and the deep feature vectors are used as the key and value vectors, respectively. The feature channels of the query and key are compressed from 128 dimensions to 64 dimensions and then matrix multiplication is performed. This is achieved using the scaled dot product attention formula (attention weight = softmax[(Query × ... )). The first approach calculates the attention level of each RGB feature point to the depth feature point, then multiplies this weight by the value to obtain the attention-weighted depth feature (highlighting depth information related to RGB features). The second approach uses the depth feature vector as the query and the RGB feature vectors as the key and value, respectively. Similarly, the channels are first compressed to 64 dimensions, then the attention weights are calculated to obtain the weighted RGB feature (1×64 dimension, highlighting color information related to depth features). This bidirectional weighting process not only achieves targeted enhancement of both types of features but also unifies the feature channels to 64 dimensions, making the fusion result more suitable for describing environmental changes.

[0078] Subsequently, the weighted features are fused and their dimensions compressed. The attention-weighted RGB features (1×64) and the weighted depth features (1×64) are added element-wise (the feature values ​​at corresponding positions are directly summed) to obtain a preliminary fused feature vector (keeping the dimension at 1×64 and retaining the core information after bidirectional attention interaction). This process achieves feature aggregation through parameter interaction input operations to reduce redundant information and strengthen core features (such as the local features of assembly connection points). This one-dimensional fused feature vector is used to describe the environmental changes in the working area, containing the correlation information between the color texture of the RGB image and the spatial distance of the depth image. It can effectively reflect key environmental states such as whether the assembly connection points are occluded and the spatial pose of visible structural components, providing high-dimensional feature support for subsequent occlusion judgment.

[0079] In summary, through the bidirectional interaction and feature fusion of the cross-attention mechanism, this process can fully explore the complementary information of RGB images and depth images, enabling the generated representation features to have a stronger ability to describe the environment, and providing a reliable basis for occlusion judgment in modular integrated building assembly scenarios.

[0080] In one possible implementation, a pose estimation algorithm is used to match the depth image with the CAD model of the visible structural component to obtain the first pose information of the visible structural component relative to the camera coordinate system of the visual perception camera, including: Based on the depth image, extract the spatial distance information corresponding to the visible structural components; Geometric features are extracted from the CAD model of visible structural components to generate CAD model data that matches the scale of spatial distance information. The geometric features include edge contours, surface normals, and key vertex coordinates. A coarse matching algorithm based on geometric features is used to establish an initial correspondence between spatial distance information and CAD model data, thereby obtaining the initial pose of visible structural components relative to the camera coordinate system. Based on the initial pose, the rotation matrix and translation vector are iteratively optimized by minimizing the Euclidean distance error between the sampling points in the spatial distance information and the corresponding points in the CAD model data until the error converges to the preset accuracy threshold. The converged rotation matrix and translation vector are used as the first pose information of the visible structure component relative to the camera coordinate system.

[0081] In some embodiments, the process of matching a depth image with a CAD model of a visible structural component using a pose estimation algorithm to obtain the first pose information aims to solve the spatial pose of the visible structural component relative to the camera coordinate system through precise alignment of three-dimensional geometric features, providing a benchmark for the subsequent derivation of the pose of the occluded assembly connection point.

[0082] First, spatial distance information corresponding to visible structural components is extracted from the depth image. Based on the depth value of each pixel in the depth image (reflecting the distance between that point and the camera), combined with the intrinsic parameter matrix of the visual perception camera (including focal length), and principal point coordinates ), through coordinate transformation formula ( Where (u,v) represents pixel coordinates and Z represents depth value, the 2D depth image is converted into a 3D point cloud (point cloud format is a set of 3D coordinates (x,y,z)). Since the original point cloud may contain noise (such as isolated points due to depth measurement errors) and redundant data (such as points in the background area), preprocessing is required: a statistical filtering algorithm (setting the number of neighboring points to 50 and the standard deviation multiple to 2.0) is used to remove noise points (points whose distance from neighboring points is far beyond the mean); voxel downsampling (voxel size set to 8mm, adjusted according to the size and assembly accuracy requirements of the visible structural components) reduces the number of point clouds (retaining approximately 3000 points), reducing the computational load of subsequent matching while ensuring the integrity of geometric features. The final spatial distance information only contains surface points of the visible structural components, accurately reflecting the spatial morphology of the visible structural components.

[0083] Secondly, geometric features are extracted from the CAD model of the visible structural components to generate a virtual point cloud. The CAD model is a 3D design model of the building structure to be assembled, from which the geometric parts of the visible structural components (such as the steel frame structure of the wall module) are extracted. To achieve matching with spatial distance information, key geometric features need to be extracted from the CAD model: edge contour points of the components (such as the intersection points and corner points of the steel frame edges) are extracted using a mesh edge detection algorithm; surface normal vectors (reflecting the orientation of the component surface) are extracted using a surface normal calculation tool (based on the average of the normal vectors of adjacent triangular facets); and key vertex coordinates (such as the endpoints and center of symmetry of the frame) are retained using a vertex filtering algorithm. Based on these geometric features, a uniform sampling strategy is used to generate a virtual point cloud (containing approximately 5000 points) from the surface of the CAD model, and the scale of the virtual point cloud (in meters) is ensured to be consistent with the spatial distance information (calibrated by a model scaling factor to ensure matching with the actual structural dimensions), so that the two have a basis for direct matching.

[0084] Next, a coarse matching algorithm based on geometric features is used to establish the initial correspondence and solve the initial pose. The core of coarse matching is to quickly find the approximate alignment between the 3D point cloud and the virtual point cloud, avoiding the subsequent fine matching from getting stuck in local optima. Specifically, the Random Sample Consensus (RANSAC) algorithm is used: four sets of points are randomly selected from both the 3D point cloud and the virtual point cloud (each set of points must satisfy geometric feature similarity, such as the distance difference between points ≤20mm and the difference in normal angle ≤5°), and these four sets of points are assumed to be corresponding point pairs; the transformation matrix (including rotation matrix and translation vector) that aligns these four sets of point pairs is calculated using the Singular Value Decomposition (SVD) algorithm; this transformation matrix is ​​applied to the CAD model data to count the number of points in the virtual point cloud and the 3D point cloud (distance ≤30mm), and after 1000 iterations, the transformation matrix with the largest number of in-place points is retained as the initial pose of the visible structural component relative to the camera coordinate system. The error of this initial pose is usually controlled within 50mm, providing a reliable starting point for subsequent fine matching.

[0085] Subsequently, fine matching is performed based on the initial pose to optimize the pose parameters. Fine matching employs an improved Iterative Closest Point (ICP) algorithm, achieving high-precision alignment by minimizing the distance error between point clouds: Based on the initial pose, the virtual point cloud is transformed to the camera coordinate system, and the Euclidean distance between each sampling point in the 3D point cloud and the nearest point in the virtual point cloud is calculated (using a kd-tree to accelerate the nearest point search); a sum of squared distance errors function (objective function) is constructed, and the rotation matrix R and translation vector t are iteratively adjusted using the Levenberg-Marquardt optimization algorithm to minimize the objective function value; after each iteration, the position of the virtual point cloud is updated, and the distance error is recalculated until the convergence condition is met, i.e., the error change between two consecutive iterations is ≤0.1mm, or the number of iterations reaches 100 (whichever comes first). This reduces the pose error to within 1mm, meeting the precision requirements of modular building assembly.

[0086] Finally, the converged rotation matrix and translation vector are used as the first pose information. This first pose information fully represents the spatial pose (rotation matrix) and position (translation vector) of the visible structural component relative to the camera coordinate system of the visual perception camera, where the rotation matrix is ​​a 3×3 orthogonal matrix (satisfying...). The vector det(R) = 1 describes the rotation angle of the component around the three axes of the camera coordinate system; the translation vector is a 3×1 vector (in meters) describing the x, y, and z coordinates of the component in the camera coordinate system. This enables precise alignment between the depth image and the CAD model, and the resulting first pose information possesses high accuracy and stability, providing a reliable foundation for pose estimation of occluded connection points in modular integrated building assembly scenarios.

[0087] In one possible implementation, the method further includes: The representation features are input into a pre-defined fully connected layer neural network structure, which contains at least two hidden layers and one output layer. The neural network structure with fully connected layers performs feature compression and classification mapping on the representation features, and outputs the occlusion probability value of the assembly connection point. If the occlusion probability value is greater than or equal to the occlusion determination threshold, the assembly connection point is determined to be occluded.

[0088] In this embodiment, the process of determining whether an assembly connection point is occluded based on its representational features is achieved through a preset fully connected layer neural network structure. This structure maps high-dimensional environmental representational features into quantified occlusion probabilities, thereby accurately determining the visibility status of the connection point.

[0089] First, a pre-defined fully connected neural network structure is constructed. This structure takes a one-dimensional feature vector (64 in length, containing RGB and deep-fused environmental features) as input and includes at least two hidden layers and one output layer: the first hidden layer has 128 neurons and uses the ReLU activation function (to extract higher-order correlations between features through non-linear mapping); the second hidden layer has 64 neurons and also uses the ReLU activation function (to further compress the feature dimension and enhance key information, such as edge features of occluded regions); the output layer has one neuron and uses the Sigmoid activation function (to map the output value to the [0,1] interval, directly representing the occlusion probability of the assembly connection point). Feature transfer between network layers is achieved through a fully connected weight matrix, and the weight parameters are optimized through offline training.

[0090] Secondly, feature compression and classification mapping are performed through a fully connected neural network. After inputting the representation feature vector (length 64) into the network, the first hidden layer performs a weighted summation transformation on the input features (multiplying the weight matrix with the feature vector and adding a bias), and then activates it with ReLU to output 128-dimensional intermediate features (filtering irrelevant features and retaining key patterns related to occlusion, such as the absence of depth mutations in the assembly connection point region). The second hidden layer performs a weighted summation transformation and ReLU activation on the 128-dimensional features again, outputting 64-dimensional refined features (focusing on the core clues for occlusion judgment, such as abnormal pixel values ​​at the connection point position in the RGB image or the absence of distance in the corresponding region in the depth image). After the output layer performs a weighted summation transformation on the 64-dimensional features, it activates it with Sigmoid to output a single occlusion probability value. The closer this occlusion probability value is to 1, the higher the probability that the assembly connection point is occluded; the closer this occlusion probability value is to 0, the lower the probability that the assembly connection point is occluded, and the more likely it is to be visible.

[0091] Next, an occlusion judgment threshold is set and the judgment is performed. The occlusion judgment threshold is determined based on the assembly scenario characteristics of modular integrated buildings and the statistical analysis of training data: a large number of samples containing "fully visible assembly connection points", "partially occluded", and "completely occluded" are collected (the number of samples for each state is no less than 5000). The representational features of the samples are input into the network for training (using the cross-entropy loss function and the Adam optimizer, iterating for 100 epochs until the loss converges). The judgment accuracy (such as the recognition rate of occlusion state) and recall (such as avoiding missing real occlusion) corresponding to different thresholds (0.5-0.8) are tested on the validation set. Finally, the optimal threshold that balances the two is selected (usually set to 0.7, that is, when the occlusion probability is ≥0.7, it is judged as occlusion).

[0092] Finally, based on the comparison between the occlusion probability value and the threshold, the judgment conclusion is output: if the occlusion probability value of the assembly connection point is greater than or equal to the optimal threshold of 0.7, the connection point is determined to be occluded (possibly due to occlusion by hoisting ropes, obstruction by adjacent modules, or camera view deviation), and the pose estimation process is triggered (the pose of the connection point is derived from the visible structural components); if the occlusion probability value of the assembly connection point is less than the optimal threshold of 0.7, the assembly connection point is determined not to be occluded, and the pose estimation algorithm can be directly used to solve the pose of the assembly connection point relative to the camera coordinate system based on the characteristics of the assembly connection point itself.

[0093] In summary, by classifying and mapping the representation features through a fully connected neural network, this process achieves the quantitative judgment of the occlusion state of assembly connection points. The judgment accuracy (≥95%) can meet the requirements of real-time performance and reliability for automatic assembly of modular integrated buildings, and provides a key basis for the dynamic switching of subsequent pose estimation strategies.

[0094] One possible implementation is, such as Figure 4 As shown, the method also includes: S401, based on the characteristic features, if it is determined that the assembly connection point is not occluded, a pose estimation algorithm is used to match the depth image with the CAD model of the assembly connection point to obtain the second pose information of the assembly connection point relative to the camera coordinate system.

[0095] In this embodiment, when it is determined that the assembly connection point is not occluded based on the characterization features, that is, the occlusion probability value is less than the occlusion determination threshold, the geometric features of the connection point can be directly observed by the visual perception camera. The pose estimation algorithm is used to match the depth image with the CAD model of the assembly connection point, and the second pose information of the assembly connection point relative to the camera coordinate system is directly solved.

[0096] First, the spatial distance information corresponding to the assembly connection points is extracted. Based on the spatial semantic information contained in the representation features obtained in step two, the pixel regions of the assembly connection points in the depth image are located, such as the circular region of bolt holes and the rectangular outline of tenons. The depth values ​​of this region are converted into three-dimensional coordinates using the camera intrinsic parameter matrix to generate the original three-dimensional point cloud of the assembly connection points, including key sampling points on the surface of the connection points, such as hole edge points and boss vertices. The original point cloud is preprocessed: radius filtering (radius 0.05 meters, with no less than 10 neighboring points) is used to remove isolated points caused by depth measurement noise; voxel downsampling (voxel size 2 mm, adapted to the fine structure of the connection points) retains about 500 points, reducing the computational load while preserving feature details (such as the circular outline of bolt holes and hole diameter), to obtain the spatial distance information used to match the assembly connection points.

[0097] Secondly, matching features are extracted from the CAD model of the assembly connection points. The CAD model is a 3D design model (in STL format) of the assembly connection points, containing precise geometric parameters of the connection points (such as the diameter, depth, and center coordinates of bolt holes, and the length, width, height, and positioning reference of tenons). Geometric features corresponding to spatial distance information are extracted from the model: edge detection tools are used to extract the contour edge points of the connection points (such as the circumference points of holes and the edge points of bosses); surface normal vectors are calculated (reflecting the orientation of the surface of the connection point, such as the normal of the inner wall of the hole pointing to the center); key feature points are selected (such as the center point of the hole and the vertex of the boss). Based on these features, uniform sampling is used to generate a virtual point cloud (containing approximately 1000 points) with the same scale as the 3D point cloud, and the geometric attributes of each point are labeled (such as whether it is an edge point and the direction of the normal) to ensure that the virtual point cloud can accurately reflect the fine structure of the connection points.

[0098] Next, a high-precision pose estimation algorithm is used for point cloud matching. Since assembly connection points are usually standardized structures manufactured with high precision (such as bolt holes with a tolerance of ±0.1mm), the FoundationPose algorithm, optimized for high-precision pose estimation of small targets, is used to achieve matching: First, an initial correspondence is established between the 3D point cloud and the virtual point cloud of the connection point through feature point matching (based on the curvature and normal direction similarity of the points). The RANSAC algorithm is used to remove outlier point pairs to obtain the initial pose (rotation matrix R0, translation vector t0), with the error controlled within 5mm. Starting from the initial pose, the Euclidean distance error between the sampled points in the 3D point cloud and the corresponding points in the virtual point cloud is minimized through iterative optimization (combining pixel gradient information from the depth image to strengthen the matching weight of edge points). During the iteration process, the Gauss-Newton method is used to optimize the rotation matrix and translation vector until the error change between two consecutive iterations is ≤0.05mm or the number of iterations reaches 50, ensuring that the pose accuracy meets the docking requirements of modular buildings (such as the fitting accuracy between bolt holes and bolts needs to be ≤0.5mm).

[0099] Finally, the converged pose parameters are output as the second pose information. This second pose information includes the rotation matrix R2 (a 3×3 orthogonal matrix describing the spatial pose of the connection point, such as the axial direction of the bolt hole) and the translation vector t2 (a 3×1 vector in meters describing the precise position of the connection point in the camera coordinate system, such as the center coordinates of the hole) of the assembly connection point relative to the camera coordinate system. Since it is directly based on the geometric features of the connection point itself, the accuracy of this pose information (translation error ≤ 0.3 mm, rotation error ≤ 0.1°) is usually higher than the derivation results under occlusion scenarios, and can be directly used to guide the assembly robot to perform high-precision docking actions (such as adjusting the posture of the robotic arm to accurately insert the bolt into the bolt hole of the connection point).

[0100] In summary, when the assembly connection points are not occluded, high-precision second pose information can be quickly obtained by directly matching their depth point cloud with the CAD model. This complements the derivation strategy in occluded scenarios, jointly ensuring the robustness and accuracy of connection point pose estimation during the automatic assembly of modular integrated buildings.

[0101] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0102] Corresponding to the pose estimation method for modular integrated buildings in the above embodiments, Figure 5 This is a structural schematic diagram of a pose estimation system for modular integrated buildings provided in an embodiment of this application, as shown below. Figure 5 As shown, the pose estimation system 500 for modular integrated buildings includes: The visual perception camera 501 is used to acquire RGB and depth images of the building structure to be assembled in the work area, wherein the building structure to be assembled includes visible structural components and assembly connection points set on the visible structural components.

[0103] The embedded AI computing unit 502, connected to the visual perception camera 501, is used to run a pose estimation method, such as any one of them, to obtain second pose information of the assembly connection point relative to the camera coordinate system of the visual perception camera.

[0104] The robot execution unit 503 is connected to the embedded AI computing unit 502 and uses an underactuated cable parallel assembly robot to plan the assembly robot's assembly actions for the building structure to be assembled based on the pose information.

[0105] In some embodiments, a visual perception camera is used to acquire real-time visual data of the building structure to be assembled within the working area, providing raw information support for subsequent pose calculations. The visual perception camera can be an industrial-grade color-depth RGB-D camera, supporting simultaneous acquisition of RGB and depth images. The RGB image resolution is set to 1280×720, the frame rate is maintained at 30Hz, and a global shutter is used to eliminate image blurring during the movement of the building structure to be assembled, ensuring clear capture of the color distribution and texture details of the structural surface, such as prefabricated markings on component surfaces and characteristic textures around assembly connection points.

[0106] The depth image resolution is consistent with the RGB image. Depth measurement is achieved using active infrared structured light, covering a range of 0.1 meters to 10 meters. Within the core assembly range of 1 to 5 meters, the measurement accuracy does not exceed 2%, accurately acquiring the 3D spatial distance information between assembly connection points and visible structural components. The visual perception camera is rigidly fixed above the assembly area, with the bracket height adjusted to 3 to 5 meters based on the size of the module to be assembled. The lens is pointed towards the movement path of the building structure to be assembled (e.g., below the hoisting trajectory), ensuring the camera's field of view completely covers the working area. The working area must include the starting position of the visible structural components, the docking target area, and the range of movement between them, avoiding blind spots in visual acquisition. Before system startup, the camera is calibrated using the Zhang Zhengyou calibration method to obtain the intrinsic parameter matrix (including focal length and principal point coordinates) and distortion coefficients. Software algorithms are used to eliminate the influence of optical distortion on subsequent pose calculations. For data transmission, the RGB-D camera connects to the embedded AI computing unit via a USB 3.2 interface, synchronously transmitting RGB images, depth images, and timestamp information. During transmission, it is necessary to ensure that the time error between the two types of images does not exceed 1 millisecond and to achieve pixel-level alignment in space, so as to provide a precise data foundation for subsequent image feature fusion.

[0107] In some embodiments, the embedded AI computing unit serves as the decision layer of the pose estimation system. It is responsible for running any of the aforementioned pose estimation methods, processing the image data acquired by the visual perception camera, and ultimately outputting the second pose information of the assembly connection point relative to the camera coordinate system of the visual perception camera. For example, the hardware configuration of this embedded AI computing unit can employ a high-performance embedded computing platform, equipped with an eight-core CPU, 32GB of memory, and a 1TB NVMe solid-state drive. This hardware configuration can meet the needs of computationally intensive tasks such as neural network inference and point cloud matching, ensuring that the processing time for a single frame image does not exceed 50 milliseconds, thus guaranteeing the real-time performance of the pose estimation system. Furthermore, the software layer of this embedded AI computing unit can integrate multiple functional modules, each with a clear division of labor and collaborative operation. The image preprocessing module processes RGB and depth images, adjusting both types of images to a preset size of 512×512 and performing normalization operations (RGB images are normalized based on pixel value range, and depth images are normalized based on the depth range of the working area). Simultaneously, it performs image distortion correction based on camera calibration parameters. The feature fusion module deploys a neural network based on a cross-attention mechanism. The network weights are optimized through offline training, effectively fusing the color and texture features of the RGB images with the spatial distance features of the depth images, outputting representational features describing changes in the working area environment. The occlusion detection module runs a fully connected neural network. After inputting the representational features into the network, it outputs the occlusion probability value of the assembly connection points through feature compression and classification mapping, combined with a preset occlusion judgment... A threshold is set to determine whether the connection point is occluded; the pose estimation module includes a point cloud extraction submodule, a CAD model matching submodule, and a coordinate transformation submodule. The point cloud extraction submodule generates spatial distance information from the depth image and performs noise reduction and downsampling processing. The CAD model matching submodule aligns the point cloud with the CAD model through coarse matching (random sampling consensus algorithm) and fine matching (iterative nearest point algorithm). The coordinate transformation submodule derives the pose of the occluded connection point based on a pre-calibrated transformation matrix; the data communication module establishes communication with the robot execution unit through the industrial Ethernet EtherCAT protocol, and transmits the second pose information of the assembly connection point relative to the camera coordinate system of the visual perception camera at a frequency of 30Hz, ensuring that the robot execution unit can acquire pose data in real time.

[0108] The core function of the embedded AI computing unit is to receive image data transmitted by the visual perception camera, run the pose estimation method according to the logical flow of "image preprocessing → feature fusion → occlusion judgment → pose estimation / derivation", calculate the pose information of the assembly connection point relative to the camera coordinate system in real time (output in the form of rotation matrix and translation vector, with the rotation matrix unit in radians and the translation vector unit in meters), and send this information to the robot execution unit synchronously.

[0109] The robot execution unit is the execution layer of the attitude estimation system. Employing an underactuated cable parallel assembly robot, it is responsible for driving the building structure to be assembled to complete the assembly action based on the pose information output by the embedded AI computing unit. For example, in terms of mechanical structure, the robot includes four high-strength aramid cables, four servo drive units, and one end effector. The aramid cables have a diameter of 8 mm and a breaking tensile strength of no less than 50 kN, meeting the load-bearing requirements of modular prefabricated building modules (such as walls and floors). The servo drive units are equipped with absolute encoders, with a position control accuracy of no more than 0.1 mm, allowing for precise adjustment of the cable length. The end effector uses electromagnetic chucks or mechanical clamps with a load-bearing capacity of no less than 5 tons, adaptable to different types of building structures (prefabricated building modules), ensuring stable clamping of the building structure during assembly. One end of each of the four cables is connected to a servo drive unit, which is fixed to the top of the rectangular steel frame at the top of the assembly area. The other end of the cables is connected to the four lifting points of the end effector. By adjusting the length difference of the four cables, the three-dimensional translation (travel range of ±3 meters) and rotation (angle range of ±30°) of the building structure to be assembled can be achieved, meeting the posture adjustment requirements in the modular building assembly process.

[0110] In terms of control logic, the assembly robot is equipped with an independent motion controller that receives pose information from the embedded AI computing unit via the EtherCAT protocol. The motion controller internally stores the target pose of the building structure to be assembled. After receiving real-time pose information, it calculates the deviation between the current pose and the target pose (including translational deviations Δx, Δy, Δz and rotational deviations Δα, Δβ, Δγ). Based on these deviations, a PID control algorithm is used to generate cable length adjustment amounts. These adjustments are then converted into control signals for the servo drive device through inverse kinematics calculations, driving the servo device to extend and retract the cable, gradually bringing the building structure closer to the target pose.

[0111] In terms of safety mechanisms, the motion controller can have a built-in anomaly response module. When it receives a "pose difference exceeding limits" signal from the embedded AI computing unit (i.e., an abnormal change in the pose of the assembly connection point), the anomaly response module immediately triggers an emergency stop command. The servo drive device locks the cable within 10 milliseconds to prevent the building structure to be assembled from continuing to move. The end effector remains in a clamping state to prevent the module from falling off. At the same time, an audible and visual alarm is activated to issue a warning signal, reminding staff to investigate the cause of the anomaly and prevent collisions or docking failures between the building structures to be assembled.

[0112] The pose estimation system for modular integrated buildings provided in this application, through the coordinated operation of various components, completes the pose estimation and automatic assembly of modular buildings. The specific workflow is as follows: First, the visual perception camera, which may, but is not limited to, use a frame rate of 30Hz, to acquire RGB and depth images of the working area in real time, and transmits the image data synchronously to the embedded AI computing unit via a USB 3.2 interface. Then, after receiving the image data, the embedded AI computing unit first performs image size adjustment, normalization, and distortion correction through the image preprocessing module; then, it outputs representational features through the feature fusion module, and the occlusion judgment module determines whether the assembly connection point is occluded; if the assembly connection point is occluded, the pose estimation module matches the CAD model of the visible structural component to obtain the first pose information, and derives the second pose information of the connection point by combining it with a preset transformation matrix; if the assembly connection point is not occluded, it directly matches the CAD model of the assembly connection point to obtain the second pose information. Subsequently, the embedded AI computing unit sends the second pose information to the robot execution unit via the EtherCAT protocol. The robot motion controller calculates the deviation between the current pose and the target pose, generates a cable length adjustment amount, and drives the servo device to adjust the position and orientation of the building structure to be assembled. Furthermore, the pose estimation system repeatedly executes the above three steps with a cycle time of 33 milliseconds until the deviation between the assembly connection point and the target docking position meets the requirements (e.g., translational deviation not exceeding 0.5 mm, rotational deviation not exceeding 0.1°). At this point, the building structure to be assembled completes precise docking, and the assembly action ends.

[0113] In summary, the pose estimation system, through the collaboration of visual perception, intelligent computing, and underactuated robots, achieves accurate estimation and dynamic control of the pose of connection points during the assembly of modular integrated buildings. It is particularly suitable for complex scenarios where assembly connection points are easily occluded, and can significantly improve the efficiency and accuracy of automated assembly.

[0114] Corresponding to the pose estimation method for modular integrated buildings in the above embodiments, Figure 6 This is a schematic diagram of a pose estimation device for modular integrated buildings provided in an embodiment of this application. The device can be implemented as part or all of a computer device, which can be software, hardware, or a combination of both. Figure 7 The electronic device shown.

[0115] Reference Figure 6 The pose estimation device 600 for modular integrated buildings includes: The acquisition unit 601 is used to acquire RGB images and depth images of the building structure to be assembled in the working area captured by the visual perception camera. The building structure to be assembled includes visible structural components and assembly connection points set on the visible structural components.

[0116] The fusion unit 602 is used to fuse RGB images and depth images using a neural network structure with a cross-attention mechanism to obtain representational features for describing environmental changes in the working area.

[0117] The matching unit 603 is used to match the depth image with the CAD model of the visible structural component using a pose estimation algorithm when the assembly connection point is determined to be occluded based on the representation features, so as to obtain the first pose information of the visible structural component relative to the camera coordinate system of the visual perception camera.

[0118] The determining unit 604 is used to determine the second pose information of the occluded assembly connection point relative to the camera coordinate system based on the pre-calibrated transformation matrix and the first pose information. The transformation matrix is ​​used to characterize the fixed pose of the assembly connection point in the local coordinate system of the visible structural component.

[0119] In one possible implementation, the first pose information includes a rotation matrix and a translation vector, and the defining unit includes: The calling subunit is used to call a transformation matrix that has been pre-calibrated based on the rigid connection relationship between the visible structural component and the assembly connection point. The transformation matrix contains the rotation and translation components of the assembly connection point in the local coordinate system of the visible structural component. The local coordinate system is a coordinate system established with the visible structural component as the reference. The operation subunit is used to perform matrix multiplication of the rotation matrix and rotation components to obtain the rotation parameters of the assembly connection point relative to the camera coordinate system; to perform product operation of the rotation matrix and translation components to obtain the equivalent translation amount of the assembly connection point from the local coordinate system of the visible structural component to the camera coordinate system; and to perform vector addition operation of the equivalent translation amount and translation vector to obtain the translation parameters of the assembly connection point relative to the camera coordinate system. The sub-unit is determined based on rotation and translation parameters to determine the second pose information of the occluded assembly connection point relative to the camera coordinate system.

[0120] In one possible implementation, the fusion unit includes: The preprocessing subunit is used to preprocess the RGB image and the depth image. The preprocessing is used to adjust the RGB image and the depth image to a preset size and normalize them respectively. The first extraction subunit is used to extract features from the preprocessed RGB image and depth image respectively through two parallel convolutional neural network branches in the neural network structure of the cross-attention mechanism, so as to obtain RGB feature vectors and depth feature vectors of the same dimension. The computational subunit is used to input the RGB feature vector and the depth feature vector into the cross-attention module in the neural network structure of the cross-attention mechanism. Using the RGB feature vector as the query vector and the depth feature vector as the key vector and value vector, it calculates the attention weight of the RGB features on the depth features to obtain the attention-weighted depth features. Using the depth feature vector as the query vector and the RGB feature vector as the key vector and value vector, it calculates the attention weight of the depth features on the RGB features to obtain the attention-weighted RGB features. The feature fusion subunit is used to add the attention-weighted RGB features and the attention-weighted depth features element-wise, and then compress and fuse the feature dimensions through the convolutional layer in the neural network structure of the cross-attention mechanism to obtain the fused feature vector. The output sub-unit is used to take the one-dimensional feature vector output by the pooling layer in the neural network structure with cross-attention mechanism, which performs global average pooling on the fused feature vector, as a representation feature to describe the environmental changes in the working area.

[0121] In one possible implementation, the matching unit includes: The second extraction subunit is used to extract the spatial distance information corresponding to the visible structural components based on the depth image; The generation sub-unit is used to extract geometric features from the CAD model of the visible structural components and generate CAD model data that matches the scale of the spatial distance information. The geometric features include edge contours, surface normals and key vertex coordinates. Sub-units are established to establish an initial correspondence between spatial distance information and CAD model data using a coarse matching algorithm based on geometric features, thereby obtaining the initial pose of the visible structural components relative to the camera coordinate system. The iterative subunit is used to iteratively optimize the rotation matrix and translation vector based on the initial pose by minimizing the Euclidean distance error between the sampling points in the spatial distance information and the corresponding points in the CAD model data, until the error converges to a preset accuracy threshold; the converged rotation matrix and translation vector are used as the first pose information of the visible structural component relative to the camera coordinate system.

[0122] In one possible implementation, the device further includes: The input subunit is used to input the representation features into a pre-defined fully connected layer neural network structure, which contains at least two hidden layers and one output layer. The output subunit is used to perform feature compression and classification mapping on the representation features through the neural network structure of the fully connected layer, and outputs the occlusion probability value of the assembly connection point. The determination subunit is used to determine that the assembly connection point is blocked if the occlusion probability value is greater than or equal to the occlusion determination threshold.

[0123] It is understood that the embodiments of the pose estimation device for modular integrated buildings and any implementation thereof correspond to the embodiments of the pose estimation method for modular integrated buildings and any implementation thereof. The technical effects corresponding to the embodiments of the pose estimation device for modular integrated buildings and any implementation thereof can be found in the above-mentioned embodiments of the pose estimation method for modular integrated buildings and any implementation thereof, and will not be repeated here.

[0124] It should be noted that the pose estimation device for modular integrated buildings provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0125] The functional units and modules in the above embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of this application.

[0126] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0127] This application also provides an electronic device, which includes one or more processors and a memory; The memory is coupled to one or more processors. The memory is used to store computer program code, which includes computer instructions. The one or more processors invoke the computer instructions to cause the electronic device to execute the pose estimation method for modular integrated buildings described above.

[0128] Figure 7This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 700 can be a mobile phone, smart screen, tablet computer, wearable electronic device, in-vehicle electronic device, augmented reality (AR) device, virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), projector, or a communication device such as a server, storage device, or base station, or a smart car, etc. This application embodiment does not impose any limitations on the specific type of electronic device.

[0129] The memory 701 can be used to store computer software programs 702 and modules. The processor 703 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 701. The memory 701 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as audio data, telephone directory, etc.). In addition, the memory 701 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0130] The processor 703 may include one or more processors such as a central processing unit (CPU), an application processor (AP), and a baseband processor. The processor can serve as the nerve center and command center of the wireless router. The processor 703 can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. The memory 701 can be used to store executable program code, including instructions. The processor 703 executes various functional applications and data processing of the network device by running the instructions stored in the memory. The memory 701 may include a program storage area and a data storage area, such as storing data for audio signals to be played. For example, the memory may be Double Data Rate Synchronous Dynamic Random Access Memory (DDR) or Flash memory.

[0131] This application also provides a computer-readable storage medium storing computer instructions; when the computer-readable storage medium is used on an electronic device, the electronic device performs the aforementioned pose estimation method for modular integrated buildings.

[0132] The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or can include one or more data storage devices such as servers or data centers that can be integrated with media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media, or semiconductor media (e.g., solid-state disks (SSDs)).

[0133] This application also provides a computer program product containing computer instructions, which, when run on an electronic device, enables the electronic device to execute the aforementioned pose estimation method for modular integrated buildings.

[0134] The computer storage medium and computer program product provided in the embodiments of this application are used to execute the methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects corresponding to the methods provided above, and will not be repeated here.

[0135] In the above embodiments, implementation can also be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, Digital Subscriber Line, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc., and the storage medium can also include combinations of the above types of memory.

[0136] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0137] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments claimed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A pose estimation method for modular integrated buildings, characterized in that, include: Acquire RGB and depth images of the building structure to be assembled in the working area captured by a visual perception camera, wherein the building structure to be assembled includes visible structural components and assembly connection points are set on the visible structural components; A neural network structure utilizing a cross-attention mechanism fuses the RGB image and the depth image to obtain representational features for describing environmental changes in the working area; When it is determined that the assembly connection point is occluded based on the characterization features, a pose estimation algorithm is used to match the depth image with the CAD model of the visible structural component to obtain the first pose information of the visible structural component relative to the camera coordinate system of the visual perception camera. Based on the pre-calibrated transformation matrix and the first pose information, the second pose information of the occluded assembly connection point relative to the camera coordinate system is determined, wherein the transformation matrix is ​​used to characterize the fixed pose of the assembly connection point in the local coordinate system of the visible structural component.

2. The method according to claim 1, characterized in that, The first pose information includes a rotation matrix and a translation vector. The step of determining the second pose information of the occluded assembly connection point relative to the camera coordinate system based on the pre-calibrated transformation matrix and the first pose information includes: The transformation matrix, which is pre-calibrated based on the rigid connection relationship between the visible structural component and the assembly connection point, is invoked. The transformation matrix includes the rotation and translation components of the assembly connection point in the local coordinate system of the visible structural component. The local coordinate system is a coordinate system established with the visible structural component as the reference. Perform matrix multiplication on the rotation matrix and the rotation components to obtain the rotation parameters of the assembly connection point relative to the camera coordinate system; The rotation matrix and the translation component are multiplied to obtain the equivalent translation amount of the assembly connection point from the local coordinate system of the visible structural component to the camera coordinate system. The equivalent translation amount and the translation vector are added together to obtain the translation parameters of the assembly connection point relative to the camera coordinate system. Based on the rotation parameters and the translation parameters, the second pose information of the obscured assembly connection point relative to the camera coordinate system is determined.

3. The method according to claim 1, characterized in that, The method further includes: The second pose information of the assembly connection point relative to the camera coordinate system is continuously acquired to form a pose sequence that is continuous over time. The pose information in the pose sequence is smoothed using a moving average filter to obtain a smoothed pose sequence. Based on the smoothed pose sequence, the pose difference between adjacent time steps is calculated; Based on the relationship between the pose difference and the preset difference threshold, the assembly robot plans the assembly actions for the building structure to be assembled. The preset difference threshold is set according to the assembly safety requirements of the building structure to be assembled and the pose change range of the assembly connection point of the building structure to be assembled under normal movement.

4. The method according to claim 3, characterized in that, The step of planning the assembly robot's assembly actions for the building structure to be assembled based on the relationship between the pose difference and a preset difference threshold includes: If the pose difference exceeds the preset difference threshold, the assembly robot is scheduled to stop performing assembly actions on the building structure to be assembled, wherein the pose difference exceeding the preset difference threshold indicates that there is an abnormal jump in the smoothed pose sequence. If the pose difference does not exceed the preset difference threshold, the assembly robot is planned to continue performing assembly actions on the building structure to be assembled, wherein the pose difference not exceeding the preset difference threshold indicates that there are no abnormal jumps in the smoothed pose sequence.

5. The method according to claim 1, characterized in that, The neural network structure utilizing the cross-attention mechanism fuses the RGB image and the depth image to obtain representational features for describing environmental changes in the working area, including: The RGB image and the depth image are preprocessed, wherein the preprocessing is used to uniformly adjust the RGB image and the depth image to a preset size and normalize them respectively; The two parallel convolutional neural network branches in the neural network structure of the cross-attention mechanism extract the features of the preprocessed RGB image and depth image respectively, and obtain RGB feature vectors and depth feature vectors of the same dimension. The RGB feature vector and the depth feature vector are input into the cross-attention module in the neural network structure of the cross-attention mechanism. The RGB feature vector is used as the query vector and the depth feature vector is used as the key vector and value vector. The attention weight of the RGB feature to the depth feature is calculated to obtain the attention-weighted depth feature. The depth feature vector is used as the query vector and the RGB feature vector is used as the key vector and value vector. The attention weight of the depth feature to the RGB feature is calculated to obtain the attention-weighted RGB feature. The fused feature vector obtained by element-wise addition of the attention-weighted RGB features and the attention-weighted depth features is used as a representation feature to describe the environmental changes in the working area.

6. The method according to any one of claims 1 to 5, characterized in that, The step of matching the depth image with the CAD model of the visible structural component using a pose estimation algorithm to obtain the first pose information of the visible structural component relative to the camera coordinate system of the visual perception camera includes: Based on the depth image, extract the spatial distance information corresponding to the visible structural components; Geometric features are extracted from the CAD model of the visible structural components to generate CAD model data that matches the scale of the spatial distance information. The geometric features include edge contours, surface normals, and key vertex coordinates. An initial correspondence is established between the spatial distance information and the CAD model data using a coarse matching algorithm based on geometric features, thereby obtaining the initial pose of the visible structural component relative to the camera coordinate system; Based on the initial pose, the rotation matrix and translation vector are iteratively optimized by minimizing the Euclidean distance error between the sampling points in the spatial distance information and the corresponding points in the CAD model data until the error converges to a preset accuracy threshold. The converged rotation matrix and translation vector are used as the first pose information of the visible structure component relative to the camera coordinate system.

7. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The representation features are input into a pre-defined fully connected layer neural network structure, wherein the fully connected layer contains at least two hidden layers and one output layer; The neural network structure of the fully connected layer is used to perform feature compression and classification mapping on the representation features, and output the occlusion probability value of the assembly connection point; If the occlusion probability value is greater than or equal to the occlusion determination threshold, then the assembly connection point is determined to be occluded.

8. The method according to any one of claims 1 to 5, characterized in that, The method further includes: If the assembly connection point is determined to be unobstructed based on the characterization features, the pose estimation algorithm is used to match the depth image with the CAD model of the assembly connection point to obtain the second pose information of the assembly connection point relative to the camera coordinate system.

9. A pose estimation system for modular integrated buildings, characterized in that, include: A visual perception camera is used to acquire RGB and depth images of the building structure to be assembled in the work area, wherein the building structure to be assembled includes visible structural components and assembly connection points are set on the visible structural components. An embedded AI computing unit, connected to the visual perception camera, is used to run the pose estimation method as described in any one of claims 1 to 8 to obtain second pose information of the assembly connection point relative to the camera coordinate system of the visual perception camera; The robot execution unit is connected to the embedded AI computing unit and uses an underactuated cable parallel assembly robot to plan the assembly robot's assembly actions for the building structure to be assembled based on the pose information.

10. A pose estimation device for modular integrated buildings, characterized in that, include: The acquisition unit is used to acquire RGB images and depth images of the building structure to be assembled in the working area captured by the visual perception camera, wherein the building structure to be assembled includes visible structural components and assembly connection points are set on the visible structural components. The fusion unit is used to fuse the RGB image and the depth image using a neural network structure with a cross-attention mechanism to obtain representational features for describing environmental changes in the working area; The matching unit is used to match the depth image with the CAD model of the visible structural component using a pose estimation algorithm when the assembly connection point is determined to be occluded based on the characterization features, so as to obtain the first pose information of the visible structural component relative to the camera coordinate system of the visual perception camera. The determining unit is used to determine the second pose information of the occluded assembly connection point relative to the camera coordinate system based on the pre-calibrated transformation matrix and the first pose information, wherein the transformation matrix is ​​used to characterize the fixed pose of the assembly connection point in the local coordinate system of the visible structural component.

Images