A three-dimensional model construction method and device based on image sequence processing

Abstract

The application relates to the field of three-dimensional model reconstruction, and provides a three-dimensional model construction method and device based on image sequence processing, which comprises the following steps: acquiring via-hole operation image sequence information; performing target detection and tracking processing according to the via-hole operation image sequence information to obtain identification point information and identity information of key parts, wherein the identity information comprises a bridge machine component to which the identification point belongs; processing the identification point information and the identity information of the key parts by using a multi-view stereo vision algorithm to obtain a three-dimensional coordinate set of the identification point at each moment in the via-hole operation; and constructing a time sequence point cloud model according to the three-dimensional coordinate set of the identification point at each moment in the via-hole operation to obtain a sparse three-dimensional point cloud model, which is used for monitoring the via-hole operation by being combined with a theoretical via-hole three-dimensional model of the bridge machine. The application significantly enhances the robustness and precision of the via-hole operation model reconstruction in a complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional model reconstruction, in particular to a three-dimensional model construction method and device based on image sequence processing. BACKGROUND

[0002] In the field of bridge construction, the safety and precision of the bridge-erecting machine over-hole operation are directly related to the overall quality and progress of the project. With the development of computer vision and three-dimensional reconstruction technology, using image data for dynamic monitoring of large structures has become an important research direction. However, for complex non-standard equipment such as bridge-erecting machines, how to reconstruct a three-dimensional model that accurately reflects the overall deformation from the operation image sequence in real time and robustly still faces significant challenges. The existing technology usually uses discrete point measurement methods such as total station and laser tracker. This method is difficult to achieve continuous dynamic tracking throughout the process, and is easily affected by on-site environmental obstructions and interference, making it difficult to meet the urgent needs of real-time, global, and quantitative safety monitoring for the over-hole operation, which is a high-risk operation link. SUMMARY

[0003] The purpose of the present application is to provide a three-dimensional model construction method and device based on image sequence processing to improve the above problems.

[0004] To achieve the above purpose, the embodiments of the present application provide the following technical solutions:

[0005] On the one hand, the embodiments of the present application provide a three-dimensional model construction method based on image sequence processing, which comprises:

[0006] Obtaining over-hole operation image sequence information, the over-hole operation image sequence information being collected by multiple different cameras;

[0007] Performing target detection and tracking processing according to the over-hole operation image sequence information to obtain identification point information and identity information of key parts, the identity information including the bridge-erecting machine components to which the identification points belong;

[0008] Processing the identification point information and the identity information of the key parts using a multi-view stereo vision algorithm to obtain a three-dimensional coordinate set of the identification points at each moment in the over-hole operation;

[0009] Constructing a time-series point cloud model according to the three-dimensional coordinate set of the identification points at each moment in the over-hole operation to obtain a sparse three-dimensional point cloud model, the sparse three-dimensional point cloud model being used to monitor the over-hole operation with a preset bridge-erecting machine theoretical over-hole three-dimensional model.

[0010] On the other hand, the embodiments of the present application provide a three-dimensional model construction device based on image sequence processing, which comprises:

[0011] An acquisition module is configured to acquire via-hole operation image sequence information, which is collected by multiple different cameras;

[0012] A first processing module is configured to perform target detection and tracking processing according to the via-hole operation image sequence information, to obtain key part identification point information and identity information, wherein the identity information includes a bridge machine component to which the identification point belongs;

[0013] A second processing module is configured to process the key part identification point information and the identity information by using a multi-view stereo vision algorithm, to obtain a three-dimensional coordinate set of identification points at each moment in the via-hole operation;

[0014] A construction module is configured to perform time-series point cloud model construction according to the three-dimensional coordinate set of identification points at each moment in the via-hole operation, to obtain a sparse three-dimensional point cloud model, which is used to monitor the via-hole operation in combination with a preset bridge machine theoretical via-hole three-dimensional model.

[0015] In a third aspect, an embodiment of the present application provides a three-dimensional model construction device based on image sequence processing, which comprises a memory and a processor. The memory is configured to store a computer program, and the processor is configured to execute the computer program to implement the steps of the three-dimensional model construction method based on image sequence processing.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the three-dimensional model construction method based on image sequence processing are implemented.

[0017] The present application has the following beneficial effects:

[0018] The present application acquires a bridge machine theoretical via-hole three-dimensional model and image sequences collected synchronously from multiple perspectives, performs target detection and tracking based on component-level identity information of identification points, reconstructs a time-series sparse three-dimensional point cloud model in combination with a multi-view stereo vision algorithm, and finally realizes real-time monitoring of the via-hole operation by comparing with an actual theoretical model. The method realizes full-automatic reconstruction from image sequences to component-level three-dimensional models, overcomes the limitations that traditional measurement methods cannot be continuously tracked and are easily disturbed by occlusion, effectively improves the integrity and reliability of pose data, significantly enhances the robustness and precision of model reconstruction in complex environments by introducing identity identification and rigid body motion constraints, and provides a global, quantitative and visual safety monitoring method for the dynamic operation process of a bridge machine.

[0019] Other features and advantages of the present application will be set forth in the following description, and in part will be apparent from the description, or can be learned by practice of the application. The objects and other advantages of the present application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0021] Figure 1 The flow chart of the three-dimensional model construction method based on image sequence processing described in the embodiments of the present application.

[0022] Figure 2 The structural schematic diagram of the three-dimensional model construction device based on image sequence processing described in the embodiments of the present application.

[0023] In the figure, the annotations are: 800, three-dimensional model construction device based on image sequence processing; 801, processor; 802, memory; 803, multimedia component; 804, I / O interface; 805, communication component. DETAILED DESCRIPTION

[0024] In order to make the objects, technical solutions and advantages of the embodiments of the present application more clear, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0025] It should be noted that: similar labels and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present application, the terms "first", "second" and the like are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0026] Embodiment 1:

[0027] The embodiment provides a three-dimensional model construction method based on image sequence processing. It can be understood that in the embodiment, a scene can be laid out, for example, in bridge construction, when a bridge erecting machine carries out a hole passing operation, a plurality of cameras synchronously collect operation image sequences, a sparse point cloud model is constructed, and a scene of comparing with a theoretical model to monitor operation safety and precision in real time.

[0028] Referring to Figure 1 , the method comprises steps S1-S4.

[0029] Step S1, obtain hole passing operation image sequence information, wherein the hole passing operation image sequence information is collected by a plurality of different cameras;

[0030] Step S2, perform target detection and tracking processing according to the hole passing operation image sequence information, to obtain identification point information and identity information of a key part, wherein the identity information comprises a bridge erecting machine component to which the identification point belongs.

[0031] It can be understood that in the hole passing process, illumination change, mechanical vibration and partial occlusion are common problems, which make it particularly difficult to stably and continuously capture the motion trajectory of the key point. In this step, identifiable identification points with unique identities are pre-set at key parts, so that the overall posture monitoring problem is converted into an accurate tracking problem of these stable feature points. This method reduces the dependence on environmental texture, and by introducing identity information, lays a foundation for subsequent attribution of discrete observation data to specific structural components, and ensures the clarity of data semantics.

[0032] The step S2 further comprises steps S21-S23, which specifically comprise:

[0033] Step S21, perform semantic segmentation processing on each frame of image in the hole passing operation image sequence information, to obtain a mask region corresponding to the identification point.

[0034] In this step, semantic segmentation is realized by a deep learning model with an encoder-decoder structure, wherein the encoder is responsible for extracting multi-scale features of the image, and the decoder gradually recovers spatial details through upsampling and jump connection operations, and finally outputs a class label for each pixel. A large number of images of the bridge erecting machine operation scene need to be collected in advance during training, and the identification points therein need to be finely labeled at the pixel level. The deep learning model with the encoder-decoder structure selects a U-Net model. Segmentation in this way can ensure that the target region can be stably extracted in a high-dynamic, low-signal-to-noise industrial site.

[0035] Step S22, process the mask region corresponding to the identification point by using a centroid extraction algorithm, to obtain coordinate information of the identification point.

[0036] The step is not simply taking the geometric center of the mask, but using a centroid extraction algorithm to calculate the gray-weighted center coordinates of all pixels in the mask area, thereby improving the positioning accuracy from the pixel level to the sub-pixel level. In the monitoring of the bridge machine, one pixel in the image may correspond to a size of several centimeters in the real object. The accuracy improvement brought by sub-pixel positioning is of decisive significance for detecting slight deformation and displacement, and is the basis for realizing high-precision three-dimensional reconstruction.

[0037] Step S23, processing according to the coordinate information of the identification point and the preset encoding information to obtain identity information.

[0038] This step assigns a unique identity information to each detected identification point, thereby determining the component to which the identification point belongs. The specific identification process is: based on the coordinate information of the identification point, a small image block containing the identification point is intercepted from the image, and then based on the encoding rule of the identification point type, the unique ID of the identification point is decoded to determine the component to which the identification point belongs, and the unique ID is bound with the coordinate information. It should be noted that the encoding rule and decoding algorithm are not limited in the present application.

[0039] After the step S23, there are also steps S24-S27, which specifically include:

[0040] Step S24, using Kalman filter algorithm to process the coordinate information and identity information of all identification points in the current frame to obtain the state prediction value and prediction uncertainty of the trajectory of all identification points in the current frame.

[0041] In this step, the Kalman filter is used to predict the position and motion state of each identification point with an established trajectory in the next frame. In the specific implementation, a state vector and a covariance matrix describing the uncertainty of the state need to be maintained for each trajectory. At the beginning of processing each frame, the state of the last frame is used to apply a preset motion model to make a prediction and update the state vector and the covariance matrix. In the case of the bridge machine via, which has a relatively smooth and predictable motion, the prediction ability of the Kalman filter can effectively reduce the search area of the next frame, and provide a probability measurement based on motion consistency for subsequent data association, thereby maintaining the continuity of the trajectory when facing temporary occlusion or detection loss. It should be noted that the preset motion model of the bridge machine via is a uniform acceleration motion model. Compared with the simple uniform speed model, the uniform acceleration model can better fit the actual motion state and provide more accurate prediction.

[0042] Step S25, performing data association cost calculation processing according to the state prediction value and prediction uncertainty of the trajectory of all identification points in the current frame and the actual detection result of the current frame to obtain an association cost matrix.

[0043] The purpose of this step is to calculate the association cost between all detected landmarks in the current frame and the predicted states of all existing trajectories. Therefore, a cost matrix is constructed, where each element represents the association cost of a landmark with a trajectory prediction. The association cost is calculated by Mahalanobis distance, which takes into account the uncertainty of the prediction. In the direction with large uncertainty (such as the motion direction), a larger position deviation is allowed, while in the direction with small uncertainty (such as the vertical motion direction), a stricter requirement is made. It should be noted that the specific process of Mahalanobis distance calculation includes:

[0044] ;

[0045] In the above formula, represents the Mahalanobis distance; represents the observation vector, i.e. the actual detected landmark coordinates in the current frame; represents the state prediction vector, i.e. the state prediction value of Kalman filter; represents the observation matrix; represents the covariance matrix, which integrates the prediction uncertainty and the observation uncertainty.

[0046] Step S26, performing global optimal association matching processing based on the Hungarian algorithm according to the association cost matrix, to obtain the optimal association matching scheme between the current frame detection result and the existing trajectory;

[0047] In this step, the Hungarian algorithm finds a set of independent zero elements (i.e. the matching pair with the lowest cost) in the matrix by adding and subtracting the rows and columns of the matrix, so that the total cost of this set of matches is minimized. The association cost matrix is used as input to process using the Hungarian algorithm. The algorithm outputs a matching list, which explicitly indicates which landmark should be assigned to which trajectory, and also identifies unmatched landmarks (which may be new landmarks) and unmatched trajectories (which may be temporarily lost landmarks). When the motion paths of multiple landmarks of the bridge machine may intersect or be in close proximity, the Hungarian algorithm can avoid the piecewise ID switching error caused by the greedy nearest neighbor matching, and ensure the overall optimality of the current frame matching scheme from a global perspective, which is the key to ensuring the long-term consistency of the trajectory ID.

[0048] Step S27, updating the tracking state according to the optimal association matching scheme and the actual detection result of the current frame to obtain the continuous trajectory set of the landmark.

[0049] It can be understood that for the successfully matched track, the update step of the Kalman filter is used to correct the predicted state vector and covariance matrix by taking the detection value of the current frame as the observation value, so as to make the track estimation more accurate. For the unmatched detection, it is initialized as a new track. For the continuously unmatched track, a loss counter is started, and when the number of lost frames exceeds a preset threshold, the track is terminated. It should be noted that in this step, the unique ID of the marker point is used as a strong constraint to check the matching result. For example, if a marker point with ID A01 is assigned to a track originally with ID B02 according to the matching result of motion association, the system will reject this matching based on the stronger ID information and try to find the correct association or mark it as temporarily lost, thereby fundamentally eliminating the ID switching problem caused by similar appearance and motion. Thus, stable and reliable identity-track data can be output in a complex industrial environment.

[0050] In step S3, the marker point information of the key parts and the identity information are processed by using a multi-view stereo vision algorithm to obtain a three-dimensional coordinate set of the marker point at each moment in the via hole operation.

[0051] This step aims to solve the conversion problem from two-dimensional image information to three-dimensional space posture. The two-dimensional image of a single perspective loses depth information, and the small deformation of a large structure needs to be accurately measured in three-dimensional space. Therefore, this step uses a multi-camera system calibrated in advance to synchronously capture the same marker point from different perspectives, and restores its three-dimensional coordinates through the principle of multi-view geometry. This process essentially improves the three-dimensional space from the two-dimensional image plane through spatial intersection, thereby obtaining quantitative data that can truly reflect the orientation of each component of the bridge machine in space, and providing indispensable spatial dimension information for overall deformation analysis.

[0052] In the step S3, steps S31-S34 are further included, which specifically include:

[0053] In step S31, the marker point information and the identity information of the key parts corresponding to the via hole operation images collected by multiple cameras at the same moment are data associated to obtain a set of multi-perspective observation data pairs.

[0054] The purpose of this step is to realize the accurate association of the observation data of the same physical marker point under the same time and different camera perspectives. Each camera outputs a detection result list, and each element in the table contains the pixel coordinates and identity ID of a marker point. The detection results in the output list of each camera are merged directly by taking the identity ID as the primary key. All detection results with the same ID (i.e., the same physical point is seen from images from different cameras) are automatically grouped into a group to form an observation data pair. This method completely avoids the instability and ambiguity of feature descriptor matching in traditional multi-view stereo vision, and provides a pure and semantically clear input for subsequent high-precision three-dimensional reconstruction.

[0055] Step S32, performing mismatch detection according to the multi-view observation data pair set and the internal and external parameter matrices of each camera to obtain a screened multi-view observation data pair set;

[0056] Since individual cameras may cause ID misidentification or coordinate extraction errors due to severe lens obstruction or strong light reflection, this step uses the epipolar geometry constraint in multi-view geometry for verification and screening. The epipolar geometry describes the mathematical relationship that the imaging points of the same three-dimensional point under different perspectives must satisfy: the image point under one perspective must be located on a specific epipolar line in the imaging under another perspective. In implementation, a robust estimation algorithm is usually used to calculate the fundamental matrix, and the re-projection error of each observation point is calculated. Observation points with errors significantly greater than the threshold (e.g., deviating from the epipolar line by more than 2 pixels) are considered as outliers and are removed from the observation group.

[0057] Step S33, processing the screened multi-view observation data pair set using a forward intersection algorithm to obtain processed three-dimensional space coordinate information, wherein the processed three-dimensional space coordinate information includes an optimal estimated three-dimensional space coordinate of each marker point;

[0058] For each marker point, the internal and external parameter matrices of multiple cameras and the accurate pixel coordinates of the point on the image plane of each camera are known. Each observation ray can be represented as a three-dimensional space straight line starting from the camera optical center and passing through the image point. In theory, these rays should intersect at a point (i.e., the true three-dimensional position of the marker point). However, due to observation noise, they do not perfectly intersect. Therefore, the algorithm solves a least squares problem to find a point in space that minimizes the sum of the perpendicular distances from all rays, effectively suppressing single-view observation noise, and ultimately obtaining the optimal estimated three-dimensional space coordinate.

[0059] Step S34, performing time sequence coordinate smoothing based on rigid body motion constraints according to the processed three-dimensional space coordinate information at each time of the via hole operation to obtain a three-dimensional coordinate set of the marker point at each time of the via hole operation.

[0060] Since the single-frame triangulation result will have random errors, the coordinate sequence will appear jitter in time sequence. Therefore, this step utilizes the strong constraint that the bridge erecting machine can be approximately regarded as a rigid body in a short time, models the motion trajectory of a single marker point as a smooth time sequence, and adopts a Kalman filter to smooth the three-dimensional coordinates in time sequence, thereby effectively filtering out high-frequency random noise, so that the output coordinate sequence can not only maintain rapid response to the real motion, but also has high smoothness and stability.

[0061] Step S4, constructing a time sequence point cloud model according to the three-dimensional coordinate set of each marker point at each moment in the via hole operation, obtaining a sparse three-dimensional point cloud model, and the sparse three-dimensional point cloud model is used to monitor the via hole operation with a preset theoretical via hole three-dimensional model of the bridge erecting machine.

[0062] Since the via hole operation of the bridge erecting machine is a complex process involving the coordinated motion of multiple bodies, it is impossible to accurately describe its dynamic behavior by simply relying on a static model or discrete points. Therefore, the present application starts from the essence of the machine, and by fusing the multi-body dynamics structural parameters and the specific via hole operation process steps, a continuous theoretical motion three-dimensional model is constructed, which can accurately reflect the whole process of the bridge erecting machine from the start to the just-in-place. This model not only defines the ideal geometric shape, but more importantly, simulates the standard operation trajectory, thereby providing a dynamic and reliable reference frame for subsequent comparison with real-time data, and solving the problem of disconnection between the benchmark and the dynamic process in the traditional method.

[0063] In the step S4, steps S41-S44 are further included, which specifically include:

[0064] Step S41, obtaining the multi-body dynamics structural parameters of the bridge erecting machine and the via hole operation steps;

[0065] In this step, the multi-body dynamics structural parameters need to be accurately extracted from the equipment design drawings, mainly including the geometric dimensions, mass distribution of each moving part (such as the main beam, the front support leg, and the rear support leg), and the type (is it a translational joint or a rotational joint) and position (such as the hinge point coordinates of the rear support leg) of the joints connecting each part. The via hole operation steps need to be decomposed according to the construction process procedures, and the continuous via hole operation is analyzed into a series of discrete and ordered process states, including but not limited to the main beam being locked on the rear support leg, the main beam being stretched forward by X meters, the front support leg being lowered to the pier, etc. The key of this step is to convert the concrete mechanical structure and the abstract operation process into a standardized parameter list that can be processed by a mathematical model, avoiding the distortion of the model caused by unclear or missing parameter definition.

[0066] Step S42, kinematically modeling the multi-body dynamics structural parameters of the bridge erecting machine and the via hole operation steps by using the D-H parameter method, to obtain the forward kinematics equation of the via hole operation of the bridge erecting machine.

[0067] D-H parameter method is a standard method for establishing an attachment coordinate system for each link in the kinematic chain, and describing the transformation relationship between adjacent coordinate systems with four standard parameters (link length, link rotation angle, joint distance, joint rotation angle). In this step, the technician needs to establish a coordinate system for each link along the motion transmission path of the bridge erecting machine (for example, from the rear leg base-main beam-front leg). Then, according to the multi-body dynamics structure parameters, the four D-H parameters connecting adjacent links are determined. Finally, by multiplying the homogeneous transformation matrix between each adjacent link, the forward kinematics equation describing the function relationship between the end pose of the bridge erecting machine (the bottom of the front leg) and each joint variable is constructed. It should be noted that the specific form of the forward kinematics equation is:

[0068] ;

[0069] In the above formula, represents the forward kinematics equation; represents the homogeneous transformation matrix from link n-1 to link n.

[0070] Step S43, processing according to the forward kinematics equation and the design parameters of the bridge line to obtain a time-joint space discrete pose sequence;

[0071] In this step, the design parameters of the bridge line are first obtained, including the coordinates of the pier center and the line curve equation, so as to determine the starting point and ending point of the hole passing path and the target landing point coordinates of the front leg; taking the total hole passing time as a period, it is discretized into a plurality of time points, and then the target motion path of the front leg is taken as a constraint condition, and the numerical inverse algorithm is used to inversely solve the numerical value of each joint variable (such as the extension amount of the main beam and the rotation angle of the leg) in the forward kinematics equation at each discrete time point. In this way, the continuous hole passing action can be converted into a trajectory in the joint space composed of a series of discrete points.

[0072] Step S44, constructing the bridge erecting machine theoretical hole passing three-dimensional model according to the time-joint space discrete pose sequence.

[0073] Since the hole passing of the bridge erecting machine is a complex process involving the coordinated motion of multiple bodies, it is impossible to accurately describe its dynamic behavior simply by relying on static models or discrete points. Therefore, the present application starts from the mechanical essence, and by fusing the multi-body dynamics structure parameters and the specific hole passing operation process, a continuous theoretical motion three-dimensional model is constructed, which can accurately reflect the whole process of the bridge erecting machine from the start to the just-in-place. This model not only defines the ideal geometric shape, but more importantly, it simulates the standard operation trajectory, thereby providing a dynamic and reliable reference system for the subsequent comparison with real-time data, and solving the problem of disconnection between the benchmark and the dynamic process in the traditional method.

[0074] The step S4 further comprises steps S45-S48, which specifically include:

[0075] In step S45, the three-dimensional coordinate set of each identification point at each moment in the via hole operation and its identity information are subjected to structured processing to obtain a point cloud after structured processing.

[0076] In this step, each identification point is traversed, and the identity information corresponding to the identification point includes a component label to which the identification point belongs. According to the label, a hash table is used to automatically classify the point cloud originally mixed together into different component subsets. In a specific embodiment, points with all labels of “MainBeam” are classified into a main beam point cloud, and points with a label of “FrontLeg” are classified into a front leg point cloud. This way uses preset semantic information to promote low-level geometric data to a component data set with clear engineering significance, which makes it possible to perform independent deformation analysis on each component subsequently, which is a prerequisite for component-level posture monitoring.

[0077] In step S46, a local coordinate system based on component geometric features is established and coordinate transformation processing is performed according to the point cloud after structured processing, to obtain a feature point cloud of each component based on the local coordinate system.

[0078] In this step, a local coordinate system that best reflects the deformation characteristics of each component is established. For each component point cloud subset, a principal component analysis algorithm is used to determine the direction of the local coordinate system. For example, for a main beam, the coordinates of the main beam point cloud are input into the PCA algorithm, which calculates three principal component directions: the first principal component direction is the longest direction of the point cloud distribution, which usually corresponds to the axial direction of the main beam; the second principal component direction corresponds to the lateral direction; and the third principal component direction corresponds to the vertical direction. Taking the three principal component directions as the axes and the center of the point cloud as the origin, the local coordinate system of the main beam can be established. Then, through coordinate transformation, the coordinates of all points of the component are converted from the world coordinate system to its own local coordinate system. In this way, the posture analysis of the component can be separated from the complex world coordinate system. In the local coordinate system of the main beam, the vertical offset directly reflects the deflection, and the lateral offset directly reflects the lateral bending, greatly simplifying the calculation and understanding of deformation.

[0079] In step S47, fitting processing is performed according to the feature point cloud based on the local coordinate system of the component to obtain fitting geometric feature parameters of each main component at each moment.

[0080] In this step, for the main beam, a curve is fitted using polynomial regression to obtain the mathematical equation of the centerline, considering the bending deformation under the weight or load; for the front leg, a straight line is fitted to represent the centerline. The fitting process minimizes the sum of the squares of the perpendicular distances from all the identified points to the fitted line. At the same time, the algorithm calculates the residual of each point to the fitted line, which reflects the degree of dispersion of the point cloud relative to the ideal shape. This step abstracts the component shape described by multiple points into a centerline with a clear mathematical expression, i.e., geometric feature parameters. This line becomes the representation of the component pose, allowing the measurement of overall displacement, bending, and other deformations of the component to be converted into the calculation of the parameters of this line, greatly reducing the data dimension.

[0081] Step S48, generating a sparse three-dimensional point cloud model according to the fitted geometric feature parameters of each main component at each time.

[0082] The fitted geometric feature parameters of each component at each time and the corresponding time stamp are packaged and stored. Finally, these parameter packages arranged in chronological order constitute a sparse three-dimensional point cloud model sequence of the entire via process. This sequence does not store all three-dimensional points of each frame, but stores a lightweight description composed of several key parameters.

[0083] The directly obtained three-dimensional point set is sparse and disorganized, making it difficult to intuitively reflect overall deformations such as main beam bending and leg inclination. The present application organizes the point cloud through the obtained identity information, i.e., the attribution of the identified points to the components, and fits characteristic elements such as centerlines that can represent the overall shape of each component based on the geometric characteristics of each component, such as the linear feature of the main beam. In this way, the measurement of multiple points is converted into the monitoring of the poses of several key characteristic lines, greatly improving the efficiency of data processing and the intuitiveness of deformation trend description.

[0084] Based on the three-dimensional model-based, comprehensive pose difference analysis, the present application can reveal potential risks such as front leg positioning deviation and main beam abnormal deflection in real time. Thus, the traditional passive, experience-based safety monitoring is transformed into active, data-driven safety warning, thereby achieving effective safety protection for the high-risk via operation link.

[0085] Embodiment 2:

[0086] The embodiment provides a three-dimensional model construction device based on image sequence processing, which comprises an acquisition module, a first processing module, a second processing module, and a construction module, and specifically comprises:

[0087] The acquisition module is used to acquire via operation image sequence information, and the via operation image sequence information is collected by multiple different cameras.

[0088] The first processing module is configured to perform target detection and tracking processing according to the via operation image sequence information, to obtain identification point information and identity information of a key part, and the identity information includes a bridge machine component to which the identification point belongs;

[0089] The second processing module is configured to process the identification point information and the identity information of the key part by using a multi-view stereo vision algorithm, to obtain a three-dimensional coordinate set of the identification point at each moment in the via operation;

[0090] The construction module is configured to perform time sequence point cloud model construction according to the three-dimensional coordinate set of the identification point at each moment in the via operation, to obtain a sparse three-dimensional point cloud model, and the sparse three-dimensional point cloud model is used to monitor the via operation together with a preset bridge machine theoretical via three-dimensional model.

[0091] In one specific embodiment of the present disclosure, the construction module includes an acquisition unit, a first processing unit, a second processing unit and a third processing unit, which specifically include:

[0092] The acquisition unit is configured to acquire multi-body dynamics structural parameters of the bridge machine and via operation steps;

[0093] The first processing unit is configured to perform kinematics modeling on the multi-body dynamics structural parameters of the bridge machine and the via operation steps by using a D-H parameter method, to obtain a forward kinematics equation of the bridge machine via operation;

[0094] The second processing unit is configured to process according to the forward kinematics equation and design parameters of a bridge line, to obtain a time-joint space discrete pose sequence;

[0095] The third processing unit is configured to construct the bridge machine theoretical via three-dimensional model according to the time-joint space discrete pose sequence.

[0096] In one specific embodiment of the present disclosure, the first processing module includes a fourth processing unit, a fifth processing unit and a sixth processing unit, which specifically include:

[0097] The fourth processing unit is configured to perform semantic segmentation processing on each frame of image in the via operation image sequence information, to obtain a mask region corresponding to the identification point;

[0098] The fifth processing unit is configured to process the mask region corresponding to the identification point by using a centroid extraction algorithm, to obtain coordinate information of the identification point;

[0099] The sixth processing unit is configured to process the coordinate information of the identification point and preset encoding information, to obtain identity information.

[0100] In one specific implementation of the present disclosure, the sixth processing unit is followed by a seventh processing unit, an eighth processing unit, a ninth processing unit, and a tenth processing unit, which specifically include:

[0101] The seventh processing unit is configured to process the coordinate information and identity information of all identified points in the current frame by using a Kalman filtering algorithm to obtain state prediction values and prediction uncertainties of all identified point trajectories in the current frame.

[0102] The eighth processing unit is configured to perform data association cost calculation processing on the state prediction values and prediction uncertainties of all identified point trajectories in the current frame and the actual detection results of the current frame to obtain an association cost matrix.

[0103] The ninth processing unit is configured to perform global optimal association matching processing based on the Hungarian algorithm according to the association cost matrix to obtain an optimal association matching scheme between the detection results of the current frame and the existing trajectories.

[0104] The tenth processing unit is configured to update the tracking state according to the optimal association matching scheme and the actual detection results of the current frame to obtain a continuous trajectory set of the identified points.

[0105] It should be noted that, as for the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment related to the method, and will not be described in detail here.

[0106] Embodiment 3:

[0107] Corresponding to the above method embodiment, the present embodiment also provides a three-dimensional model construction device based on image sequence processing. The three-dimensional model construction device based on image sequence processing described below can be mutually corresponding to the three-dimensional model construction method based on image sequence processing described above.

[0108] Figure 2 is a block diagram of a three-dimensional model construction device 800 based on image sequence processing according to an example embodiment. As shown in Figure 2 the three-dimensional model construction device 800 based on image sequence processing can include a processor 801 and a memory 802. The three-dimensional model construction device 800 based on image sequence processing can also include one or more of a multimedia component 803, an I / O interface 804, and a communication component 805.

[0109] The processor 801 is configured to control overall operation of the image sequence processing based three-dimensional model construction device 800 to complete all or part of the steps of the image sequence processing based three-dimensional model construction method described above. The memory 802 is configured to store various types of data to support operation of the image sequence processing based three-dimensional model construction device 800, which can include, for example, instructions for any application or method operating on the image sequence processing based three-dimensional model construction device 800, and application-related data, such as contact data, sent and received messages, pictures, audio, video, and the like. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The multimedia component 803 can include a screen and an audio component. The screen can be, for example, a touch screen, and the audio component is configured to output and / or input audio signals. For example, the audio component can include a microphone configured to receive external audio signals. The received audio signals can be further stored in the memory 802 or transmitted through the communication component 805. The audio component also includes at least one speaker configured to output audio signals. The I / O interface 804 provides an interface between the processor 801 and other interface modules, which can be a keyboard, a mouse, a button, and the like. The buttons can be virtual buttons or physical buttons. The communication component 805 is configured to enable wired or wireless communication between the image sequence processing based three-dimensional model construction device 800 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, near field communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 805 can include a Wi-Fi module, a Bluetooth module, an NFC module.

[0110] In an example embodiment, the three-dimensional model construction device 800 based on image sequence processing can be implemented by one or more Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor or other electronic elements for executing the above-mentioned three-dimensional model construction method based on image sequence processing.

[0111] In another example embodiment, a computer readable storage medium including program instructions is also provided, which when executed by a processor, implements the steps of the above-mentioned three-dimensional model construction method based on image sequence processing. For example, the computer readable storage medium can be the above-mentioned memory 802 including program instructions, which can be executed by the processor 801 of the three-dimensional model construction device 800 based on image sequence processing to complete the above-mentioned three-dimensional model construction method based on image sequence processing.

[0112] Embodiment 4:

[0113] Corresponding to the above method embodiments, the present embodiment also provides a readable storage medium, and the readable storage medium described below can be referred to in correspondence with the above-mentioned three-dimensional model construction method based on image sequence processing.

[0114] A readable storage medium, the readable storage medium has a computer program stored thereon, and the computer program, when executed by a processor, implements the steps of the above-mentioned three-dimensional model construction method based on image sequence processing of the method embodiments.

[0115] The readable storage medium can be specifically a U disk, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various readable storage media that can store program codes.

[0116] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0117] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A three-dimensional model construction method based on image sequence processing, characterized by, include: Acquire via operation image sequence information, which is captured by multiple different cameras; Based on the image sequence information of the through-hole operation, target detection and tracking processing are performed to obtain the identification point information and identity information of key parts. The identity information includes the bridge erecting machine component to which the identification point belongs. The identification information and identity information of the key parts are processed using a multi-view stereo vision algorithm to obtain the three-dimensional coordinate set of the identification points at each moment during the through-hole operation. A time-series point cloud model is constructed based on the set of three-dimensional coordinates of the marker points at each moment during the hole-passing operation, resulting in a sparse three-dimensional point cloud model. The sparse three-dimensional point cloud model is used to compare with the preset theoretical three-dimensional hole-passing model of the bridge erecting machine to monitor the hole-passing operation. The method for constructing the theoretical three-dimensional model of the bridge erecting machine includes: Obtain the multibody dynamics structural parameters and the span-crossing operation steps of the bridge erecting machine; The kinematic modeling of the multibody dynamics structural parameters and the span-crossing operation steps of the bridge erecting machine was carried out using the DH parameter method, and the forward kinematic equations of the span-crossing operation of the bridge erecting machine were obtained. Based on the aforementioned forward kinematics equations and the design parameters of the bridge track, a time-joint spatial discrete pose sequence is obtained; The theoretical three-dimensional model of the bridge erecting machine through the hole is constructed based on the time-joint spatial discrete pose sequence.

2. The three-dimensional model construction method based on image sequence processing according to claim 1, characterized in that, Target detection and tracking processing is performed based on the via operation image sequence information, including: Semantic segmentation is performed on each frame of the through-hole operation image sequence information to obtain the mask region corresponding to the marker point; The centroid extraction algorithm is used to process the mask region corresponding to the marker point to obtain the coordinate information of the marker point; The identity information is obtained by processing the coordinate information of the marker point with the preset encoding information.

3. The method according to claim 2, wherein, After processing based on the coordinate information of the marker point and the preset encoding information, the process includes: The Kalman filter algorithm is used to process the coordinate and identity information of all marker points in the current frame to obtain the state prediction value and prediction uncertainty of the trajectory of all marker points in the current frame. Based on the state prediction values ​​and prediction uncertainties of all the marker point trajectories in the current frame and the actual detection results of the current frame, data association cost calculation is performed to obtain the association cost matrix; Based on the association cost matrix, a global optimal association matching process based on the Hungarian algorithm is performed to obtain the optimal association matching scheme between the current frame detection result and the existing trajectory. The tracking state is updated based on the optimal association matching scheme and the actual detection results of the current frame to obtain a continuous trajectory set of the marker points.

4. The three-dimensional model construction method based on image sequence processing according to claim 1, characterized in that, A temporal point cloud model is constructed based on the set of three-dimensional coordinates of the marker points at each time point during the via operation, including: The structured point cloud is obtained by performing structured processing on the set of three-dimensional coordinates and their identity information of the marker points at each moment during the via operation. Based on the structured point cloud, a local coordinate system based on the geometric features of the component is established and coordinate transformation is performed to obtain the feature point cloud based on the local coordinate system of the component at each time step. According to the feature point cloud based on the component local coordinate system, fitting processing is performed to obtain fitting geometric feature parameters of each main component at each moment; According to the fitting geometric feature parameters of each main component at each moment, a sparse three-dimensional point cloud model is generated.

5. The three-dimensional model construction method based on image sequence processing according to claim 1, characterized in that, The forward kinematics equation of the bridge-erecting machine hole passing operation specifically includes: ； In the above formula, denotes the forward kinematics equation; denotes the homogeneous transformation matrix from link n-1 to link n.

6. A three-dimensional model construction apparatus based on image sequence processing, characterized by comprising: It includes: An acquisition module is configured to acquire hole passing operation image sequence information collected by a plurality of different cameras; A first processing module is configured to perform target detection and tracking processing according to the hole passing operation image sequence information to obtain identification point information and identity information of key parts, the identity information including bridge-erecting machine components to which the identification points belong; A second processing module is configured to process the identification point information and the identity information of the key parts by using a multi-view stereo vision algorithm to obtain a three-dimensional coordinate set of identification points at each moment in the hole passing operation; A construction module is configured to construct a time sequence point cloud model according to the three-dimensional coordinate set of identification points at each moment in the hole passing operation to obtain a sparse three-dimensional point cloud model, which is used to compare with a preset bridge-erecting machine theoretical hole passing three-dimensional model to monitor the hole passing operation; The construction module includes: An acquisition unit is configured to acquire multi-body dynamics structure parameters of the bridge-erecting machine and hole passing operation steps; A first processing unit is configured to perform kinematics modeling on the multi-body dynamics structure parameters of the bridge-erecting machine and the hole passing operation steps by using a D-H parameter method to obtain a forward kinematics equation of the bridge-erecting machine hole passing operation; A second processing unit is configured to process according to the forward kinematics equation and design parameters of the bridge line to obtain a time-joint space discrete pose sequence; A third processing unit is configured to construct the bridge-erecting machine theoretical hole passing three-dimensional model according to the time-joint space discrete pose sequence.

7. The apparatus according to claim 6, wherein The first processing module includes: A fourth processing unit is configured to perform semantic segmentation processing on each frame of image in the hole passing operation image sequence information to obtain a mask region corresponding to the identification point; A fifth processing unit is configured to process the mask region corresponding to the identification point by using a centroid extraction algorithm to obtain coordinate information of the identification point; A sixth processing unit is configured to process the identity information according to the coordinate information of the identification point and preset encoding information.

8. The apparatus according to claim 7, wherein After the sixth processing unit, it includes: A seventh processing unit is configured to process the coordinate information and identity information of all identification points in the current frame by using a Kalman filtering algorithm to obtain a state prediction value and a prediction uncertainty of a trajectory of all identification points in the current frame; An eighth processing unit is configured to perform data association cost calculation processing on the state prediction value and the prediction uncertainty of the trajectory of all identification points in the current frame and the actual detection result of the current frame to obtain an association cost matrix; A ninth processing unit is configured to perform global optimal association matching processing based on a Hungarian algorithm according to the association cost matrix to obtain an optimal association matching scheme between the current frame detection result and the existing trajectory; A tenth processing unit is configured to update the tracking state according to the optimal association matching scheme and the actual detection result of the current frame to obtain a continuous trajectory set of the identification point.

Images