Structural deformation monitoring method and system based on binocular vision and coordinate transformation

Abstract

The invention provides a structural deformation monitoring method and system based on binocular vision and coordinate transformation, and the method comprises the steps: carrying out the calibration of a binocular stereoscopic vision left camera and a binocular stereoscopic vision right camera based on a calibration plate, and obtaining a plurality of calibration plate images of the calibration plate under different postures; collecting a target structure image based on a binocular stereoscopic vision camera; calibration parameter calculation is carried out based on the plurality of calibration plate images, and a rigidity conversion matrix when the calibration plates coincide with the target structural surface is obtained; coordinate transformation is carried out on the measuring points of the target structure image based on the stiffness transformation matrix; and calculating in-plane displacement and out-of-plane displacement of the target structure based on the converted three-dimensional coordinates. According to the method, the 2D checkerboard target is utilized, the three-dimensional world coordinate system located on the left camera system is converted into the three-dimensional world coordinate system on the structural surface through coordinate conversion, the morphological characteristics and displacement information of the structural surface are more visually shown, the calculated amount is small, the precision is high, the distance measuring range is large, and the cost is low.

Description

Technical Field

[0001] The present invention belongs to the technical field of structural health detection, and relates to a structural detection method, and in particular to a structural deformation monitoring method and system based on binocular vision and coordinate conversion. Background Art

[0002] Structural deformation detection is an important part of structural health detection. The study and prediction of the mechanical behavior of the structure under load is achieved on the basis of accurate measurement of structural deformation.

[0003] At present, many mature methods have been developed for the detection of structural deformation. The commonly used methods in engineering are: displacement sensor, laser displacement sensor, GPS method, photogrammetry, etc.

[0004] Among the above methods, the displacement sensor method is a contact measurement method, which requires the sensor to be deployed on the surface or inside the object to be measured in advance. The preparation work is complicated, most of them require a long time for equipment installation and debugging, and are greatly affected by environmental factors.

[0005] Although the laser displacement sensor method is a non-contact measurement method, its measurement accuracy is greatly affected by the slope and reflectivity of the laser projection plane, and only one measuring point can be measured at a time, and only displacement information in one direction can be obtained.

[0006] As a non-contact measurement technology, GPS can obtain continuous and real-time three-dimensional coordinate and speed information, but it can only measure one measuring point during measurement, and there is also the problem of transition.

[0007] Photogrammetry is a non-contact optical measurement method developed since the 1960s. This method has been widely used in the engineering field because it can quickly and accurately obtain the full-field displacement and strain information of the measured object in a non-contact manner. Among the photogrammetry methods, binocular stereo vision measurement is a method based on the bionic principle of the human binocular vision system to obtain the three-dimensional geometric information of the object by analyzing parallax.

[0008] In binocular vision measurement, two cameras are generally used to simultaneously capture two digital images of the object being measured from their respective angles, and based on the principle of parallax, the three-dimensional geometric information of the object being measured can be restored to reconstruct the three-dimensional shape of the object being measured. With the development of computer vision technology, binocular stereo vision measurement plays an increasingly important role in modern industrial measurement due to its advantages of non-contact, high precision, simple operation, strong adaptability, and multi-point real-time measurement, and has a wide range of applicability.

[0009] However, in traditional binocular stereo vision measurement technology, since the measurement methods have certain restrictions on the system optical path settings or the optical path settings are complex, during the measurement process, especially when measuring the displacement of actual engineering structures, the measurement system has certain restrictions on the relative positions of the two cameras, which makes the measurement method limited in its measurement scope, and has low universality and flexibility; at the same time, multiple coordinate transformations and calibrations are required during measurement to obtain relevant parameters, which requires large amounts of calculations and low accuracy. Summary of the invention

[0010] In view of the shortcomings of the prior art mentioned above, the purpose of the present application is to provide a structural deformation monitoring method and system based on binocular vision and coordinate transformation, which is used to solve the problems in the prior art of implementing the structural deformation monitoring technology based on binocular vision and coordinate transformation, such as the limitation of the measurement scope due to the limitation of the system optical path setting or the complex optical path setting and the limitation of the relative position of the two cameras, and the low universality and flexibility; as well as the large amount of calculation and low accuracy during measurement.

[0011] To achieve the above-mentioned purpose and other related purposes, in a first aspect, the present application provides a structural deformation monitoring method based on binocular vision and coordinate transformation, comprising the following steps: calibrating a binocular stereo vision camera based on a calibration plate, and obtaining a calibration plate image of the calibration plate in different postures; the binocular stereo vision camera includes a left camera and a right camera; acquiring a target structure image based on the binocular stereo vision camera; calculating calibration parameters based on the calibration plate image, and obtaining a stiffness conversion matrix when the calibration plate coincides with the target structure surface; wherein the stiffness conversion matrix is ​​a conversion matrix that converts the left camera coordinate system to the target structure surface coordinate system; performing coordinate transformation on the measuring points of the target structure image based on the stiffness conversion matrix to obtain three-dimensional coordinates on the target structure surface coordinate system; and calculating the in-plane displacement and out-of-plane displacement of the target structure based on the three-dimensional coordinates on the target structure surface coordinate system.

[0012] In an implementation of the first aspect, a binocular stereo vision camera is calibrated based on a calibration plate, and obtaining a calibration plate image of the calibration plate in different postures includes the following steps: selecting a 2D checkerboard of a preset size as the calibration plate, synchronously acquiring multiple calibration plate images based on the left camera and the right camera for each posture, and performing image averaging on the multiple calibration plate images to obtain the calibration plate image in the posture.

[0013] In an implementation of the first aspect, capturing a target structure image based on a binocular stereo vision camera includes the following steps: based on the left camera and the right camera, synchronously capturing a plurality of target structure images and performing image averaging to obtain an averaged target structure image.

[0014] In an implementation of the first aspect, calibration parameters are calculated based on the calibration plate image, and the stiffness conversion matrix when the calibration plate coincides with the target structure surface is obtained, including the following steps: using a MATLAB camera calibration tool to obtain calibration parameters of the calibration plate image after image averaging; wherein the calibration parameters include: the effective focal lengths of the left camera and the right camera, and the stiffness conversion matrix between the left camera and the right camera coordinate systems.

[0015] In an implementation of the first aspect, performing coordinate transformation on the measuring points of the target structure image based on the stiffness transformation matrix includes the following steps: acquiring the three-dimensional coordinates of the measuring points of the target structure image in the left camera coordinate system; and transforming the three-dimensional coordinates into the target structure surface coordinate system based on the stiffness transformation matrix.

[0016] In an implementation of the first aspect, the three-dimensional coordinates of the measuring point of the target structure image in the left camera coordinate system are obtained according to the following formula:

[0017]

[0018] Among them, f l , f r Indicates the effective focal length of the left camera and the right camera; r i ,t i Represents the elements in the rotation and translation matrix between the left camera and the right camera coordinate system; X l ,Y l Indicates the physical coordinates of the measurement point in the image captured by the left camera; X r ,Y r Indicates the physical coordinates of the measurement point in the image captured by the right camera; x p ,y p ,z p Indicates the three-dimensional coordinates of the measuring point in the left camera coordinate system.

[0019] In an implementation of the first aspect, calculating the in-plane displacement and the out-of-plane displacement of the target structure based on the three-dimensional coordinates on the target structure plane coordinate system comprises the following steps:

[0020] according to calculating the in-plane displacement of the target structure;

[0021] According to d 离面i =Zi -Z0 calculates the out-of-plane displacement of the target structure;

[0022] Among them, (X0, Y0, Z0) represents the three-dimensional coordinates of the measuring point P at time t0; (X i ,Y i ,Z i ) represents t after structural deformation i The three-dimensional coordinates of the measuring point P at time.

[0023] In the second aspect, the present application provides a structural deformation monitoring system based on binocular vision and coordinate transformation, including: a system calibration module, used to calibrate a binocular stereo vision camera based on a calibration plate, and obtain a calibration plate image of the calibration plate in different postures; the binocular stereo vision camera includes a symmetrically arranged left camera and a right camera; an image acquisition module, used to acquire a target structure image based on the binocular stereo vision camera; a deformation monitoring module, used to calculate calibration parameters based on the calibration plate image, and obtain a stiffness conversion matrix when the calibration plate coincides with the target structure surface; wherein the stiffness conversion matrix is ​​a conversion matrix that converts the left camera coordinate system to the target structure surface coordinate system; based on the stiffness conversion matrix, the measuring points of the target structure image are coordinate transformed to obtain three-dimensional coordinates on the target structure surface coordinate system; based on the three-dimensional coordinates on the target structure surface coordinate system, the in-plane displacement and out-of-plane displacement of the target structure are calculated.

[0024] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the method for structural deformation monitoring based on binocular vision and coordinate transformation as described in any one of claims 1 to 8 is implemented.

[0025] In the last aspect, the present application provides a structural deformation monitoring device based on binocular vision and coordinate transformation, comprising: a processor and a memory. The memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the structural deformation monitoring device based on binocular vision and coordinate transformation executes the structural deformation monitoring method based on binocular vision and coordinate transformation.

[0026] As described above, the structural deformation monitoring method and system based on binocular vision and coordinate transformation of the present invention have the following beneficial effects:

[0027] (1) The structural deformation monitoring method based on binocular vision and coordinate transformation proposed in the present application only needs to calibrate the imaging system once for the binocular camera by adjusting the spatial posture of the 2D checkerboard calibration plate, so as to obtain all the parameters required for calculating the three-dimensional coordinates of the structure based on the world coordinate system on the surface of the structure to be measured; and the measurement method only needs to transform the three-dimensional world coordinate system located in the left camera system in the traditional binocular vision model on the plane of the structure to be measured through only one coordinate transformation, which greatly reduces the amount of calculation.

[0028] (2) The structural deformation monitoring method based on binocular vision and coordinate transformation of the present application establishes a world coordinate system on the plane of the structure to be measured and performs structural displacement measurement based on this, which can make the measurement results easier to understand and can more intuitively show the morphological characteristics and displacement information of the structure surface.

[0029] (3) The structural deformation monitoring method based on binocular vision and coordinate transformation of the present application is based on the binocular stereo vision of the PC platform. The structural deformation monitoring system based on binocular vision and coordinate transformation does not require the relative position of the two cameras, is more universal and flexible, and is more suitable for measuring the displacement of actual engineering structures. It can achieve higher measurement accuracy and has the advantages of good portability, long ranging range, no need for active lighting, and low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic flow chart of a method for monitoring structural deformation based on binocular vision and coordinate transformation in one embodiment of the present invention.

[0031] Figure 2 It is a schematic flow chart of S11 in the structural deformation monitoring method based on binocular vision and coordinate transformation of the present invention.

[0032] Figure 3 Shown is a schematic diagram of a binocular stereo vision measurement mathematical model of an arbitrary camera position relationship in the structural deformation monitoring method based on binocular vision and coordinate transformation of the present invention.

[0033] Figure 4 It is a schematic flow chart of S14 in the structural deformation monitoring method based on binocular vision and coordinate transformation of the present invention.

[0034] Figure 5 It shows a schematic diagram of the displacement of measuring points before and after structural deformation in the world coordinate system established on the structural surface in the structural deformation monitoring method based on binocular vision and coordinate transformation of the present invention.

[0035] Figure 6 It is a schematic diagram of the principle structure of a structural deformation monitoring system based on binocular vision and coordinate transformation in one embodiment of the present invention.

[0036] Figure 7 It is a schematic diagram of the principle structure of a structural deformation monitoring device based on binocular vision and coordinate transformation in one embodiment of the present invention.

[0037] Component number description

[0038] 61 System calibration module

[0039] 62 Image acquisition module

[0040] 63 Deformation monitoring module

[0041] 71 Processor

[0042] 72 Memory

[0043] Steps S11 to S15 DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0046] The structural deformation monitoring method based on binocular vision and coordinate transformation of the present invention, based on the traditional binocular stereoscopic vision three-dimensional coordinate measurement method, uses a 2D checkerboard target to transform the three-dimensional world coordinate system located in the left camera system to the three-dimensional world coordinate system on the structural surface through coordinate transformation, making the measurement results more intuitive and easy to understand, and more intuitively showing the morphological characteristics and displacement information of the structural surface.

[0047] The structural deformation monitoring method based on binocular vision and coordinate transformation provided in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.

[0048] See also Figure 1 , which is a flow chart of a structural deformation monitoring method based on binocular vision and coordinate transformation in one embodiment of the present invention. Figure 1As shown, this embodiment provides a structural deformation monitoring method based on binocular vision and coordinate transformation.

[0049] The structural deformation monitoring method based on binocular vision and coordinate transformation specifically comprises the following steps:

[0050] S11, calibrate the binocular stereo vision camera based on the calibration plate, and obtain the calibration plate images of the calibration plate in different postures. Figure 2 , which is a flow chart of S11 in the structural deformation monitoring method based on binocular vision and coordinate transformation of the present invention. Figure 2 As shown, the S11 comprises the following steps:

[0051] First, set up the binocular stereo vision camera. Adjust the camera position, focal length, aperture, etc. of the integrated PC-based binocular stereo vision structural deformation monitoring system to the best position where the binocular stereo vision camera can clearly capture the surface morphology of the structure, and keep the positions of the two cameras unchanged.

[0052] The binocular stereo vision camera includes a left camera and a right camera. The relative positions of the left camera and the right camera are not required. The clarity of the formed image is used as the system adjustment standard. The left and right cameras can be adjusted to the best position where they can clearly capture the surface morphology of the structure, and then the positions of the two cameras are kept unchanged.

[0053] S111, selecting a 2D chessboard of a preset size as a calibration plate, and synchronously acquiring a plurality of calibration plate images based on the left camera and the right camera for each posture.

[0054] The Zhang Zhengyou calibration method is used to calibrate the binocular camera of the binocular stereo vision imaging system.

[0055] According to the size of the shooting field of view, a 2D checkerboard target of preset size is selected as the calibration plate, and the calibration plate is adjusted to several spatial postures of different positions and directions near the surface of the structure to be measured. In the last posture, the calibration plate is attached to the surface of the structure to be measured so that it coincides with the target structure surface.

[0056] In this embodiment, a 2D chessboard of a preset size is selected as a calibration plate according to the size of the shooting field of view. The preset size needs to be set according to the target structure to be measured. Then, the calibration plate is placed near the target structure to be measured, and after setting different spatial postures, multiple calibration plate images are synchronously captured by the left camera and the right camera, and the calibration plate is attached to the surface of the target structure to be measured in the last posture so that it coincides with the surface of the target structure to be measured.

[0057] In this embodiment, it is preferred to take 4 images as a group of images. That is, for the target structure to be measured in the state to be collected, 4 images are taken by the left camera as a group; at the same time, 4 images are also taken by the right camera synchronously as a group of target structure images; and then the images are averaged; finally, an image pair of the target structure to be measured in the acquisition state is obtained.

[0058] Specifically, after the left camera and the right camera are set in a fixed position, a 2D chessboard that matches the size of the target structure to be measured is selected as the calibration plate. Then, near the target structure to be measured, the calibration plate is adjusted to a posture, and the left camera and the right camera are synchronously photographed to obtain the calibration plate image in the posture. Different postures are randomly adjusted in turn and then synchronously photographed by the left camera and the right camera to obtain multiple groups of calibration plate images. In this embodiment, 20 groups of calibration plate images in different postures are preferably collected, and finally 20 groups of calibration plate images taken by the left camera (a total of 80 calibration plate images) and 20 groups of calibration plate images taken by the right camera (a total of 80 calibration plate images) are obtained. And the posture of the 20th group of calibration plate images is adjusted to a direction and position that coincides with the surface of the target structure to be measured.

[0059] S112, performing image averaging on the multiple calibration plate images to obtain a calibration plate image in the posture.

[0060] In this embodiment, for each calibration plate posture, four image pairs are synchronously captured by the left camera and the right camera for image averaging and denoising, and finally an averaged calibration plate image pair is obtained. At this time, the calibration plate image pair is averaged to reduce the noise of the calibration plate image, making the obtained image clearer.

[0061] S12, collecting target structure images based on a binocular stereo vision camera.

[0062] Based on the left camera and the right camera, a plurality of target structure images are synchronously collected and image averaged to obtain an averaged target structure image.

[0063] In this embodiment, it is preferred to shoot 4 target structure images. The left camera and the right camera are used to synchronously shoot 4 image pairs of the target structure to be measured in the acquisition state, that is, 4 target structure images shot by the left camera and 4 target structure images shot by the right camera. Then, the 4 target structure images shot by the left camera and the 4 target structure images shot by the right camera are respectively averaged to obtain 1 averaged target structure image shot by the left camera and 1 target structure image shot by the right camera. The averaged target structure image is mainly used for the three-dimensional reconstruction calculation in the subsequent steps.

[0064] The left camera and the right camera may be CCD cameras, but are not limited to this type of camera.

[0065] S13, calculating calibration parameters based on the calibration plate image, and obtaining a stiffness conversion matrix when the calibration plate coincides with the target structure surface.

[0066] In this embodiment, the MATLAB camera calibration tool is used to obtain the calibration parameters of the calibration plate image after image averaging.

[0067] The averaged calibration plate image obtained in step S112 is imported into the MATLAB camera calibration toolbox to obtain the calibration parameters of the imaging system, wherein the calibration parameters include: the effective focal length of the left camera and the right camera, the stiffness conversion matrix between the left camera and the right camera coordinate system, etc.

[0068] See also Figure 3 , which is a schematic diagram of a binocular stereo vision measurement mathematical model of an arbitrary camera position relationship in the structural deformation monitoring method based on binocular vision and coordinate transformation of the present invention.

[0069] In this embodiment, the calibration plate images captured by the left camera and the right camera are averaged and imported into the MATLAB camera calibration toolbox to solve the calibration parameters of the imaging system, and the effective focal lengths of the left camera and the right camera and the elements in the rotation and translation matrix between the left camera and the right camera coordinate systems are obtained. At the same time, the stiffness conversion matrix [RT] between the left camera coordinate system (i.e., the original world coordinate system) and the new world coordinate system established on the structural surface is obtained under the last calibration plate posture.

[0070] Specifically, the MATLAB camera calibration toolbox is used to solve the parameters based on the calibration plate images collected by the left camera and the right camera to obtain the calibration parameters of the imaging system. l is the optical center position of the left camera, o r is the optical center position of the right camera, and point P is the position of the target structure to be measured. The left camera and the right camera simultaneously shoot images of the spatial measurement point P. The left camera coordinate system o l -x l y l z l Coincident with the world coordinate system o-xyz, the coordinates of the spatial measurement point P are (x p ,y p ,z p ), the physical coordinate system of the left camera image is O l -X l Y l , the effective focal length is f l ; The right camera coordinate system is o r -x ry r z r , the physical coordinate system of the right camera image is O r -X r Y r , the effective focal length is f r .

[0071] S14, performing coordinate transformation on the measurement points of the target structure image based on the stiffness transformation matrix to obtain the three-dimensional coordinates on the target structure surface coordinate system. Figure 4 , which is a flow chart of S14 in the structural deformation monitoring method based on binocular vision and coordinate transformation of the present invention. Figure 4 As shown, the S14 comprises the following steps:

[0072] S141, obtaining the three-dimensional coordinates of the measuring points of the target structure image in the left camera coordinate system.

[0073] In this embodiment, three-dimensional coordinate reconstruction is performed according to calibration parameters.

[0074] By projective transformation we can get:

[0075]

[0076] Among them, p l ,p r Respectively represent the image pixel coordinates of the target structure measurement point in the left camera and the right camera, M l ,M r Respectively represent the projection matrix of the left camera and the right camera, X w Indicates the three-dimensional coordinates of the target structure measurement point in the world coordinate system. l -x l y l z l Coordinate system and o r -x r y r z r The relationship between the coordinate systems can be expressed by the transformation matrix:

[0077]

[0078] According to the above calculation formula, the three-dimensional coordinate calculation formula of the measuring point P of the target structure to be measured can be obtained:

[0079]

[0080] Among them, f l , f r Indicates the effective focal length of the left camera and the right camera; r i ,t iRepresents the elements in the rotation and translation matrix between the left camera and the right camera coordinate system; X l ,Y l Indicates the physical coordinates of the measurement point in the image captured by the left camera; X r ,Y r Indicates the physical coordinates of the measurement point in the image captured by the right camera; x p ,y p ,z p Indicates the three-dimensional coordinates of the measuring point in the left camera coordinate system.

[0081] The three-dimensional coordinates in the left camera coordinate system (ie, the original world coordinate system o-xyz) can be obtained by the above three-dimensional coordinate calculation formula of the measuring point P of the target structure to be measured.

[0082] S142, transforming the three-dimensional coordinates into a target structural surface coordinate system based on the stiffness transformation matrix.

[0083] According to the calculation formula in step S141, the three-dimensional coordinates of the measuring point P in the left camera coordinate system can be obtained. Then, based on the stiffness transformation matrix [RT], the three-dimensional coordinates of the measuring point P are transformed into the coordinates of the target structure surface, as shown below:

[0084]

[0085] S15, calculating the in-plane displacement and out-of-plane displacement of the target structure based on the three-dimensional coordinates on the target structure plane coordinate system. Figure 5 , which is a schematic diagram of the displacement of measuring points before and after structural deformation in the world coordinate system established on the structural surface in the structural deformation monitoring method based on binocular vision and coordinate transformation of the present invention.

[0086] like Figure 5 As shown, by obtaining the three-dimensional coordinates of the measuring point P in the world coordinate system O-XYZ established on the structural surface, the in-plane displacement and out-of-plane displacement of the structural measuring point are calculated respectively.

[0087] In this embodiment, it is assumed that the three-dimensional coordinates of the measuring point P at time t0 are (X0, Y0, Z0). Due to structural deformation, t i The three-dimensional coordinates of the measuring point P at the time are (X i ,Y i ,Z i ),but:

[0088] (1) The calculation formula for the in-plane displacement of the target structure is:

[0089]

[0090] (2) The calculation formula for the off-plane displacement of the target structure is:

[0091] d 离面i =Z i -Z0

[0092] Among them, (X0, Y0, Z0) represents the three-dimensional coordinates of the measuring point P at time t0; (X i ,Y i ,Z i ) represents t after structural deformation i The three-dimensional coordinates of the measuring point P at time.

[0093] Binocular vision three-dimensional coordinate measurement is a method based on the principle of parallax. Two cameras simultaneously capture two digital images of the object to be measured from their respective angles, and the three-dimensional geometric information of the object is obtained by analyzing the parallax.

[0094] This application is based on the traditional binocular stereoscopic vision three-dimensional coordinate measurement method, using a 2D checkerboard target to transform the three-dimensional world coordinate system located in the left camera system to the three-dimensional world coordinate system on the structural surface through coordinate transformation. The measurement results are more intuitive and easy to understand, and the morphological characteristics and displacement information of the structural surface are more intuitively displayed. The main functions it realizes include: image acquisition, real-time display, storage, analysis and processing, and output of structural deformation results.

[0095] The structural deformation monitoring method based on binocular vision and coordinate transformation of the present application can break the restrictions of traditional measurement methods on the system optical path setting during the implementation of structural deformation monitoring technology, avoid the limitation of the measurement scope due to complex optical path setting and certain restrictions on the relative positions of the two cameras, and has universality and high flexibility; as well as the characteristics of small amount of calculation required for measurement and high precision.

[0096] The protection scope of the structural deformation monitoring method based on binocular vision and coordinate transformation described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0097] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the following is achieved: Figure 1 The structural deformation monitoring method based on binocular vision and coordinate transformation.

[0098] At any possible level of technical detail combination, the present application may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0099] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0100] The computer-readable program described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives a computer-readable program instruction from the network, and forwards the computer-readable program instruction for storage in the computer-readable storage medium in each computing / processing device. The computer program instruction for performing the operation of the present application can be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, an integrated circuit configuration data, or a source code or object code written in any combination of one or more programming languages, wherein the programming language includes an object-oriented programming language, such as Smalltalk, C++, etc., and a procedural programming language, such as "C" language or similar programming language. Computer readable program instructions can be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In certain embodiments, by using the state information of computer readable program instructions to personalize electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), the electronic circuits can execute computer readable program instructions, thereby realizing various aspects of the present application.

[0101] An embodiment of the present application also provides a structural deformation monitoring system based on binocular vision and coordinate transformation. The structural deformation monitoring system based on binocular vision and coordinate transformation can implement the structural deformation monitoring method based on binocular vision and coordinate transformation described in the present application. However, the implementation device of the structural deformation monitoring method based on binocular vision and coordinate transformation described in the present application includes but is not limited to the structure of the structural deformation monitoring system based on binocular vision and coordinate transformation listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0102] The structural deformation monitoring system based on binocular vision and coordinate transformation provided by this embodiment will be described in detail below with reference to the drawings.

[0103] This embodiment provides a structural deformation monitoring system based on binocular vision and coordinate transformation, including:

[0104] See also Figure 6 , which is a schematic diagram of the principle structure of a structural deformation monitoring system based on binocular vision and coordinate transformation in one embodiment of the present invention. Figure 6 As shown, the structural deformation monitoring system based on binocular vision and coordinate transformation includes: a system calibration module 61 , an image acquisition module 62 and a deformation monitoring module 63 .

[0105] The system calibration module 61 is used to calibrate the binocular stereo vision camera based on a calibration plate, and obtain the calibration plate image of the calibration plate in different postures; the binocular stereo vision camera includes a left camera and a right camera.

[0106] In this embodiment, a binocular stereo camera is first set up. The camera position, focal length, aperture, etc. of the integrated structural deformation monitoring system based on binocular vision and coordinate transformation are adjusted to the best position where the binocular stereo camera can clearly capture the surface morphology of the structure, and the positions of the two cameras are kept unchanged. Among them, there is no requirement for the relative position of the left camera and the right camera. The clarity of the image formed is used as the system adjustment standard. It can be adjusted to the best position where the left and right cameras can clearly capture the surface morphology of the structure, and then the positions of the two cameras are kept unchanged.

[0107] A 2D chessboard of a preset size is selected as a calibration plate, and a plurality of calibration plate images are synchronously collected based on the left camera and the right camera for each posture.

[0108] The Zhang Zhengyou calibration method is used to calibrate the binocular camera of the binocular stereo vision imaging system.

[0109] According to the size of the shooting field of view, a 2D checkerboard target of preset size is selected as the calibration plate, and the calibration plate is adjusted to several spatial postures of different positions and directions near the surface of the structure to be measured. In the last posture, the calibration plate is attached to the surface of the structure to be measured so that it coincides with the target structure surface.

[0110] In this embodiment, a 2D chessboard of a preset size is selected as a calibration plate according to the size of the shooting field of view. The preset size needs to be set according to the target structure to be measured and the field of view. Then, the calibration plate is placed near the target structure to be measured, and after setting different spatial postures, multiple calibration plate images are synchronously captured by the left camera and the right camera, and the calibration plate is attached to the surface of the target structure to be measured in the last posture so that it coincides with the surface of the target structure to be measured.

[0111] The plurality of calibration plate images are averaged to obtain a calibration plate image in the posture.

[0112] In this embodiment, for each calibration plate posture, four image pairs are synchronously captured by the left camera and the right camera for image averaging and denoising, and finally an averaged calibration plate image pair is obtained. At this time, the calibration plate image pair is averaged to reduce the noise of the calibration plate image, making the obtained image clearer.

[0113] The image acquisition module 62 is connected to the system calibration module 61 and is used to acquire target structure images based on a binocular stereo vision camera.

[0114] In this embodiment, based on the left camera and the right camera, a plurality of target structure images are synchronously captured and averaged to obtain an averaged target structure image.

[0115] Specifically, it is preferred to shoot 4 target structure images. The left camera and the right camera are used to synchronously shoot 4 image pairs of the target structure to be measured in the acquisition state, that is, 4 target structure images shot by the left camera and 4 target structure images shot by the right camera. Then, the 4 target structure images shot by the left camera and the 4 target structure images shot by the right camera are respectively averaged to obtain 1 averaged target structure image shot by the left camera and 1 averaged target structure image shot by the right camera. The averaged target structure image is mainly used for 3D reconstruction calculation in subsequent steps.

[0116] The left camera and the right camera may be CCD cameras, but are not limited to this type of camera. The hardware integration used includes visual sensors, optical lenses, gigabit network cards, etc.

[0117] The deformation monitoring module 63 is used to calculate calibration parameters based on the calibration plate image, and obtain the stiffness conversion matrix when the calibration plate coincides with the target structure surface; wherein the stiffness conversion matrix is ​​a conversion matrix that converts the left camera coordinate system to the target structure coordinate system; based on the stiffness conversion matrix, coordinates of the measuring points of the target structure image are converted to obtain three-dimensional coordinates on the target structure surface coordinate system; based on the three-dimensional coordinates on the target structure surface coordinate system, the in-plane displacement and out-of-plane displacement of the target structure are calculated.

[0118] In this embodiment, multiple calibration plate images taken by the left camera are averaged, and multiple calibration plate images taken by the right camera are also averaged to obtain averaged calibration plate images taken by the left camera and the right camera, respectively.

[0119] Specifically, the calibration plate images captured by the left and right cameras are averaged and imported into the MATLAB camera calibration toolbox to solve the calibration parameters of the imaging system, and the effective focal lengths of the left and right cameras and the elements in the rotation and translation matrix between the left and right camera coordinate systems are obtained. At the same time, the stiffness transformation matrix [RT] between the left camera coordinate system (i.e., the original world coordinate system) and the new world coordinate system established on the structural surface is obtained under the last calibration plate posture.

[0120] Then, coordinate transformation is performed on the measuring points of the target structure image based on the stiffness transformation matrix.

[0121] In this embodiment, the three-dimensional coordinate calculation formula of the target structure measurement point to be measured can be obtained through the relevant calculation formula, that is, the three-dimensional coordinates of the target structure measurement point to be measured in the left camera coordinate system (ie, the original world coordinate system) can be obtained.

[0122] The calculated three-dimensional coordinates are converted into a target structural surface coordinate system based on the stiffness conversion matrix.

[0123] Then, based on the three-dimensional coordinates, the in-plane displacement and out-of-plane displacement of the target structure are calculated.

[0124] Binocular vision three-dimensional coordinate measurement is a method based on the principle of parallax. Two cameras simultaneously capture two digital images of the object to be measured from their respective angles, and the three-dimensional geometric information of the object is obtained by analyzing the parallax.

[0125] This application is based on the traditional binocular stereoscopic three-dimensional coordinate measurement method, using a 2D checkerboard target to transform the three-dimensional world coordinate system located in the left camera system to the three-dimensional world coordinate system on the structural surface through coordinate transformation, making the measurement results more intuitive and easy to understand, and more intuitively showing the morphological characteristics and displacement information of the structural surface.

[0126] The structural deformation monitoring system based on binocular vision and coordinate transformation provided by the present application can break the limitations of traditional measurement methods on the system optical path setting in the process of implementing the structural deformation monitoring technology, avoid the limitation of the measurement scope caused by the complex optical path setting and certain restrictions on the relative positions of the two cameras, and has universality and high flexibility; as well as the characteristics of small amount of calculation required for measurement and high precision.

[0127] It should be noted that it should be understood that the division of the various modules of the above system is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the x module can be a separately established processing element, or it can be integrated in a certain chip of the above system for implementation. In addition, it can also be stored in the memory of the above system in the form of program code, and called and executed by a certain processing element of the above system. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0128] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0129] See also Figure 7 , which is a schematic diagram of the principle structure of a structural deformation monitoring device based on binocular vision and coordinate transformation in one embodiment of the present invention. Figure 7 As shown, this embodiment provides a structural deformation monitoring device based on binocular vision and coordinate transformation, and the structural deformation monitoring device based on binocular vision and coordinate transformation includes: a processor 71 and a memory 72; the memory 72 is used to store computer programs; the processor 71 is connected to the memory 72, and is used to execute the computer program stored in the memory 72, so that the structural deformation monitoring device based on binocular vision and coordinate transformation performs the various steps of the structural deformation monitoring method based on binocular vision and coordinate transformation as described above.

[0130] Preferably, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0131] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0132] In summary, the structural deformation monitoring method and system based on binocular vision and coordinate transformation provided by the present application have the following beneficial effects:

[0133] This application proposes a structural deformation monitoring method based on the traditional binocular stereo vision measurement method. This method is a new structural deformation monitoring technology based on binocular stereo vision and coordinate transformation. This technology uses a 2D checkerboard target to transform the three-dimensional world coordinate system located in the left camera system in the traditional binocular vision model into a three-dimensional world coordinate system on the structural surface through coordinate transformation, so that the measurement results are more intuitive and easy to understand, and the morphological characteristics and displacement information of the structural surface are more intuitively displayed. The integrated binocular stereo vision structural deformation monitoring system based on the PC platform can achieve high measurement accuracy, and has the advantages of good portability, long ranging range, no need for active lighting, and low cost. The system does not require the relative position of the two cameras, and is more universal and flexible. It is suitable for non-contact dynamic real-time measurement of displacement of actual engineering structures.

[0134] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A structural deformation monitoring method based on binocular vision and coordinate transformation, characterized in that: The following steps are involved: Calibrate the binocular stereo vision camera based on a calibration plate, and obtain the calibration plate image of the calibration plate in different postures; the binocular stereo vision camera includes a left camera and a right camera; Capture target structure images based on binocular stereo vision cameras; Calibration parameters are calculated based on the calibration plate image, and a stiffness conversion matrix is ​​obtained when the calibration plate coincides with the target structure surface; wherein the stiffness conversion matrix is ​​a conversion matrix for converting the left camera coordinate system to the target structure surface coordinate system; Performing coordinate transformation on the measuring points of the target structure image based on the stiffness transformation matrix to obtain three-dimensional coordinates on the target structure surface coordinate system; Based on the three-dimensional coordinates of the target structure on the plane coordinate system, the in-plane displacement and the out-of-plane displacement of the target structure are calculated.

2. The structural deformation monitoring method based on binocular vision and coordinate transformation according to claim 1 is characterized in that: Calibrate the binocular stereo vision camera based on the calibration plate, and obtain the calibration plate image of the calibration plate in different postures, including the following steps: A 2D chessboard of a preset size is selected as a calibration plate, and for each posture, a plurality of calibration plate images are synchronously captured based on the left camera and the right camera, and the plurality of calibration plate images are averaged to obtain the calibration plate image under the posture.

3. The structural deformation monitoring method based on binocular vision and coordinate transformation according to claim 1 is characterized in that: The following steps are included in acquiring the target structure image based on the binocular stereo vision camera: Based on the left camera and the right camera, a plurality of target structure images are synchronously collected and image averaged to obtain an averaged target structure image.

4. The structural deformation monitoring method based on binocular vision and coordinate transformation according to claim 1 is characterized in that: Calculating calibration parameters based on the calibration plate image and obtaining a stiffness conversion matrix when the calibration plate coincides with the target structure surface comprises the following steps: The MATLAB camera calibration tool is used to obtain calibration parameters of the calibration plate image after image averaging; wherein the calibration parameters include: the effective focal lengths of the left camera and the right camera, and the stiffness conversion matrix between the left camera and the right camera coordinate systems.

5. The structural deformation monitoring method based on binocular vision and coordinate transformation according to claim 1 is characterized in that: The coordinate transformation of the measuring points of the target structure image based on the stiffness transformation matrix comprises the following steps: Obtaining the three-dimensional coordinates of the measuring points of the target structure image in the left camera coordinate system; The three-dimensional coordinates are converted into a target structure surface coordinate system based on the stiffness conversion matrix.

6. The structural deformation monitoring method based on binocular vision and coordinate transformation according to claim 5 is characterized in that: The three-dimensional coordinates of the measuring point of the target structure image in the left camera coordinate system are obtained according to the following formula: Among them, f l , f r Indicates the effective focal length of the left camera and the right camera; r i ,t i Represents the elements in the rotation and translation matrix between the left camera and the right camera coordinate system; X l ,Y l Indicates the physical coordinates of the measurement point in the image taken by the left camera; X r ,Y r Indicates the physical coordinates of the measurement point in the image captured by the right camera; x p ,y p ,z p Indicates the three-dimensional coordinates of the measuring point in the left camera coordinate system.

7. The structural deformation monitoring method based on binocular vision and coordinate transformation according to claim 1 is characterized in that: Calculating the in-plane displacement and out-of-plane displacement of the target structure based on the three-dimensional coordinates on the target structure plane coordinate system comprises the following steps: according to calculating the in-plane displacement of the target structure; According to d 离面i =Z i -Z0 calculates the out-of-plane displacement of the target structure; Among them, (X0, Y0, Z0) represents the three-dimensional coordinates of the measuring point P at time t0; (X i ,Y i ,Z i ) represents t after structural deformation i The three-dimensional coordinates of the measuring point P at time.

8. A structural deformation monitoring system based on binocular vision and coordinate transformation, characterized in that: include: A system calibration module, used to calibrate the binocular stereo vision camera based on a calibration plate, and obtain calibration plate images of the calibration plate in different postures; The binocular stereo vision camera includes a left camera and a right camera; An image acquisition module, used for acquiring target structure images based on a binocular stereo vision camera; A deformation monitoring module, used to calculate calibration parameters based on the calibration plate image, and obtain a stiffness conversion matrix when the calibration plate coincides with the target structure surface; wherein the stiffness conversion matrix is ​​a conversion matrix for converting the left camera coordinate system to the target structure surface coordinate system; based on the stiffness conversion matrix, coordinate conversion is performed on the measuring points of the target structure image to obtain three-dimensional coordinates on the target structure surface coordinate system; Based on the three-dimensional coordinates of the target structure on the plane coordinate system, the in-plane displacement and the out-of-plane displacement of the target structure are calculated.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for monitoring structural deformation based on binocular vision and coordinate transformation as described in any one of claims 1 to 7 is implemented.

10. A structural deformation monitoring device based on binocular vision and coordinate transformation, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is connected to the memory and is used to execute the computer program stored in the memory so that the structural deformation monitoring device based on binocular vision and coordinate transformation executes the structural deformation monitoring method based on binocular vision and coordinate transformation according to any one of claims 1 to 7.

Images