Human bridge mutual feeding displacement monitoring and early warning system and method based on visual technology

Abstract

The application provides a pedestrian bridge mutual feedback displacement monitoring and early warning system and method based on visual technology, mainly comprising a pedestrian monitoring module, a structure monitoring module, a pedestrian-structure mutual feedback system module and a structure early warning module. The application can realize synchronous monitoring of pedestrian trajectories and structure displacement, can analyze the human-induced load condition causing vibration problems according to the reconstruction of foot bottom force based on the pedestrian trajectories, and can also realize early warning when the structure vibration acceleration reaches a threshold value.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of civil construction, and particularly relates to a visual technology-based human bridge mutual feedback displacement monitoring and early warning system and method. BACKGROUND

[0002] The natural frequency of large-span light flexible structures is low, and the damping is small. The structural safety and vibration comfort under human-induced load are important control indicators in structural design and use. Ensuring and improving structural safety and vibration comfort are the due of creating green and low-carbon buildings. It is of great significance to ensure the service safety of structures and improve the service performance of structures to realize the synchronous monitoring of structural responses and human-induced loads, and on this basis, to realize the real-time evaluation of the service state of structures, and to intervene and control when necessary.

[0003] Current pedestrian motion monitoring is mainly used in the fields of traffic engineering and security, and through the arrangement of cameras at public places (such as stations, crossroads, squares, and entrances of sports venues), real-time trajectory monitoring and posture recognition of pedestrians are realized to achieve personnel identity verification or traffic diversion based on trajectory monitoring results to improve personnel passing efficiency. The current structural health monitoring system only focuses on the monitoring of structural state (strain, displacement, acceleration, etc.), and mainly uses traditional contact sensors for measurement, which has the limitations of requiring pre-installation and pre-embedding of sensors, high cost, and insufficient durability.

[0004] Since structural vibration is the result of personnel motion (load action), there is a one-to-one correspondence between the two, and if real-time monitoring of the structure state is to be achieved, both load and structural response must be monitored simultaneously, and the existing structural monitoring or pedestrian monitoring technology obviously does not take into account both aspects.

[0005] In recent years, with the popularization of cameras and the development of image processing technology, computer vision methods have become more and more common in the field of civil engineering. For vibration-sensitive structures such as pedestrian bridges, stadium stands, etc. with low natural frequency and damping, the displacement response of key parts of the structure can be obtained through visual methods, and then the acceleration information can be obtained. This method improves on the original contact and fixed sensors, realizing the leap from contact measurement to non-contact measurement.

[0006] In summary, the related methods have been developed by visual technology around structure monitoring and pedestrian monitoring, but only focus on a certain aspect of structure or pedestrian, and do not combine the two for analysis, which makes it impossible to immediately obtain the position and motion characteristics of the pedestrian that causes the response when the structure response amplitude exceeds the warning value. Therefore, how to realize the position recognition and motion characteristic extraction of the pedestrian in a complex environment, establish the mapping relationship between the pedestrian motion and the structure response, and realize the more targeted structure warning, has become a problem to be solved by the person skilled in the art. SUMMARY

[0007] In order to solve the above technical problems, the present application provides a pedestrian-bridge mutual feedback displacement monitoring and warning system and method based on visual technology, which can realize the synchronous monitoring of pedestrian trajectory and structure displacement, and analyze the pedestrian-induced load condition according to the foot force reconstructed from the pedestrian trajectory, and finally give a warning when the structure vibration acceleration reaches the threshold value.

[0008] According to one aspect of the present application, the present application provides a pedestrian-bridge mutual feedback displacement monitoring and warning system based on visual technology, comprising: a pedestrian monitoring module, a structure monitoring module, a pedestrian-structure mutual feedback system module and a structure warning system module;

[0009] The pedestrian monitoring module is used to obtain the body weight information of the pedestrian based on a deep learning target monitoring algorithm, measure the three-dimensional displacement of the pedestrian by using binocular stereo vision, derive the vertical displacement to obtain the acceleration information of the pedestrian, and reconstruct the foot force according to the body weight information and the acceleration information of the pedestrian.

[0010] The structure monitoring module is used to capture the landmark points of the bridge based on a Hough transform detection algorithm, measure the structure displacement at the landmark points by using binocular stereo vision, and derive the structure displacement to obtain the structure acceleration of the bridge.

[0011] The pedestrian-structure mutual feedback system module is used to perform Fourier transform on the foot force time history curve of the pedestrian and the bridge structure displacement time history curve in each calculation period, compare the structure response spectrum with the foot force spectrum in real time, and obtain the contribution of the foot force in the structure response.

[0012] The structure warning system module is used to take the structure acceleration peak value as the warning discrimination basis, give a warning when the structure acceleration peak value reaches the upper limit, and obtain the pedestrian-induced load in the corresponding frequency band in the group that causes excessive vibration of the bridge.

[0013] Further, the pedestrian monitoring module specifically comprises:

[0014] The pedestrian dataset creation and training module is configured to construct a pedestrian motion database based on a preset pedestrian dataset, and train the pedestrian dataset based on a target monitoring algorithm of deep learning.

[0015] The target continuous tracking module is configured to extract images from videos in the pedestrian motion database, pre-process the extracted images, obtain a pedestrian recognition frame of each image, and extract a pedestrian recognition frame image of the same pedestrian in continuous image frames.

[0016] The pedestrian displacement calculation module is configured to establish a world coordinate system at a light center of a left camera according to a three-dimensional coordinate calculation formula of binocular stereo vision, calculate three-dimensional coordinates of the pedestrian in the left camera coordinate system according to a pixel coordinate change sequence of the pedestrian recognition frame of the same pedestrian in the continuous image frames, and obtain image displacements of the pedestrian in the X-axis and Y-axis directions.

[0017] The plantar force reconstruction module is configured to estimate the height and weight of the pedestrian by using size information of the recognition frame, obtain vertical acceleration by derivation of the vertical displacement in the motion process of the pedestrian, and reconstruct the plantar force according to Newton's second law.

[0018] Preferably, the target monitoring algorithm based on deep learning includes a YOLOv5s algorithm.

[0019] Preferably, the preset pedestrian dataset includes a CUHK Occlusion Dataset pedestrian dataset.

[0020] Preferably, the multi-target tracking algorithm includes a DeepSort algorithm.

[0021] According to another aspect of the present application, the present application provides a human-bridge mutual displacement monitoring and early warning method based on visual technology, including the following steps:

[0022] S1.1: The target monitoring algorithm based on deep learning obtains the weight information of the pedestrian, measures the three-dimensional displacement of the pedestrian by using binocular stereo vision, obtains the acceleration information of the pedestrian by derivation of the vertical displacement, and reconstructs the plantar force according to the weight information and the acceleration information of the pedestrian.

[0023] S1.2: The Hough transform detection algorithm captures the landmark points of the bridge, measures the structural displacement of the landmark points by using binocular stereo vision, and obtains the structural acceleration of the bridge by derivation of the structural displacement.

[0024] S2: In each calculation period, the Fourier transform is performed on the foot force time history curve of the pedestrian and the bridge structure displacement time history curve respectively, the structure response spectrum is compared with the foot force spectrum of the pedestrian in real time, and the contribution of the foot force in the structure response is obtained;

[0025] S3: Taking the peak value of the structure acceleration as the early warning judgment basis, early warning is performed when the peak value of the structure acceleration reaches the upper limit, and the human-induced load causing excessive vibration of the bridge in the corresponding frequency band in the group is obtained.

[0026] Further, the step S1.1 specifically comprises:

[0027] S1.1.1: Based on the preset pedestrian data set, a pedestrian motion database is constructed, and a target monitoring algorithm based on deep learning is used to train the pedestrian data set;

[0028] S1.1.2: Image extraction is performed on the video in the pedestrian motion database, and the extracted images are preprocessed to obtain the pedestrian recognition frame of each image, and then the pedestrian recognition frame images of the same pedestrian in consecutive image frames are extracted; in the environment where the crowd is shielded, the prediction result and the monitoring result are cascaded and matched by using the intersection over union matching operation, so that the continuous tracking of the target is realized;

[0029] S1.1.3: According to the three-dimensional coordinate calculation formula of binocular stereo vision, the world coordinate system is established at the light center of the left camera, the pixel coordinate change sequence of the pedestrian recognition frame of the same pedestrian in consecutive image frames is obtained, and then the three-dimensional coordinates of the pedestrian in the left camera coordinate system are calculated by using the binocular stereo vision three-dimensional coordinate calculation, so that the image displacement of the pedestrian in the X axis and Y axis directions is obtained;

[0030] S1.1.4: The height and weight of the pedestrian are estimated by using the size information of the recognition frame, the vertical acceleration obtained by deriving the vertical displacement in the motion process of the pedestrian, and the foot force is reconstructed according to Newton's second law.

[0031] Preferably, the target monitoring algorithm based on deep learning comprises a YOLOv5s algorithm.

[0032] Preferably, the preset pedestrian data set comprises a CUHK Occlusion Dataset pedestrian data set.

[0033] Preferably, the multi-target tracking algorithm comprises a DeepSort algorithm.

[0034] The technical scheme provided by the application has the following beneficial effects:

[0035] (1) Traditional techniques often only focus on structural vibration or human motion, as structural vibration is caused by human motion acting on it, there is obviously a corresponding relationship between the two, and ignoring any one aspect cannot achieve real-time monitoring of the structural state. The present application provides a pedestrian-bridge mutual feedback displacement monitoring and early warning system and method based on visual technology, which can realize synchronous monitoring of pedestrian trajectory and structural vibration;

[0036] (2) When the structural acceleration reaches the threshold, the system reaches the early warning state. The present application analyzes the frequency information of the load causing excessive vibration of the structure by using the frequency spectrum of the structural response, and then compares it with the frequency spectrum made by the reconstructed plantar force according to the pedestrian trajectory, analyzes and finds out the human-induced load situation that causes the structural vibration problem. BRIEF DESCRIPTION OF DRAWINGS

[0037] The present application will be further described below in conjunction with the drawings and examples, wherein:

[0038] Figure 1 It is a structural schematic diagram of the pedestrian-bridge mutual feedback displacement monitoring and early warning system based on visual technology of the present application;

[0039] Figure 2 It is a pedestrian monitoring technology principle diagram of the present application;

[0040] Figure 3 It is a structural monitoring technology principle diagram of the present application;

[0041] Figure 4 It is an execution flowchart of the pedestrian-bridge mutual feedback displacement monitoring and early warning method based on visual technology of the present application;

[0042] Figure 5 It is a YOLOv5s network model structure diagram of the present application;

[0043] Figure 6 It is a convolution operation principle diagram of the present application;

[0044] Figure 7 It is a coordinate conversion relationship diagram of the present application;

[0045] Figure 8 It is a binocular stereo vision measurement coordinate system of the present application;

[0046] Figure 9 It is a reconstructed plantar force time history curve (a) and Fourier transform spectrum (b) of the present application;

[0047] Figure 10 It is a structural response time history curve (a) and Fourier transform spectrum (b) of the present application. DETAILED DESCRIPTION

[0048] In order to make the technical features, objectives and effects of the present application more clearly understood, the specific embodiments of the present application will now be described in detail with reference to the drawings.

[0049] Reference Figure 1 The embodiment of the present application provides a pedestrian-bridge mutual feedback displacement monitoring and early warning system based on visual technology, mainly including a pedestrian monitoring module, a structure monitoring module, a pedestrian-structure mutual feedback system module and a structure early warning module.

[0050] The pedestrian monitoring module:

[0051] The module needs to realize multi-target tracking of pedestrians and displacement recognition of each target, and the difficulty lies in solving the problems of multi-target recognition and environmental interference. The technical principle of the pedestrian monitoring module is as shown in Figure 2 .

[0052] The pedestrian motion database construction and training module is used for constructing a pedestrian motion database based on a preset pedestrian data set, and training the pedestrian data set based on a target monitoring algorithm of deep learning.

[0053] At present, many data sets for pedestrian detection have been disclosed, and the embodiment of the present application preferably uses the CUHK Occlusion Dataset pedestrian data set. Compared with similar data sets, the CUHK Occlusion Dataset contains multiple scenes, multiple perspectives and labeled human bodies under occlusion, and is therefore very suitable for the present application. The target monitoring algorithm based on deep learning is preferably a YOLOv5s algorithm, which is used to train the CUHK Occlusion Dataset pedestrian data set.

[0054] It should be noted that the binocular visual pedestrian monitoring camera is installed above the pedestrian motion area to ensure that the camera field of view can clearly capture all pedestrian images on the structure; the binocular visual bridge displacement camera is installed above the shooting area to ensure smooth and unobstructed view, and the left and right pictures of the two cameras can cover all the motion areas of pedestrians on the structure, and a laser range finder is used to determine the pitch angle of the camera; the shooting direction of the pedestrian includes the front, left front, right front, side, back, left back and right back directions.

[0055] The target continuous tracking module is used for image extraction of the video in the pedestrian motion database, pre-processing of the extracted images, obtaining the pedestrian recognition frame of each image, and extracting the pedestrian recognition frame image of the same pedestrian in the continuous image frame; in the environment with occlusion in the crowd, the prediction result and the monitoring result are cascaded and matched and the intersection over union matching operation is performed through the multi-target tracking algorithm, so as to realize continuous tracking of the target.

[0056] The pedestrian displacement calculation module calculates the three-dimensional coordinates of the pedestrian in the left camera coordinate system according to the three-dimensional coordinate calculation formula of binocular stereo vision, and obtains the image displacement of the pedestrian in the X-axis and Y-axis directions.

[0057] The foot force reconstruction module is used for estimating the height and weight of the pedestrian by using the size information of the recognition frame, and reconstructing the foot force according to the vertical acceleration obtained by derivation of the vertical displacement in the pedestrian movement process according to Newton's second law.

[0058] The structure monitoring module:

[0059] In the monitoring process of the structure state (strain, displacement, acceleration, etc.), the target object is fixed at the key position of the structure, the environmental interference is small, but the monitoring accuracy is high, and the technical principle of the structure monitoring module is as shown in Figure 3

[0060] (1) According to the actual monitoring requirements, install a plurality of marking points at different positions of the structure, and the midspan position is the necessary main monitoring point; the binocular vision structure monitoring module is installed at a position where the ground is flat and the line of sight is not blocked, and the left and right pictures of the two cameras need to collect all the markers of the bridge, and the distance between the device and the marker is within a suitable range. Use a laser range finder to determine the pitch angle of the camera. In addition, the camera frame rate needs to be ensured to ensure that the marker in the image is not distorted after the video is converted into an image.

[0061] (2) The same coordinate conversion method as in the binocular vision pedestrian monitoring module is used to convert the image coordinate system to the world coordinate system, and the vertical displacement of each marker of the structure in the world coordinate system is obtained. The time history curve of the displacement is derived to obtain the acceleration information of the structure.

[0062] The human-structure mutual feedback system module:

[0063] In each calculation period, Fourier transform is performed on the time history curve of the pedestrian foot force and the time history curve of the structure displacement, respectively. Real-time comparison of the frequency spectrum of the structure response and the frequency spectrum of the pedestrian foot force can know the contribution of the pedestrian foot force in the structure response.

[0064] The structure warning module:

[0065] ​The specifications for vibration comfort in various countries are different, but generally limit the vertical fundamental frequency and vertical acceleration value of the structure. The present application takes the peak value of the structure acceleration as the early warning basis. When the peak value of the structure acceleration reaches the upper limit, early warning is performed. When the structure reaches the early warning state, the corresponding component of the pedestrian-induced excessive vibration load of the bridge can be judged by comparing the foot force frequency spectrum in the group. Finally, the staff can guide the pedestrians who induce excessive vibration of the bridge, and solve the problem of excessive vibration of the bridge.

[0066] Please refer to Figure 4 The embodiment of the present application provides a person-bridge mutual feedback displacement monitoring and early warning method based on visual technology, which comprises the following steps:

[0067] S1.1: The target monitoring algorithm based on deep learning obtains the weight information of the pedestrian, uses binocular stereo vision to measure the three-dimensional displacement information of the pedestrian, and obtains the acceleration information of the pedestrian by deriving the vertical displacement, and reconstructs the foot force according to the weight information and the acceleration information of the pedestrian.

[0068] Specifically, the YOLOv5s algorithm used in the embodiment belongs to a single-stage target monitoring algorithm, has the characteristics of fast monitoring speed and good accuracy, and is very suitable for real-time monitoring of crowd information.

[0069] The structure of YOLOv5s includes an input end, a Backbone, a Neck and a Head, as shown in Figure 5 The Backbone is a convolutional neural network that aggregates and forms image features at different image granularities. The image features are extracted through multiple convolution operations, and the principle of convolution operation is as shown in Figure 6 Since the YOLOv5s algorithm and its structure are a relatively mature prior art, they will not be described in detail here.

[0070] The Neck part is used to generate a feature pyramid network (FPN), which can enhance the network structure to monitor targets of different sizes, so as to monitor pedestrians of different sizes.

[0071] In the Head part of YOLOv5s, K-means clustering is used to extract prior boxes, so that the algorithm can be applied to more different sizes of target objects and improve the monitoring accuracy.

[0072] In addition, the present application uses YOLOV5s algorithm as a monitor, and inputs the recognition result into DeepSort algorithm to jointly constitute a target tracking system. The DeepSort algorithm realizes the matching problem of multiple target objects in each frame of video image and the occlusion problem between each target object.

[0073] Through the above steps, the pedestrian trajectory is accurately recognized, and the recognized pedestrian is framed, so that the three-dimensional displacement of the pedestrian is measured.

[0074] In binocular vision principle, the conversion relationship among the world coordinate system, the camera coordinate system and the image coordinate system is shown in Figure 7 .

[0075] The conversion relationship among the coordinate systems is as follows:

[0076]

[0077] In the formula, s is a scale factor; (u, v) is the pixel coordinate of the target point; (k u ,k v ) respectively represent the proportional relationship between the pixels and the actual length in the horizontal direction and the vertical direction; (u0, v0) is the pixel coordinate of the image center point; f is the focal length of the camera; R and T represent the rotation matrix and the offset vector of the conversion from the world coordinate system to the camera coordinate system; (X w ,Y w ,Z w ) is the coordinate of the target point in the world coordinate system.

[0078] When the coordinate system conversion is performed, the positive direction of the three-dimensional coordinate system of the object surface is defined as: facing the object, parallel to the measured object surface and horizontally to the right as the positive direction of the x-axis; vertically downward as the positive direction of the y-axis; and vertically pointing to the measured object surface as the positive direction of the z-axis, as shown in Figure 8 .

[0079] Considering that the parameters of the left and right cameras in the binocular system cannot be completely the same, the parameters are distinguished by subscripts l and r. For the left and right camera coordinate systems and the world coordinate system, there are the following relationship formulas.

[0080]

[0081]

[0082] In the formula, s l and s r are scale factors; (X l ,Y l ) is the coordinate of the target point in the camera coordinate system; (x l ,y l ,z l ) is the coordinate of the target point in the world coordinate system.

[0083] At this time, the left camera coordinate system o l -x l y l z l and the right camera coordinate system or -x r y r z r The mutual positional relationship between them can be represented by formula (4).

[0084]

[0085] In the formula: matrix M lr represents the space conversion matrix from the left camera coordinate system o l -x l y l z l to the right camera coordinate system o r -x r y r z r .

[0086] Substitute formula (3) into formula (2) to obtain the corresponding relationship of the points on the image planes of the two cameras.

[0087]

[0088] Solve formula (4) and formula (1) to obtain the coordinates of a point in space in the left camera coordinate system.

[0089]

[0090] At this time, the origin of the world coordinate system is set at the optical center of the left camera, and the coordinates of a point in space in the world coordinate system are obtained.

[0091] Through this step, the pedestrian recognition frame obtained in step S1.1 is operated to obtain the time history curves of the vertical displacement and horizontal displacement of the pedestrian motion.

[0092] S1.2: Capture the bridge marker points based on the Hough transform detection algorithm, measure the structural displacement of the marker points using binocular stereo vision, and derive the structural displacement to obtain the structural acceleration of the bridge.

[0093] Specifically, the marker points are arranged at the key positions of the structure in the middle of the structure, the binocular vision structure monitoring module is installed at a place where the ground is stable and the line of sight is not blocked, and all markers of the bridge are captured by the left and right pictures of the two cameras. The distance between the device and the marker points is within a suitable range.

[0094] The above marker point monitoring is performed after image binarization processing, and the marker points are set as white background and black chessboard to improve the monitoring accuracy.

[0095] According to the binocular vision measurement principle in step S1.1, the bridge marker points are identified, and the algorithm principle of step S1.1 is used for operation to obtain the vertical displacement of the key positions of the bridge.

[0096] S2: In each calculation period, the Fourier transform is performed on the foot force time history curve of the pedestrian and the bridge structure displacement time history curve respectively, the structure response spectrum is compared with the foot force spectrum of the pedestrian in real time, and the contribution of the foot force in the structure response is obtained.

[0097] It should be noted that, since the Fourier transform needs to process data for a certain period of time, the calculation period is introduced, and a certain time and a period of time before that are set as the calculation period.

[0098] Specifically, after collecting the foot force time history curve of the pedestrian and the bridge structure displacement time history curve, the frequency spectrum characteristics of the two are analyzed by using the Fourier transform, the frequency bands with concentrated vibration energy in the frequency domain are compared, and the proportion of human-induced excitation in the structure response is analyzed.

[0099] Finally, the staff guides the pedestrian who causes excessive vibration of the bridge, and solves the problem of excessive vibration of the bridge.

[0100] S3: Taking the peak value of the structure acceleration as the early warning criterion, early warning is performed when the peak value of the structure acceleration reaches the upper limit, and the human-induced load in the group under the corresponding frequency band causing excessive vibration of the bridge is obtained.

[0101] Specifically, the evaluation standard referred to by the present application is the ATC (The Applied Technology Council) Design Guide 1: Minimizing Floor Vibration (1999) specification, ISO10137 (2007) and the German Guidelines for the Design of Footbridges.

[0102] The ATC (1999) specification only uses the peak value of the vertical acceleration for evaluation; the ISO10137 uses the frequency-weighted acceleration root mean square (R.M.S.) for evaluation, and the evaluation process is relatively complicated; the German Guidelines for the Design of Footbridges adopts a method combining the natural frequency of the bridge with the peak value of the human-induced vibration of the bridge to divide the comfort level of the footbridge, and the evaluation process is not only simple and detailed, but also divides the comfort level into different grades, and comprehensively considers the peak value of the human-induced vibration of the bridge and the natural frequency of the bridge, so the present application analyzes and evaluates the comfort level index of the pedestrian crossing the bridge according to the relevant indexes of the German Guidelines for the Design of Footbridges.

[0103] The step S2 obtains the vertical displacement time history curve at the key point of the structure, and the vertical acceleration time history curve of the structure is obtained by derivation, and the comfort level is determined according to the comfort index.

[0104] In the German Bridge Design Guidelines, the comfort of pedestrians crossing the bridge is determined by the acceleration of the pedestrian bridge. The specification recommends four comfort levels, as shown below.

[0105] Table 1 Pedestrian bridge acceleration comfort index

[0106]

[0107] When the structural acceleration reaches 1.0 m / s 2 , the structure reaches the warning state, and when the structure reaches the warning state, the corresponding frequency band in the group can be determined to induce excessive vibration of the bridge.

[0108] Experimental verification:

[0109] In the experiment, a group of 2.5 Hz pedestrian queue crossing the bridge was set up, and the bridge natural frequency selected in the experiment is shown in Table 2.

[0110] Table 2 Natural frequency characteristics

[0111]

[0112]

[0113] The following steps are performed.

[0114] S1 Pedestrian monitoring and structure vibration monitoring

[0115] S1.1 Pedestrian monitoring

[0116] YOLOv5+DeepSort algorithm is used to monitor people in real time.

[0117] The actual weight information of the subjects is compared with the monitored weight information, and the results are shown in the following table.

[0118] Table 3 Comparison of subject weight information

[0119]

[0120] The above results show that the use of monitoring means to estimate weight information has an error within an acceptable range.

[0121] Using the principle of binocular stereo vision measurement, the three-dimensional displacement of the subjects is measured, and the vertical displacement is derived to obtain the vertical acceleration time history curve of the pedestrian motion.

[0122] Through the monitored pedestrian weight information and vertical acceleration information, the foot force of the pedestrian is reconstructed using Newton's second law, as shown in the formula below.

[0123] GRF-G=ma (7)

[0124] wherein GRF is the reconstructed pedestrian foot force, and m, a in the above equation are the pedestrian weight information and the vertical acceleration of motion obtained by the above steps, and G is the human body gravity.

[0125] The time history curve of the foot force finally obtained and the frequency spectrum after Fourier transform are shown in Figure 9 .

[0126] S1.2: Structure monitoring

[0127] When the pedestrian moves on the bridge structure, the binocular vision monitoring method is used to measure the structure displacement, the structure response time history curve and the frequency spectrum after Fourier transform are shown in Figure 10 .

[0128] S2: Human-bridge mutual feedback

[0129] In the above experiment, two peak values are contained in the frequency spectrum of the structure response, which are 2.76 Hz and 2.5 Hz, the frequency spectrum of the structure response is compared with the structure natural vibration characteristics and the foot force frequency spectrum of the pedestrian, it is judged that the peak value 2.76 Hz is the natural vibration frequency of the bridge structure, and the peak value 2.5 Hz is the pedestrian-induced load frequency of the bridge vibration, i.e. the foot force frequency 2.5 Hz of the pedestrian.

[0130] S3: Structure early warning

[0131] The structure frequency spectrum analysis is performed according to the above steps, taking the above experiment as an example, the peak acceleration of the structure response is 0.15 m / s 2 , the structure response does not reach the early warning state, in actual application, if the peak acceleration exceeds the standard, the frequency spectrum comparison is performed according to the above steps, the pedestrian-induced load of the bridge excessive vibration in the corresponding frequency band in the group can be judged.

[0132] It should be noted that in this document, the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or system that includes a series of elements not only includes those elements, but also includes other elements not explicitly listed, or inherent to such a process, method, article or system. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of another identical element in the process, method, article or system that includes the element.

[0133] The above-mentioned embodiment numbers of the application are only for description, and do not represent the advantages and disadvantages of the embodiments. In the unit claims of several devices, several of these devices can be embodied by the same hardware item. The use of the words first, second, and third does not represent any order, and these words can be interpreted as identifiers.

[0134] The above merely describes the preferred embodiments of the present application, and is not intended to limit the patent scope of the present application, and any equivalent structure or equivalent process conversion, or direct or indirect application in other related technical fields, which are made by using the content of the present application specification and drawings, are also included in the patent protection scope of the present application.

Claims

1. A human-bridge mutual displacement monitoring and early warning system based on vision technology, characterized in that, include: Pedestrian monitoring module, structure monitoring module, human-structure feedback system module, and structure early warning system module; The pedestrian monitoring module is used to obtain pedestrian weight information based on a deep learning-based target monitoring algorithm, measure pedestrian three-dimensional displacement using binocular stereo vision, obtain pedestrian acceleration information by differentiating the vertical displacement, and reconstruct plantar force based on pedestrian weight and acceleration information. The structural monitoring module is used to capture the bridge's landmark points based on the Hough transform detection algorithm, measure the structural displacement at the landmark points using binocular stereo vision, and obtain the bridge's structural acceleration by differentiating the structural displacement. The human-structure feedback system module is used to perform Fourier transforms on the pedestrian foot force time history curve and the bridge structure displacement time history curve in each calculation cycle, and compare the structural response spectrum and the pedestrian foot force spectrum in real time to obtain the contribution of foot force in the structural response. The structural early warning system module is used to use the peak value of structural acceleration as the basis for early warning. When the peak value of structural acceleration reaches the upper limit, an early warning is issued, and the man-made loads that cause excessive vibration of the bridge in the corresponding frequency band in the group are obtained.

2. The human-bridge mutual displacement monitoring and early warning system according to claim 1, characterized in that, The pedestrian detection module specifically includes: The pedestrian dataset creation and training module is used to build a pedestrian motion database based on a preset pedestrian dataset and to train the pedestrian dataset based on a deep learning-based target detection algorithm. The target continuous tracking module is used to extract images from videos in the pedestrian motion database, preprocess the extracted images to obtain pedestrian recognition boxes for each frame, and then extract the pedestrian recognition box images of the same pedestrian in consecutive image frames; in environments where there is occlusion in the crowd, the multi-target tracking algorithm performs cascade matching and cross-union matching operations on the prediction results and monitoring results to achieve continuous tracking of the target; The pedestrian displacement calculation module is used to establish the world coordinate system at the optical center of the left camera according to the three-dimensional coordinate calculation formula of binocular stereo vision. Based on the pixel coordinate change sequence of the pedestrian recognition box in consecutive frame images, the three-dimensional coordinates of the pedestrian in the left camera coordinate system are calculated using the three-dimensional coordinates of binocular stereo vision, thereby obtaining the image displacement of the pedestrian in the X-axis and Y-axis directions. The plantar force reconstruction module is used to estimate the height and weight of pedestrians using the size information of the recognition box, and to reconstruct the plantar force based on the vertical acceleration obtained by differentiating the vertical displacement during the pedestrian's movement and Newton's second law.

3. The human-bridge mutual displacement monitoring and early warning system according to claim 2, characterized in that, The deep learning-based target detection algorithm includes the YOLOv5s algorithm.

4. The human-bridge mutual displacement monitoring and early warning system according to claim 2, characterized in that, The preset pedestrian dataset includes the CUHK Occlusion Dataset pedestrian dataset.

5. The human-bridge mutual displacement monitoring and early warning system according to claim 2, characterized in that, The multi-target tracking algorithm includes the DeepSort algorithm.

6. A method for monitoring and early warning of displacement between a human and a bridge based on vision technology, characterized in that, Includes the following steps: S1.1: A deep learning-based target detection algorithm is used to obtain pedestrian weight information, and binocular stereo vision is used to measure the pedestrian's three-dimensional displacement. The vertical displacement is differentiated to obtain the pedestrian's acceleration information, and the plantar force is reconstructed based on the pedestrian's weight and acceleration information. S1.2: Based on the Hough transform detection algorithm, the bridge's landmark points are captured, the structural displacement at the landmark points is measured using binocular stereo vision, and the structural acceleration of the bridge is obtained by differentiating the structural displacement. S2: In each calculation cycle, Fourier transforms are performed on the pedestrian plantar force time history curve and the bridge structure displacement time history curve, respectively. The structural response spectrum and the pedestrian plantar force spectrum are compared in real time to obtain the contribution of plantar force in the structural response. S3: Using the peak value of structural acceleration as the basis for early warning, an early warning is issued when the peak value of structural acceleration reaches the upper limit, and the man-made load that causes excessive vibration of the bridge in the corresponding frequency band in the population is obtained.

7. The human-bridge mutual displacement monitoring and early warning method according to claim 6, characterized in that, Step S1.1 specifically includes: S1.1.1: Based on the pre-set pedestrian dataset, construct a pedestrian motion database, and train the pedestrian dataset based on a deep learning-based target detection algorithm; S1.1.2: Extract images from videos in the pedestrian motion database, preprocess the extracted images to obtain pedestrian recognition boxes for each frame, and then extract the pedestrian recognition box images of the same pedestrian in consecutive image frames; in environments with occlusion in the crowd, use a multi-target tracking algorithm to perform cascade matching and cross-union matching operations on the prediction results and monitoring results to achieve continuous tracking of the target; S1.1.3: Based on the three-dimensional coordinate calculation formula of binocular stereo vision, the world coordinate system is established at the optical center of the left camera. According to the pixel coordinate change sequence of the pedestrian recognition box in consecutive frame images of the same pedestrian, the three-dimensional coordinates of the pedestrian in the left camera coordinate system are calculated using the three-dimensional coordinates of binocular stereo vision, thereby obtaining the image displacement of the pedestrian in the X-axis and Y-axis directions. S1.1.4: Estimate the height and weight of pedestrians using the size information of the recognition box, obtain the vertical acceleration by differentiating the vertical displacement during the pedestrian's movement, and reconstruct the plantar force according to Newton's second law.

8. The method for monitoring and early warning of displacement between a human-bridge system according to claim 7, characterized in that, The deep learning-based target detection algorithm includes the YOLOv5s algorithm.

9. The human-bridge mutual displacement monitoring and early warning method according to claim 7, characterized in that, The preset pedestrian dataset includes the CUHK Occlusion Dataset pedestrian dataset.

10. The human-bridge mutual displacement monitoring and early warning method according to claim 7, characterized in that, The multi-target tracking algorithm includes the DeepSort algorithm.

Images