Model training method, vibration monitoring method, device, equipment, medium and product

Abstract

The application provides a model training method, a vibration monitoring method, a device, equipment, a medium and a product, and relates to the technical field of target measurement. The model training method comprises the following steps: acquiring a plurality of first target images, and performing feature extraction processing on each first target image through a convolutional neural network to obtain a first feature image corresponding to each first target image; performing channel enhancement processing, spatial enhancement processing and detail extraction processing on the first feature image corresponding to each first target image to obtain a third feature image corresponding to each first target image; obtaining a target data set according to the third feature image corresponding to each first target image, and then training a target recognition model; and the target recognition model is used for identifying the position of a visual target from a second target image, and then obtaining the vibration change of a structure. The method provided by the application improves the monitoring frequency and meets the efficient, continuous and automatic vibration monitoring requirements of the structure.

Description

Technical Field

[0001] This application relates to the field of target measurement, and more particularly to a model training method, vibration monitoring method, device, equipment, medium, and product. Background Technology

[0002] During long-term service, structures such as roads and bridges experience continuous and cumulative vibration changes due to minute strains. Although these vibration changes are small in magnitude, they have a potential impact on the safety and durability of the structure. Therefore, high-precision and high-frequency monitoring of these minute vibration changes is particularly crucial.

[0003] Currently, structural monitoring mainly relies on traditional total station equipment, which combines high-precision polar coordinate laser ranging technology to accurately measure the three-dimensional coordinates of targets installed on structures, thereby enabling the monitoring of vibration changes in structures.

[0004] However, traditional total station equipment usually requires manual intervention and adopts a point-by-point scanning method for data collection. This not only limits the monitoring frequency, but also increases the complexity and cost of long-term monitoring, making it difficult for traditional total station equipment to meet the needs of efficient, continuous and automated vibration monitoring of structures. Summary of the Invention

[0005] This application provides a model training method, vibration monitoring method, device, equipment, medium, and product, which improves the monitoring frequency and meets the needs of efficient, continuous, and automated vibration monitoring of structures.

[0006] A first aspect of this application provides a model training method, the method comprising:

[0007] Multiple first target images are acquired, and feature extraction processing is performed on each first target image through a preset convolutional neural network to obtain a first feature image corresponding to each first target image; wherein, the first target image shows a visual target installed on a structure;

[0008] Channel enhancement and spatial enhancement are performed on the first feature image corresponding to each first target image to obtain the second feature image corresponding to each first target image. Then, detail extraction is performed on the second feature image corresponding to each first target image to obtain the third feature image corresponding to each first target image.

[0009] Based on the third feature image corresponding to each first target image, a target dataset is obtained, and a target recognition model is trained based on the target dataset. The target recognition model is used to identify the position of the visual target from the second target image captured by the vibration monitoring device. The position of the visual target in the second target image is used to indicate the vibration change of the structure.

[0010] In one possible design, the target image is any one of a plurality of first target images; for each target image, channel enhancement and spatial enhancement processing are performed on the first feature image corresponding to each first target image to obtain a second feature image corresponding to each first target image, including:

[0011] Based on the average pixel values ​​of the first feature image corresponding to the target image in multiple channels, the channel description vector of each channel is obtained, and the spatial normalization process of the first feature image corresponding to the target image is performed to obtain the spatial weight matrix.

[0012] Based on multiple channel description vectors and spatial weight matrices, the dynamic enhancement weights for each channel are calculated using an adaptive weight generation algorithm.

[0013] Based on multiple dynamic enhancement weights and the first feature image, the second feature image corresponding to the target image is obtained by multiplying them point by point.

[0014] In one possible design, based on multiple channel description vectors and a spatial weight matrix, an adaptive weight generation algorithm is used to calculate the dynamic enhancement weights for each channel, including:

[0015] Based on multiple channel description vectors and spatial weight matrices, the comprehensive feature description of each channel is calculated through point-by-point multiplication;

[0016] Based on multiple comprehensive feature descriptions, the dynamic enhancement weights for each channel are calculated using a dynamic mapping function that includes a multilayer perceptron.

[0017] In one possible design, the target image is any one of a plurality of first target images; for each target image, detail extraction processing is performed on the second feature image corresponding to each first target image to obtain a third feature image corresponding to each first target image, including:

[0018] The number of channels in the second feature image corresponding to the target image is compressed to obtain a channel-compressed feature image;

[0019] The channel compressed feature image is input into the first convolutional module, which includes multiple first convolutional layers, to obtain the scale feature image output by each first convolutional layer. The receptive field of the channel compressed feature image is then expanded by dilated convolution to obtain the receptive field expanded feature image.

[0020] Interacting multiple scale feature images and receptive field expansion feature images yields an interactive information feature image.

[0021] The interactive information feature image is dynamically enhanced by local region response through a local region attention mechanism to obtain a locally enhanced feature image.

[0022] By dynamically selecting and enhancing multiple channels of the local enhanced feature image through adaptive feature selection, a third feature image corresponding to the target image is obtained.

[0023] In one possible design, multiple scale feature images and receptive field expansion feature images are interacted to obtain an interactive information feature image, including:

[0024] Based on feature images at multiple scales and feature images of expanded receptive fields, the intermediate feature image of interactive information is obtained by adding elements one by one.

[0025] The intermediate feature image of the interaction information is input into the second convolutional module, which includes multiple second convolutional layers, to obtain the interaction information feature image output by the second convolutional module.

[0026] In one possible design, a local region attention mechanism is used to dynamically enhance the interactive information feature image to obtain a locally enhanced feature image, including:

[0027] By selecting regions of interest, the local feature regions where the visual target is located are extracted from the interactive information feature images.

[0028] The third convolutional layer optimizes local features in the local feature regions to obtain a locally enhanced feature image.

[0029] In one possible design, multiple channels of the locally enhanced feature image are dynamically selected and enhanced through adaptive feature selection to obtain a third feature image corresponding to the target image, including:

[0030] The importance weight of the channel compressed feature image in each channel is calculated through global pooling operation;

[0031] Based on multiple importance weights and local enhancement feature images, the third feature image corresponding to the target image is obtained by weighted averaging.

[0032] In one possible design, a target recognition model is trained based on a target dataset, including:

[0033] The target dataset is divided into a model training dataset and a model validation dataset.

[0034] The target recognition model is trained based on the model training dataset.

[0035] The target recognition model was validated using the model validation dataset.

[0036] A second aspect of this application provides a vibration monitoring method applied to a vibration monitoring device, the method comprising:

[0037] Periodically photograph the visual target mounted on the structure to obtain a second target image each time;

[0038] The location of the visual target is identified from each second target image using a pre-trained target recognition model; wherein the target recognition model is trained using the model training method described in the first aspect of the invention.

[0039] Based on the position of the visual target in each second target image, the first monitoring result of the vibration change of the structure is obtained.

[0040] In one possible design, based on the position of the visual target in each second target image, the first monitoring result of the vibration change of the structure is obtained, including:

[0041] Based on the position of the visual target in each second target image, the three-dimensional coordinates of the visual target in the corresponding second target image are calculated through a pre-stored first mapping relationship; wherein, the first mapping relationship is used to indicate the mapping relationship between the position of the visual target and the three-dimensional coordinates;

[0042] The first monitoring result of the vibration change of the structure is obtained based on the three-dimensional coordinates of the visual target in each second target image.

[0043] In one possible design, the vibration monitoring equipment includes a total station and a machine vision device mounted on the total station; wherein the machine vision device is fixed in position relative to the laser rangefinder of the total station.

[0044] Before periodically photographing a visual target mounted on a structure to obtain a second image of the target after each photograph, the method further includes:

[0045] The laser rangefinder is calibrated using a reference target until the laser emitted by the laser rangefinder is projected onto the reference target; when the laser rangefinder calibration is complete, the second target image displays a visual target.

[0046] In one possible design, a laser target is also mounted on the structure, in which case the method further includes:

[0047] The laser target is measured using a laser rangefinder, and the three-dimensional coordinates of the laser target are obtained for each measurement.

[0048] The second monitoring result of the vibration change of the structure is obtained based on the three-dimensional coordinates of the laser target measured each time.

[0049] In one possible design, the accuracy of the three-dimensional coordinates of the visual target is less than that of the three-dimensional coordinates of the laser target.

[0050] When the first monitoring result of vibration change indicates that the structure is undergoing vibration change, the laser target is measured using a laser rangefinder to obtain the three-dimensional coordinates of the laser target measured each time, including:

[0051] Based on the first monitoring result of vibration changes, the offset of the visual target is obtained;

[0052] The offset of the laser target is calculated based on the offset of the visual target through a pre-stored second mapping relationship; wherein, the second mapping relationship is used to indicate the mapping relationship between the relative positions of the visual target and the laser target.

[0053] Based on the offset of the laser target, the laser rangefinder is controlled to deflect, and the laser target is measured through the deflected laser rangefinder to obtain the three-dimensional coordinates of the laser target measured each time.

[0054] A third aspect of this application provides a model training apparatus, the apparatus comprising:

[0055] The feature extraction module is used to acquire multiple first target images and perform feature extraction processing on each first target image through a preset convolutional neural network to obtain a first feature image corresponding to each first target image; wherein, the first target image displays a visual target installed on a structure;

[0056] The feature processing module is used to perform channel enhancement processing and spatial enhancement processing on the first feature image corresponding to each first target image to obtain the second feature image corresponding to each first target image, and to perform detail extraction processing on the second feature image corresponding to each first target image to obtain the third feature image corresponding to each first target image.

[0057] The model training module is used to obtain a target dataset based on the third feature image corresponding to each first target image, and to train a target recognition model based on the target dataset. The target recognition model is used to identify the position of the visual target from the second target image captured by the vibration monitoring device. The position of the visual target in the second target image is used to indicate the vibration change of the structure.

[0058] A fourth aspect of this application provides a model training apparatus located within a vibration monitoring device, the apparatus comprising:

[0059] The image capturing module is used to periodically capture images of visual targets mounted on the structure, obtaining a second target image each time.

[0060] A location recognition module is used to identify the location of a visual target from each second target image using a pre-trained target recognition model; wherein the target recognition model is trained using the model training method described in the first aspect of the invention.

[0061] The first monitoring module is used to obtain the first monitoring result of the vibration change of the structure based on the position of the visual target in each second target image.

[0062] A fifth aspect of this application provides an electronic device, including: a memory, and a memory communicatively connected to a processor;

[0063] The memory stores the instructions that the computer executes;

[0064] When the processor executes computer execution instructions stored in memory, it is used to implement the model training method of any one of the first aspects, or the vibration monitoring method of any one of the second aspects.

[0065] The sixth aspect of this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the model training method of any one of the first aspects or the vibration monitoring method of any one of the second aspects.

[0066] The seventh aspect of this application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the model training method of any one of the first aspects, or the vibration monitoring method of any one of the second aspects.

[0067] This application provides a model training method, vibration monitoring method, device, equipment, medium, and product. The model training method includes: acquiring multiple first target images, and performing feature extraction processing on each first target image using a convolutional neural network to obtain a first feature image corresponding to each first target image; performing channel enhancement processing, spatial enhancement processing, and detail extraction processing on the first feature image corresponding to each first target image to obtain a third feature image corresponding to each first target image; obtaining a target dataset based on the third feature image corresponding to each first target image, and then training a target recognition model; the target recognition model is used to identify the position of the visual target from the second target image, thereby obtaining the vibration change of the structure. The following technical effects are achieved: monitoring the vibration change of the structure by using visual targets installed on the structure increases the monitoring frequency and meets the requirements for efficient, continuous, and automated vibration monitoring of the structure; performing channel enhancement and spatial enhancement processing on the first feature image corresponding to each first target image enhances the display effect of each visual target in the first feature image; performing detail extraction processing on the second feature image corresponding to each first target image effectively captures the detailed information of each visual target in the second feature image. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0069] Figure 1 A schematic diagram illustrating a scenario for the model training method and vibration monitoring method provided in the embodiments of this application;

[0070] Figure 2 A flowchart illustrating the model training method provided in the embodiments of this application. Figure 1 ;

[0071] Figure 3 A flowchart illustrating the model training method provided in the embodiments of this application. Figure 2 ;

[0072] Figure 4 A schematic flowchart of the vibration monitoring method provided in the embodiments of this application;

[0073] Figure 5 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application;

[0074] Figure 6 This is a schematic diagram of the structure of the vibration monitoring device provided in the embodiments of this application;

[0075] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0076] Figure label:

[0077] 110-Vibration monitoring equipment; 111-Machine vision instrument; 112-Total station; 1121-Laser rangefinder; 113-Leveler; 114-Tripod; 120-Structure; 121-Bridge; 130-Visual target; 140-Laser target;

[0078] 510 - Feature extraction module; 520 - Feature processing module; 530 - Model training module;

[0079] 610 - Image capture module; 620 - Location recognition module; 630 - First monitoring module. Detailed Implementation

[0080] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0081] In this application, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, nor do they necessarily imply difference. It should be noted that in this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. In this application, "at least one" means one or more, and "more than one" means two or more.

[0082] It should be noted that the phrase "at the moment when..." in this application can refer to the instant at which a certain situation occurs, or to a period of time after the occurrence of a certain situation; this application does not impose a specific limitation in this regard. Furthermore, the model training method and vibration monitoring method provided in this application are merely examples, and the model training method and vibration monitoring method may also include more or less content. The user information (including but not limited to user device information and user personal information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in one or more embodiments of this application are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of related data must comply with relevant laws, regulations, and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0083] To clearly understand the technical solution of this application, the solutions of the prior art will be described in detail first.

[0084] During long-term service, structures such as roads and bridges experience minute strains due to factors such as vehicle loads, wind force, and temperature changes. These minute strains lead to continuous and cumulative vibrational changes in the structures. Although the amplitude of these vibrational changes is small, they have a potential impact on the safety and durability of the structure. Therefore, high-precision and high-frequency monitoring of these minute vibrational changes is particularly crucial.

[0085] Currently, structural monitoring mainly relies on traditional total station equipment. By combining high-precision polar coordinate laser ranging technology, it measures the horizontal and vertical angles of the target to determine the direction relative to the target, and measures the reflection time of the laser to determine the distance relative to the target. Finally, it uses the principle of triangulation to achieve accurate measurement of the three-dimensional coordinates of the target, thereby realizing the monitoring of vibration changes in the structure.

[0086] However, traditional total stations typically require manual projection of the laser emitted by the total station's laser rangefinder onto the target, and data acquisition is done through point-by-point scanning. Each measurement can only obtain the three-dimensional coordinates of one target, thus requiring a considerable amount of time to collect the three-dimensional coordinates of different targets one by one. Furthermore, if real-time monitoring of subtle vibration changes in a structure is required, the point-by-point scanning of traditional total stations may not provide sufficient timeliness. This not only limits the monitoring frequency but also increases the complexity and cost of long-term monitoring, making it difficult for traditional total stations to meet the needs of efficient, continuous, and automated vibration monitoring of structures.

[0087] Therefore, to address the aforementioned issues, the research found that machine vision monitoring technology can be used to monitor vibration changes in structures. Machine vision monitoring technology is a two-dimensional dynamic monitoring technology based on a fixed field of view constraint. Its core principle is to achieve accurate perception of vibration changes in structures within a fixed field of view through a pixel difference algorithm on a frame-by-frame basis.

[0088] Furthermore, considering that machine vision monitoring technology has low accuracy in monitoring long-distance small targets such as visual targets, the laser ranging technology of the total station and the machine vision monitoring technology of the machine vision instrument are combined to achieve accurate perception of structural vibration changes.

[0089] Based on the above-mentioned inventive discovery, the technical solution of this application is proposed.

[0090] The following section introduces the application scenarios of the model training method and vibration monitoring method provided in this application.

[0091] Figure 1 This is a schematic diagram illustrating a scenario for the model training method and vibration monitoring method provided in the embodiments of this application. It should be noted that... Figure 1 The examples shown are merely examples of scenarios in which this application can be applied, to help those skilled in the art understand the technical content of this application, but do not mean that this application cannot be used in other devices, systems, environments or scenarios.

[0092] like Figure 1 As shown, the application scenario includes: vibration monitoring equipment 110, structure 120, and visual target 130 installed on structure 120.

[0093] The vibration monitoring device 110 includes a machine vision instrument 111, which may further include a zoom lens, a complementary metal-oxide-semiconductor (CMOS) sensor, an image analysis processor, and an image output device. The machine vision instrument 111 is used to achieve high-definition and high-frequency imaging of the visual target 130 at different distances, and to analyze the minute displacements of the visual target 130 in different directions.

[0094] Structure 120 can be a road, bridge, urban rail transit, or high-rise building, etc. For example, structure 120 includes a bridge 121 and an embankment 122 for supporting the bridge 121. Visual targets 130 are installed on both the bridge 121 and the embankment 122.

[0095] The visual target 130 can be a circular target or a checkerboard target, etc. The vibration monitoring device 110 captures the visual target 130 to obtain the target image, and then obtains the vibration change monitoring results of the structure 120.

[0096] Furthermore, the vibration monitoring device 110 also includes a total station 112, a leveler 113, and a tripod 114. A machine vision instrument 111 is mounted on the total station 112; specifically, the machine vision instrument 111 is fixed in position relative to the laser rangefinder 1121 of the total station 112. The leveler 113 is used to realize the automatic horizontal and vertical orientation adjustment function of the total station 112, and the tripod 114 is used to support the leveler 113.

[0097] Laser targets 140 are also installed on bridge 121 and embankment 122. The laser targets 140 are prisms used to reflect the lasers projected by laser rangefinders 1121. The laser rangefinders 1121 measure the laser targets 140 to obtain the three-dimensional coordinates of the laser targets 140, and thus obtain the vibration change monitoring results of the structure 120.

[0098] The technical solutions of this application will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0099] Figure 2 A flowchart illustrating the model training method provided in the embodiments of this application. Figure 1 .like Figure 2 As shown in the embodiments of this application, the execution entity can be a model training device, which can be located in an electronic device, and the device can be a data processing server. The model training method provided in the embodiments of this application includes the following steps:

[0100] S201. Acquire multiple first target images, and perform feature extraction processing on each first target image through a preset convolutional neural network to obtain the first feature image corresponding to each first target image.

[0101] Specifically, the first target image is used to train the target recognition model. It can be an image captured by vibration monitoring equipment specifically for model training, or it can be an image captured in the past for structural vibration monitoring. The first target image displays visual targets installed on the structure. If multiple visual targets are installed on the structure, each first target image can display all visual targets, or each first target image can display a portion of the visual targets. Multiple first target images are combined or stitched together to display all visual targets.

[0102] After acquiring multiple first target images, these images are input into a convolutional neural network (CNN) to perform feature extraction on each first target image, resulting in a first feature image corresponding to each first target image output by the CNN. The CNN can be a fully convolutional network (FCN), a spatial-based convGNN, a diffusion convolutional neural network (DCNN), a mixture model network (MoNET), or a graph attention network (GAT), etc. The dimensions of the output first feature image can be represented as C×H×W, where C represents the number of channels in the corresponding first feature image, and H and W represent the height and width of the corresponding first feature image, respectively.

[0103] S202. Perform channel enhancement and spatial enhancement processing on the first feature image corresponding to each first target image to obtain the second feature image corresponding to each first target image, and perform detail extraction processing on the second feature image corresponding to each first target image to obtain the third feature image corresponding to each first target image.

[0104] Specifically, the visual target is a small target, and its proportion in the first target image is usually small. Therefore, in order to enhance the display effect of each visual target in each first feature image, it is necessary to perform channel enhancement processing and spatial enhancement processing on the first feature image corresponding to each first target image.

[0105] Channel enhancement primarily focuses on optimizing information in different channels of the first feature image, such as the RGB channels and multispectral channels, aiming to enhance useful information in specific channels and suppress noise. Spatial enhancement primarily focuses on adjusting information in the spatial domain of the first feature image, aiming to improve its spatial resolution and detail representation. Combining channel enhancement and spatial enhancement synergistically applies the process to the first feature image to obtain the second feature image corresponding to each first target image.

[0106] Meanwhile, visual targets are usually installed on key nodes of the structure, and they may present a problem of sparse image and complex background in the first target image. Therefore, in order to effectively capture the detailed information of each visual target in each second feature image, it is necessary to perform detail extraction processing on the second feature image corresponding to each first target image.

[0107] The detailed information of the visual target includes edges, textures, and corners. The focus of detail extraction processing is to achieve feature integration and information interconnection of the visual target. Through multi-angle feature processing methods, the visual target and background are understood and distinguished from the second feature image to obtain the third feature image corresponding to each first target image. Specifically, this step combines local feature enhancement and fine-grained adjustment, adaptive feature selection, and other techniques to dynamically optimize the perception capability of the target region, reduce background interference in the second feature image, and improve the ability to identify visual targets in complex environments.

[0108] S203. Obtain the target dataset based on the third feature image corresponding to each first target image, and train the target recognition model based on the target dataset.

[0109] Specifically, the third feature images corresponding to each first target image are sorted and organized. For example, these third feature images are first classified according to factors such as the number of targets and shooting conditions. Then, the background part of the third feature images is cropped to create a target dataset focused on the target.

[0110] In training the target recognition model, the first step is to select a suitable model architecture based on task requirements and data characteristics. Common model architectures include Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Long Short-Term Memory Networks (LSTM). After selecting the model, it is trained using a target dataset. First, the model's predicted output is calculated using the forward propagation algorithm. Then, based on the loss function between the model's predicted output and the true label, the gradient of this loss function with respect to the model parameters is calculated using the backpropagation algorithm. Finally, optimization algorithms such as stochastic gradient descent are used to adjust the model parameters to minimize the loss function.

[0111] Furthermore, after training the target recognition model, its performance can be verified by inputting a validation dataset.

[0112] Once the target recognition model is trained, it can be applied to the second target images captured by the vibration monitoring equipment. The model identifies the position of the visual target in the second target images captured by the vibration monitoring equipment; the position of the visual target in the second target images indicates changes in the structure's vibration. Specifically, by comparing the changes in the position of the visual target in the second target images taken at different time points, the displacement change of the structure during vibration can be calculated. Based on this displacement change, vibration characteristic parameters such as the vibration amplitude and frequency of the structure can be further calculated. These vibration characteristic parameters are crucial for determining whether the structure's vibration state meets the expected range and provide important basis for subsequent structural health monitoring. When abnormal vibration is detected, timely early warning information can be issued so that appropriate measures can be taken.

[0113] This application provides a model training method, which includes: acquiring multiple first target images, and performing feature extraction processing on each first target image using a convolutional neural network to obtain a first feature image corresponding to each first target image; performing channel enhancement processing, spatial enhancement processing, and detail extraction processing on the first feature image corresponding to each first target image to obtain a third feature image corresponding to each first target image; obtaining a target dataset based on the third feature image corresponding to each first target image, and then training a target recognition model; the target recognition model is used to identify the position of a visual target from a second target image, thereby obtaining the vibration change of the structure. This achieves the following technical effects: monitoring the vibration change of the structure using visual targets installed on the structure increases the monitoring frequency and meets the requirements for efficient, continuous, and automated vibration monitoring of the structure; performing channel enhancement and spatial enhancement processing on the first feature image corresponding to each first target image enhances the display effect of each visual target in the first feature image; and performing detail extraction processing on the second feature image corresponding to each first target image effectively captures the detailed information of each visual target in the second feature image.

[0114] Figure 3 A flowchart illustrating the model training method provided in the embodiments of this application. Figure 2 .like Figure 3 As shown, the model training method provided in this application embodiment is... Figure 2 Based on the model training method provided in this embodiment, further refinement is made. If the target image is any one of multiple first target images, then for that target image, the model training method provided in this embodiment includes the following steps.

[0115] S301. Acquire multiple first target images, and perform feature extraction processing on each first target image through a preset convolutional neural network to obtain the first feature image corresponding to each first target image.

[0116] S302. Based on the average pixel values ​​of the first feature image corresponding to the target image in multiple channels, obtain the channel description vector of each channel, and perform spatial normalization processing on the first feature image corresponding to the target image to obtain the spatial weight matrix.

[0117] Specifically, the first feature image corresponding to the target image is represented as X. In existing technologies, spatial enhancement processing is usually performed only on each channel of X, such as global max pooling and global average pooling. The improvement of this application embodiment is that, for the obtained X, statistical features of X in both channel and spatial dimensions are calculated through joint modeling of channel dimension analysis and spatial dimension analysis. The generation of statistical information at the channel and spatial levels provides a basis for subsequent adaptive weight generation, which helps to understand and identify the possible location and related features of small targets such as visual targets. The channel enhancement processing and spatial enhancement processing of X include the following two steps:

[0118] The first step is to calculate the pixel mean of X in each of the multiple channels of X, thereby obtaining the channel description vector for each channel. . The formula used to indicate the salient features included in the corresponding channel is as follows:

[0119]

[0120] in, H refers to the channel description vector of the c-th channel, representing the average value of that channel across all pixel locations; H and W refer to the height and width of the c-th channel, respectively.

[0121] The second step, in order to capture the spatial distribution of each location in X, is to perform spatial normalization on the entire X using normalization functions such as the normalized exponential function (Softmax function), the sigmoid function, min-max normalization, or local response normalization (LRN), thereby obtaining the spatial weight matrix. To highlight the area where the visual target appears.

[0122] After executing S302, based on the description vectors of multiple channels and the spatial weight matrix, an adaptive weight generation algorithm is used to calculate the dynamic enhancement weight for each channel. Specifically, for the spatial weight matrix obtained through global modeling, an adaptive weight generation algorithm is introduced to enhance the feature representation of the visual target region and channels in X. This algorithm uses high-order feature modeling to provide additional attention to the visual target while suppressing the influence of irrelevant background, thereby improving the visual target detection performance. This step specifically includes:

[0123] S303. Based on the multiple channel description vectors and spatial weight matrix, the comprehensive feature description of each channel is calculated by point-by-point multiplication.

[0124] Specifically, the channel statistics vector for each channel and spatial weight matrix By combining point-by-point multiplication, a comprehensive feature description of each channel is calculated. The calculation formula is as follows:

[0125]

[0126] in, This indicates point-by-point multiplication.

[0127] S304. Based on multiple comprehensive feature descriptions, the dynamic enhancement weights of each channel are calculated through a dynamic mapping function including a multilayer perceptron.

[0128] Specifically, by combining learning channels and spatial statistical information, multiple comprehensive feature descriptions are fed into a dynamic mapping function including a multilayer perceptron (MLP) to calculate the dynamic enhancement weights for each channel, which are used to weight the input first feature map X. The calculation formula is as follows:

[0129]

[0130] in, This refers to the dynamic enhancement weight of each channel.

[0131] After S304 is executed, continue with S305.

[0132] S305. Based on multiple dynamic enhancement weights and the first feature image, the second feature image corresponding to the target image is obtained by multiplying them point by point.

[0133] Specifically, using dynamic boosting weights The first feature map X is subjected to channel enhancement and spatial enhancement processing. The output target image corresponding to the second feature image retains both global spatial information and local salient features. Its calculation formula is as follows:

[0134]

[0135] in, It refers to the second feature image corresponding to the target image.

[0136] S306. Compress the number of channels in the second feature image corresponding to the target image to obtain a channel-compressed feature image.

[0137] Specifically, visual targets typically occupy a small spatial area and may exhibit different characteristics at multiple scales. To enable the target recognition model to perceive visual target features simultaneously at multiple scales, it is first necessary to reduce the size of the second feature image through compression. This increases the number of channels, thereby improving computational efficiency while preserving the most representative visual target features. For the second feature image... The number of channels is compressed to obtain a channel compressed feature image. The calculation formula is as follows:

[0138]

[0139] in, This refers to the convolution kernel; for example, using a 1×1 convolution operation for compression. This refers to the activation function used being a leaky modified linear unit function.

[0140] In other embodiments, the activation function may be a modified linear unit function (ReLU), a parametric modified linear unit function (PReLU), or a softmax function, etc.

[0141] S307. Input the channel compressed feature image into the first convolutional module, which includes multiple first convolutional layers, to obtain the scale feature image output by each first convolutional layer. Then, perform receptive field expansion processing on the channel compressed feature image through dilated convolution to obtain the receptive field expanded feature image.

[0142] Specifically, for channel compression feature images The input is a first convolutional module that includes multiple first convolutional layers, for example, a first convolutional module that includes two first convolutional layers, and the output is a channel-compressed feature image. The formula for calculating scale feature images at different scales is as follows:

[0143]

[0144]

[0145] in, and These refer to the two first convolutional layers. and These refer to the scale feature images output by each first convolutional layer.

[0146] In visual target detection tasks, expanding the receptive field is crucial for capturing contextual information at long distances. To effectively expand the receptive field, dilated convolution can be employed. Dilated convolution compresses the feature image across channels. Expanding the receptive field allows for the extension of the convolutional kernel's receptive range without increasing computational cost. This step enhances the target recognition model's ability to perceive global contextual information, thereby improving its detection accuracy and robustness in complex scenes. The formula for dilated convolution is:

[0147]

[0148] in, This refers to the receptive field extension feature image. This refers to the expansion rate, which is set to 3, indicating the degree of kernel expansion. DilatedConv refers to dilated convolution operation.

[0149] After executing S307, the feature images at multiple scales and the receptive field expansion feature image are interacted to obtain the interactive information feature image. Specifically, by interacting with the feature images at multiple scales and the receptive field expansion feature image, it is ensured that feature information can flow between different feature branches, thereby avoiding feature information loss or conflict. This step specifically includes:

[0150] S308. Based on multiple scale feature images and receptive field expansion feature images, the intermediate feature image of interactive information is obtained by adding elements one by one.

[0151] Specifically, the features of multi-scale branches and and features from the receptive field enhancement branch By adding elements one by one, the intermediate feature image of the interactive information is calculated using the following formula:

[0152]

[0153] in, It refers to the intermediate feature image of interactive information.

[0154] S309. Input the intermediate feature image of the interaction information into the second convolutional module, which includes multiple second convolutional layers, to obtain the interaction information feature image output by the second convolutional module.

[0155] Specifically, for the intermediate feature images of the obtained interaction information The data is then fed into a second convolutional module, which includes multiple second convolutional layers, for further interaction feature processing. The resulting interaction information feature image is calculated using the following formula:

[0156]

[0157] in, and These refer to the two second convolutional layers. It refers to the interactive information feature image.

[0158] After executing S309, a local region response dynamic enhancement process is applied to the interactive information feature image using a local region attention mechanism, resulting in a locally enhanced feature image. Specifically, This includes fused information from convolutions of different scales and with dilated convolutions. Since visual targets are easily overlooked or falsely detected in complex backgrounds, special attention needs to be paid to the details and accuracy of the local feature regions where the visual target is located. In this embodiment, a local region attention mechanism is used to dynamically enhance the response of local feature regions. This step specifically includes:

[0159] S310. By selecting regions of interest, the local feature region where the visual target is located is extracted from the interactive information feature image.

[0160] Specifically, this involves selecting regions of interest (ROIs) from interactive information feature images. The local feature region where the visual target is located is extracted, and its calculation formula is as follows:

[0161]

[0162] Extract refers to... Extracting local feature regions In the operation, ROI_coordinates refers to the coordinates of the region of interest.

[0163] S311. The local feature region is optimized by the third convolutional layer to obtain the locally enhanced feature image.

[0164] Specifically, for the extracted local feature regions The local features are further optimized through a third convolutional layer to highlight the details of the visual target. The calculation formula is as follows:

[0165]

[0166] in, It refers to the local enhancement feature image, and Conv5 refers to the third convolutional layer.

[0167] After executing S311, adaptive feature selection dynamically selects and enhances multiple channels of the local enhanced feature image to obtain the third feature image corresponding to the target image. Specifically, the above steps have completed feature fusion, but the resulting local enhanced feature image may still contain feature redundancy. This step uses adaptive feature selection to dynamically select and retain the most relevant features, avoiding interference from redundant features on the visual target features. This step specifically includes:

[0168] S312. The importance weight of the channel compressed feature image in each channel is calculated through global pooling operation;

[0169] Specifically, the formula for calculating importance weight is:

[0170]

[0171] in, It refers to the sigmoid activation function, and GlobalPool refers to the global pooling operation. This refers to the importance weight calculated for the c-th channel of the local feature region. It is positively correlated with the recognition performance of visual targets.

[0172] S313. Based on multiple importance weights and local enhancement feature images, the third feature image corresponding to the target image is obtained by weighted averaging.

[0173] Specifically, the importance weight of each channel is obtained. Then, different channels are dynamically selected and enhanced using a weighted average method, the calculation formula of which is:

[0174]

[0175] in, It refers to the c-th channel of the locally enhanced feature image. It refers to the third feature image corresponding to the target image.

[0176] After S313 is executed, continue with S314.

[0177] S314. Obtain the target dataset based on the third feature image corresponding to each first target image, and train the target recognition model based on the target dataset.

[0178] In one possible design, S314 trains a target recognition model based on the target dataset, including:

[0179] S3141. Divide the target dataset into a model training dataset and a model validation dataset.

[0180] For example, the target dataset can be divided into a model training dataset and a model validation dataset in a 7:3 ratio.

[0181] S3142. Train the target recognition model based on the model training dataset.

[0182] S3143. Verify the target recognition model based on the model validation dataset.

[0183] Figure 4 This is a schematic flowchart illustrating the vibration monitoring method provided in an embodiment of this application. Figure 4 As shown in the figure, this application also provides a vibration monitoring method. In this application embodiment, the executing entity can be a vibration monitoring device, which can be located in an electronic device. The vibration monitoring method provided by this application embodiment includes the following steps:

[0184] S401. Periodically photograph the visual target installed on the structure to obtain the second target image captured each time.

[0185] S402. Using a pre-trained target recognition model, identify the position of the visual target from each second target image.

[0186] The target recognition model is through Figure 2 or Figure 3 The model is trained using the method described in the embodiment. Its implementation principle and technical effects are similar to... Figure 2 or Figure 3 The methods shown in the embodiments are similar, and will not be described again in the embodiments of this application.

[0187] S403. Based on the position of the visual target in each second target image, the first monitoring result of the vibration change of the structure is obtained.

[0188] In one possible design, S403 includes:

[0189] S4031. Based on the position of the visual target in each second target image, calculate the three-dimensional coordinates of the visual target in the corresponding second target image through the pre-stored first mapping relationship.

[0190] Specifically, the first mapping relationship is a predefined mathematical model used to indicate the mapping relationship between the position of the visual target and its three-dimensional coordinates. The first mapping relationship can be established in various ways, such as through manual measurement of a known visual target, stereo vision technology, or structured light scanning.

[0191] Once the position of each visual target in each second target image is obtained, this first mapping relationship can be used to convert these positions into three-dimensional coordinates. This means that for each visual target identified in the second target image, its specific position (x, y, z) within the structure can be found.

[0192] S4032. Based on the three-dimensional coordinates of the visual target in each second target image, the first monitoring result of the vibration change of the structure is obtained.

[0193] Specifically, based on a dataset of three-dimensional coordinates of a visual target at different time points or under different conditions, the dynamic characteristics of a structure can be analyzed. The key here is to compare the three-dimensional coordinate values ​​at different times to observe and quantify any displacement or deformation of the structure. The analysis results of dynamic characteristics include, but are not limited to, parameters such as frequency response, amplitude variation, and phase difference, which are all important indicators for understanding the vibration behavior of a structure.

[0194] Ultimately, based on the above analysis results, the first monitoring results regarding the vibration changes of the structure can be obtained. These results not only reveal the current vibration state of the structure but can also serve as the basis for subsequent, more in-depth analyses such as health monitoring and damage detection.

[0195] Furthermore, after executing S4032, the amplitude curve of the structure is plotted based on the first monitoring results of vibration changes.

[0196] The technical effect of this application embodiment is: based on multiple second target images obtained by periodically photographing the structure, the first monitoring result of the vibration change of the structure is obtained, realizing the simultaneous measurement of all targets installed on the structure, thereby realizing the real-time monitoring of the subtle vibration changes of the structure; through the target recognition model, the recognition accuracy of small targets such as visual targets on the surface of the structure is enhanced.

[0197] In one possible design, the vibration monitoring equipment includes a total station and a machine vision instrument mounted on the total station; wherein the machine vision instrument is fixed in position relative to the laser rangefinder of the total station.

[0198] Before executing S401, the vibration monitoring method also includes:

[0199] The laser rangefinder is calibrated using a reference target until the laser emitted by the laser rangefinder is projected onto the reference target; when the laser rangefinder calibration is complete, the second target image displays a visual target.

[0200] Specifically, before periodically photographing the visual target mounted on the structure, the machine vision instrument is visually verified and calibrated using a reference target to ensure the accuracy of the second target image. The calibration process of the machine vision instrument includes:

[0201] First, install the reference target. The reference target, which can be a prism, can be installed on a fixed platform near the structure to establish a reference coordinate system and determine the coordinates of the vibration monitoring equipment.

[0202] Secondly, the laser rangefinder is calibrated. Specifically, the laser rangefinder is controlled to emit a laser, and it is observed whether the laser is projected onto the reference target.

[0203] When the laser is projected onto the reference target, it is confirmed that the laser rangefinder has been calibrated. Since the machine vision device is fixed in position relative to the laser rangefinder, it can also be confirmed that the machine vision device has been calibrated. The machine vision device is then pulled back, and it is controlled to periodically photograph the visual target mounted on the structure.

[0204] If the laser does not reach the reference target, it is determined that the laser rangefinder has not been calibrated. Then, the algorithm is used to adjust the angle of the laser rangefinder within a certain area and observe again until the laser emitted by the laser rangefinder reaches the reference target.

[0205] The technical effect of this application embodiment is that by calibrating the laser rangefinder with a reference target, the accuracy of the second target image is improved, thereby improving the accuracy of the first monitoring result of the vibration change of the structure.

[0206] In one possible design, a laser target is also installed on the structure, in which case the vibration monitoring method would also include:

[0207] The laser target is measured using a laser rangefinder, and the three-dimensional coordinates of the laser target are obtained for each measurement.

[0208] The second monitoring result of the vibration change of the structure is obtained based on the three-dimensional coordinates of the laser target measured each time.

[0209] Specifically, the installation methods of visual targets and laser targets are different, and the roles of the first monitoring result and the second monitoring result of vibration changes are also different.

[0210] Visual targets and laser targets can be installed on different parts of a structure, and both are used to monitor different frequencies of the structure. For example, the structure includes bridges and embankments; bridges require high-frequency monitoring, while embankments only require low-frequency monitoring. Based on this, if all visual targets are installed on the bridge, the first monitoring result of vibration change is used to indicate the vibration change of the bridge; if all laser targets are installed on the embankment, the second monitoring result of vibration change is used to indicate the vibration change of the embankment.

[0211] The following content is based on the embodiments of this application. Figure 1 The application scenarios provided above offer specific explanations of the cases.

[0212] A reference target is installed within the visual range of the bridge monitoring points, a visual target is installed at the bridge monitoring points, and a laser target is installed at the embankment monitoring points. First, a laser rangefinder is used to measure the reference target to calibrate the laser rangefinder. Second, the vibration monitoring equipment is rotated and fixed to the bridge's viewing angle to control a machine vision device to capture images of the visual target, thus obtaining a second target image and a first vibration change monitoring result. This first vibration change monitoring result is used for high-frequency monitoring of bridge vibration changes. Finally, the vibration monitoring equipment is rotated and fixed to the embankment's measurement angle to control a laser rangefinder to measure the laser target's three-dimensional coordinates, thus obtaining a second vibration change monitoring result. This second vibration change monitoring result is used for low-frequency monitoring of embankment vibration changes.

[0213] Visual targets and laser targets can also be installed on the same parts of a structure, and both can be used to monitor the structure at different times. For example, during the day or under good lighting conditions, the first monitoring result of vibration changes obtained by photographing the visual target indicates the vibration changes of the structure; at night or under poor lighting conditions, the second monitoring result of vibration changes obtained by measuring the laser target indicates the vibration changes of the structure.

[0214] Visual targets and laser targets can also be installed on the same parts of the structure. Visual targets are used to improve the monitoring efficiency of laser targets. For example, considering that structural vibration may cause the laser rangefinder or laser target to shift, the offset of the laser target is first determined based on the first monitoring result of vibration change. Then, the laser rangefinder is controlled to measure the laser target based on this offset, thereby improving the monitoring efficiency of the second monitoring result of vibration change.

[0215] Visual targets and laser targets can also be installed on the same parts of the structure. Laser targets are used to improve the monitoring accuracy of visual targets. For example, if a single second target image cannot display the entire visual target, combining or stitching multiple second target images to display the entire visual target may reduce the monitoring accuracy of the first vibration change monitoring result. Therefore, multiple laser targets are installed on the structure according to the shooting range of the machine vision instrument. Before each shooting, the machine vision instrument adjusts its own angle according to the laser targets within the corresponding shooting range to improve the monitoring accuracy of the first vibration change monitoring result.

[0216] The technical effect of this application embodiment is that different monitoring needs under different working conditions are achieved through visual targets and laser targets.

[0217] In one possible design, the offset of the laser rangefinder or laser target can lead to a longer time for the laser to be projected onto the laser target, or even prevent the laser from being projected onto the laser target at all. At the same time, considering that the accuracy of the three-dimensional coordinates of the visual target is less than that of the three-dimensional coordinates of the laser target when monitoring over long distances, the monitoring efficiency of the laser target can be improved by using the visual target.

[0218] When the first monitoring result of vibration change indicates that the structure is undergoing vibration change, the laser target is measured using a laser rangefinder to obtain the three-dimensional coordinates of the laser target measured each time, including:

[0219] Based on the first monitoring results of vibration changes, the offset of the visual target is obtained.

[0220] Specifically, once it is determined that the structure has undergone vibration changes, these changes need to be quantified. This step involves comparing the positional differences of the visual targets at the current moment with a reference moment (which could be the moment of the initial state or the moment of the state in the previous cycle) to determine the offset of each visual target. The offset can include both the direction and magnitude of the offset.

[0221] The offset of the laser target is calculated based on the offset of the visual target through a pre-stored second mapping relationship; wherein, the second mapping relationship is used to indicate the mapping relationship between the relative positions of the visual target and the laser target.

[0222] Specifically, the second mapping relationship can be constructed based on the measurement data of the relative positions of the two targets in a static state, or it can be constructed based on the historical first and second monitoring results of vibration changes. It describes the corresponding shift of the laser target when the visual target shifts. With the shift of the visual target, the shift of the laser target can be predicted using the above-mentioned second mapping relationship.

[0223] Based on the offset of the laser target, the laser rangefinder is controlled to deflect, and the laser target is measured through the deflected laser rangefinder to obtain the three-dimensional coordinates of the laser target measured each time.

[0224] Specifically, based on the calculated offset of the laser target, the direction of the laser rangefinder is automatically adjusted to re-align it with the new laser target position. This process may involve fine-tuning the horizontal and vertical angles of the laser rangefinder to ensure that the laser is accurately projected onto the target laser.

[0225] After the orientation adjustment is completed, the laser rangefinder is controlled to measure the position of the laser target again to obtain its latest three-dimensional coordinates. These three-dimensional coordinates not only reflect the exact current position of the laser target, but also include any displacement information caused by structural vibrations.

[0226] Finally, based on the three-dimensional coordinates of the laser target obtained from multiple measurements, the dynamic characteristics of the structure were further analyzed to obtain the second monitoring result of the vibration change of the structure.

[0227] The technical effect of this application embodiment is that by combining the machine vision monitoring technology of the machine vision instrument with the laser ranging technology of the laser rangefinder, the monitoring accuracy of long-distance small targets is improved.

[0228] This application also provides a vibration monitoring device, such as... Figure 1 As shown, the vibration monitoring device includes:

[0229] The total station, and the machine vision instrument mounted on the total station; wherein the machine vision instrument is fixed in position relative to the laser rangefinder of the total station, and the machine vision instrument and the laser rangefinder can rotate synchronously.

[0230] Vibration monitoring equipment is used to perform Figure 4 The vibration monitoring method shown in the embodiment is based on the principle and technical effects of vibration monitoring. Figure 4 The method embodiments shown are similar, and will not be described again in the embodiments of this application.

[0231] In one possible design, the vibration monitoring equipment is used to photograph a visual target mounted on the structure and to measure a laser target mounted on the structure, as well as a reference target mounted near the structure.

[0232] The reference target can be installed on a fixed platform within the visual range corresponding to the visual target and the measurement range corresponding to the laser target, to facilitate the confirmation of the reference coordinate system during monitoring. If it is not possible to set it within this range due to limitations, the angle between the reference target and other targets should be minimized as much as possible. After confirming the reference coordinate system through the reference target during monitoring, the vibration monitoring equipment should be rotated to the corresponding target position and the horizontal and vertical rotation angles should be recorded to facilitate the reading of the target's three-dimensional coordinates.

[0233] The technical effect of this application embodiment is that by installing a machine vision instrument on the basis of an existing total station, the vibration monitoring equipment can simultaneously capture images of visual targets and measure laser targets.

[0234] Figure 5 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application, as shown below. Figure 5 As shown in this embodiment, the model training device can be located in an electronic device. The model training device includes:

[0235] The feature extraction module 510 is used to acquire multiple first target images and perform feature extraction processing on each first target image through a preset convolutional neural network to obtain a first feature image corresponding to each first target image; wherein, the first target image displays a visual target installed on a structure;

[0236] The feature processing module 520 is used to perform channel enhancement processing and spatial enhancement processing on the first feature image corresponding to each first target image to obtain the second feature image corresponding to each first target image, and to perform detail extraction processing on the second feature image corresponding to each first target image to obtain the third feature image corresponding to each first target image.

[0237] The model training module 530 is used to obtain a target dataset based on the third feature image corresponding to each first target image, and to train a target recognition model based on the target dataset; wherein, the target recognition model is used to identify the position of the visual target from the second target image captured by the vibration monitoring device; the position of the visual target in the second target image is used to indicate the vibration change of the structure.

[0238] The model training device provided in this application embodiment can perform... Figure 2 The technical solution of the method embodiment shown has the same implementation principle and technical effect as... Figure 2 The method embodiments shown are similar, and will not be described again in the embodiments of this application.

[0239] Meanwhile, the model training device provided in this application embodiment is further refined based on the model training device provided in the previous application embodiment.

[0240] In one possible design, the target image is any one of a plurality of first target images; for the target image, the feature processing module 520 includes:

[0241] The vector matrix calculation module is used to obtain the channel description vector of each channel based on the pixel mean of the first feature image corresponding to the target image in multiple channels, and to perform spatial normalization processing on the first feature image corresponding to the target image to obtain the spatial weight matrix.

[0242] The enhanced weight calculation module is used to calculate the dynamic enhanced weight of each channel based on multiple channel description vectors and spatial weight matrices using an adaptive weight generation algorithm.

[0243] The second feature calculation module is used to calculate the second feature image corresponding to the target image by multiplying multiple dynamic enhancement weights and the first feature image point by point.

[0244] In one possible design, the weight calculation module is enhanced by including:

[0245] The feature description calculation module is used to calculate the comprehensive feature description of each channel by point-by-point multiplication based on multiple channel description vectors and spatial weight matrix.

[0246] The dynamic mapping calculation module is used to calculate the dynamic enhancement weight of each channel based on multiple comprehensive feature descriptions through a dynamic mapping function including a multilayer perceptron.

[0247] In one possible design, the target image is any one of a plurality of first target images; for the target image, the feature processing module 520 further includes:

[0248] The channel compression module is used to compress the number of channels in the second feature image corresponding to the target image to obtain a channel-compressed feature image.

[0249] The receptive field expansion module is used to input the channel compressed feature image into the first convolutional module, which includes multiple first convolutional layers, to obtain the scale feature image output by each first convolutional layer, and to perform receptive field expansion processing on the channel compressed feature image through dilated convolution to obtain the receptive field expanded feature image.

[0250] The interaction information determination module is used to interact with multiple scale feature images and receptive field extension feature images to obtain interaction information feature images;

[0251] The feature enhancement module is used to perform local region response dynamic enhancement processing on the interactive information feature image through a local region attention mechanism to obtain a locally enhanced feature image;

[0252] The third feature calculation module is used to dynamically select and enhance multiple channels of the local enhanced feature image through adaptive feature selection to obtain the third feature image corresponding to the target image.

[0253] In one possible design, the interaction information determination module includes:

[0254] The first module for determining interactive information is used to calculate the intermediate feature image of interactive information by adding elements one by one based on feature images at multiple scales and feature images of receptive field expansion.

[0255] The interaction information determination second module is used to input the intermediate feature image of the interaction information into the second convolutional module, which includes multiple second convolutional layers, to obtain the interaction information feature image output by the second convolutional module.

[0256] In one possible design, the feature enhancement module includes:

[0257] The first feature enhancement module is used to extract the local feature region where the visual target is located from the interactive information feature image through region of interest selection;

[0258] The second feature enhancement module is used to optimize local features in local feature regions through the third convolutional layer to obtain a locally enhanced feature image.

[0259] In one possible design, the third feature calculation module includes:

[0260] The fourth feature calculation module is used to calculate the importance weight of the channel compressed feature image in each channel through global pooling operation;

[0261] The fifth feature calculation module is used to calculate the third feature image corresponding to the target image by weighted averaging based on multiple importance weights and local enhanced feature images.

[0262] In one possible design, the model training module 530 includes:

[0263] The data partitioning module is used to divide the target dataset into a model training dataset and a model validation dataset;

[0264] The dataset training module is used to train the target recognition model based on the model training dataset.

[0265] The dataset validation module is used to validate the target recognition model based on the model validation dataset.

[0266] The model training apparatus provided in this application embodiment can execute... Figure 2 or Figure 3 The technical solution of the method embodiment shown has the same implementation principle and technical effect as... Figure 2 or Figure 3 The method embodiments shown are similar, and will not be described again in the embodiments of this application.

[0267] Figure 6 This is a schematic diagram of the structure of the vibration monitoring device provided in the embodiments of this application, as shown below. Figure 6 As shown in this embodiment, the vibration monitoring device can be located in an electronic device. The vibration monitoring device includes:

[0268] The image capturing module 610 is used to periodically capture images of visual targets installed on the structure, and obtain a second target image captured each time.

[0269] The position recognition module 620 is used to identify the position of the visual target from each second target image using a pre-trained target recognition model; wherein, the target recognition model is obtained through... Figure 2 or Figure 3 The model is trained using the method shown in the embodiment.

[0270] The first monitoring module 630 is used to obtain the first monitoring result of the vibration change of the structure based on the position of the visual target in each second target image.

[0271] The vibration monitoring device provided in this application embodiment can perform... Figure 4 The technical solution of the method embodiment shown has the same implementation principle and technical effect as... Figure 4 The method embodiments shown are similar, and will not be described again in the embodiments of this application.

[0272] Meanwhile, the vibration monitoring device provided in this application embodiment is a further refinement based on the vibration monitoring device provided in the previous application embodiment.

[0273] In one possible design, the first monitoring module 630 includes:

[0274] The first coordinate calculation module is used to calculate the three-dimensional coordinates of the visual target in each second target image based on the position of the visual target in each second target image and through a pre-stored first mapping relationship; wherein, the first mapping relationship is used to indicate the mapping relationship between the position of the visual target and the three-dimensional coordinates;

[0275] The second monitoring module is used to obtain the first monitoring result of the vibration change of the structure based on the three-dimensional coordinates of the visual target in each second target image.

[0276] In one possible design, the vibration monitoring equipment includes a total station and a machine vision device mounted on the total station; wherein the machine vision device is fixed in position relative to the laser rangefinder of the total station.

[0277] The vibration monitoring device also includes:

[0278] The instrument calibration module is used to calibrate the laser rangefinder using a reference target until the laser emitted by the laser rangefinder is projected onto the reference target; when the laser rangefinder calibration is completed, the second target image displays a visual target.

[0279] In one possible design, a laser target is also mounted on the structure, in which case the vibration monitoring device would also include:

[0280] The second coordinate calculation module is used to measure the laser target using a laser rangefinder and obtain the three-dimensional coordinates of the laser target measured each time.

[0281] The third monitoring module is used to obtain the second monitoring result of the vibration change of the structure based on the three-dimensional coordinates of the laser target measured each time.

[0282] In one possible design, the accuracy of the three-dimensional coordinates of the visual target is less than that of the three-dimensional coordinates of the laser target.

[0283] When the first monitoring result of vibration change indicates that the structure has undergone vibration change, the second coordinate calculation module includes:

[0284] The offset calculation module is used to obtain the offset of the visual target based on the first monitoring result of vibration change;

[0285] The offset mapping module is used to calculate the offset of the laser target based on the offset of the visual target through a pre-stored second mapping relationship; wherein, the second mapping relationship is used to indicate the mapping relationship between the relative positions of the visual target and the laser target.

[0286] The fourth coordinate calculation module is used to control the deflection of the laser rangefinder based on the offset of the laser target, and then measure the laser target through the deflected laser rangefinder to obtain the three-dimensional coordinates of the laser target measured each time.

[0287] The vibration monitoring device provided in this application embodiment can perform... Figure 4 The technical solution of the method embodiment shown has the same implementation principle and technical effect as... Figure 4 The method embodiments shown are similar, and will not be described again in the embodiments of this application.

[0288] This application also provides an electronic device. Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device includes at least one processor 710 and a memory 720. The electronic device also includes a communication component 730. The processor 710, memory 720, and communication component 730 are connected via a bus 740.

[0289] In the specific implementation process, at least one processor 710 executes computer execution instructions stored in memory 720, so that at least one processor 710 is used to implement the model training method and vibration monitoring method of the above embodiments.

[0290] The specific implementation process of processor 710 can be found in the above method embodiments, and its implementation principle and technical effect are similar. The embodiments of this application will not be repeated here.

[0291] In the above embodiments, it should be understood that the processor 710 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0292] The memory 720 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage.

[0293] Bus 740 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 740 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 740 in the accompanying drawings of this application is not limited to only one bus or one type of bus.

[0294] The above description addresses the functions implemented by electronic devices and main control devices, and introduces the solutions provided in the embodiments of this application. It is understood that, in order to achieve the above functions, the electronic device or main control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments disclosed in this application, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of this application.

[0295] This application also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, these instructions are used to implement the model training method and vibration monitoring method described above. In the specific implementation of the aforementioned model training method and vibration monitoring method, each module can be implemented as a processor.

[0296] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0297] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, the processor and the readable storage medium can exist as discrete components in an electronic device or a host device.

[0298] This application also provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the model training method and vibration monitoring method of the above embodiments.

[0299] The computer program is stored in a readable storage medium, and at least one processor can read the computer program from the readable storage medium and execute the computer program to perform the scheme provided in any of the above embodiments.

[0300] Those skilled in the art will understand that all or part of the steps in the above-described embodiments can be implemented using hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps included in the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0301] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A model training method, characterized in that, The method includes: Multiple first target images are acquired, and feature extraction processing is performed on each first target image through a preset convolutional neural network to obtain a first feature image corresponding to each first target image; wherein, the first target image displays a visual target installed on a structure; Channel enhancement and spatial enhancement processing are performed on the first feature image corresponding to each first target image to obtain a second feature image corresponding to each first target image. Then, detail extraction processing is performed on the second feature image corresponding to each first target image to obtain a third feature image corresponding to each first target image. The third feature image is used to enhance the display effect of the visual target and capture its detailed information to distinguish it from the background in the first target image. A target dataset is obtained based on the third feature image corresponding to each first target image, and a target recognition model is trained based on the target dataset; wherein, the target recognition model is used to identify the position of the visual target from the second target image captured by the vibration monitoring device; the position of the visual target in the second target image is used to indicate the vibration change of the structure.

2. The model training method according to claim 1, characterized in that, The target image is any one of the plurality of first target images; for each target image, performing channel enhancement and spatial enhancement processing on the first feature image corresponding to each first target image to obtain a second feature image corresponding to each first target image includes: Based on the pixel mean values ​​of the first feature image corresponding to the target image in multiple channels, a channel description vector for each channel is obtained, and the first feature image corresponding to the target image is spatially normalized to obtain a spatial weight matrix. Based on the multiple channel description vectors and the spatial weight matrix, the dynamic enhancement weight of each channel is calculated using an adaptive weight generation algorithm. Based on the multiple dynamic enhancement weights and the first feature image, the second feature image corresponding to the target image is obtained by multiplying them point by point.

3. The model training method according to claim 2, characterized in that, The step of calculating the dynamic enhancement weight of each channel based on the multiple channel description vectors and the spatial weight matrix using an adaptive weight generation algorithm includes: Based on the multiple channel description vectors and the spatial weight matrix, the comprehensive feature description of each channel is calculated by point-by-point multiplication; Based on the multiple comprehensive feature descriptions, the dynamic enhancement weights of each channel are calculated using a dynamic mapping function including a multilayer perceptron.

4. The model training method according to any one of claims 1 to 3, characterized in that, The target image is any one of the plurality of first target images; for each target image, the step of performing detail extraction processing on the second feature image corresponding to each first target image to obtain a third feature image corresponding to each first target image includes: The number of channels in the second feature image corresponding to the target image is compressed to obtain a channel-compressed feature image; The channel compressed feature image is input into a first convolutional module including multiple first convolutional layers to obtain a scale feature image output by each first convolutional layer. The receptive field of the channel compressed feature image is then expanded by dilated convolution to obtain a receptive field expanded feature image. The interaction between multiple scale feature images and the receptive field expansion feature images is used to obtain an interaction information feature image; The interactive information feature image is dynamically enhanced by local region response through a local region attention mechanism to obtain a locally enhanced feature image. By dynamically selecting and enhancing multiple channels of the local enhanced feature image through adaptive feature selection, a third feature image corresponding to the target image is obtained.

5. The model training method according to claim 4, characterized in that, The step of interacting with multiple scale feature images and receptive field expansion feature images to obtain an interaction information feature image includes: Based on multiple scale feature images and receptive field expansion feature images, an intermediate feature image of interactive information is obtained by adding elements one by one; The intermediate feature image of the interactive information is input into a second convolutional module that includes multiple second convolutional layers to obtain the interactive information feature image output by the second convolutional module.

6. The model training method according to claim 4, characterized in that, The step of performing local region response dynamic enhancement processing on the interactive information feature image through a local region attention mechanism to obtain a locally enhanced feature image includes: By selecting regions of interest, the local feature region where the visual target is located is extracted from the interactive information feature image; The local feature image is obtained by performing local feature optimization on the local feature region through the third convolutional layer.

7. The model training method according to claim 4, characterized in that, The step of dynamically selecting and enhancing multiple channels of the local enhanced feature image through adaptive feature selection to obtain the third feature image corresponding to the target image includes: The importance weight of the channel compressed feature image in each channel is calculated through a global pooling operation. Based on the multiple importance weights and the local enhanced feature images, the third feature image corresponding to the target image is obtained by weighted averaging.

8. The model training method according to claim 1, characterized in that, The target recognition model trained based on the target dataset includes: The target dataset is divided into a model training dataset and a model validation dataset; The target recognition model is trained based on the model training dataset. The target recognition model is validated using the model validation dataset.

9. A vibration monitoring method, characterized in that, The method is applied to a vibration monitoring device, and the method includes: Periodically photograph the visual target mounted on the structure to obtain a second target image each time; The location of the visual target is identified from each of the second target images using a pre-trained target recognition model; wherein the target recognition model is trained using the model training method as described in any one of claims 1 to 8. Based on the position of the visual target in each of the second target images, a first monitoring result of the vibration change of the structure is obtained.

10. The vibration monitoring method according to claim 9, characterized in that, The step of obtaining the first monitoring result of the vibration change of the structure based on the position of the visual target in each of the second target images includes: Based on the position of the visual target in each second target image, the three-dimensional coordinates of the visual target in the corresponding second target image are calculated through a pre-stored first mapping relationship; wherein, the first mapping relationship is used to indicate the mapping relationship between the position of the visual target and the three-dimensional coordinates; Based on the three-dimensional coordinates of the visual target in each of the second target images, a first monitoring result of the vibration change of the structure is obtained.

11. The vibration monitoring method according to claim 10, characterized in that, The vibration monitoring equipment includes a total station and a machine vision instrument mounted on the total station; wherein the position of the machine vision instrument relative to the laser rangefinder of the total station is fixed. Before periodically photographing the visual target mounted on the structure to obtain a second target image each time, the method further includes: The laser rangefinder is calibrated using a reference target until the laser emitted by the laser rangefinder is projected onto the reference target; wherein, when the laser rangefinder calibration is completed, the second target image displays the visual target.

12. The vibration monitoring method according to claim 11, characterized in that, The structure is also equipped with a laser target, and the method further includes: The laser target is measured using the laser rangefinder, and the three-dimensional coordinates of the laser target are obtained for each measurement. Based on the three-dimensional coordinates of the laser target measured each time, a second monitoring result of the vibration change of the structure is obtained.

13. The vibration monitoring method according to claim 12, characterized in that, The accuracy of the three-dimensional coordinates of the visual target is less than the accuracy of the three-dimensional coordinates of the laser target; When the first monitoring result of the vibration change indicates that the structure has undergone vibration change, the step of measuring the laser target with the laser rangefinder to obtain the three-dimensional coordinates of the laser target for each measurement includes: Based on the first monitoring result of the vibration change, the offset of the visual target is obtained; The offset of the laser target is calculated based on the offset of the visual target using a pre-stored second mapping relationship; wherein the second mapping relationship is used to indicate the mapping relationship between the relative positions of the visual target and the laser target. Based on the offset of the laser target, the laser rangefinder is controlled to deflect, and the laser target is measured by the deflected laser rangefinder to obtain the three-dimensional coordinates of the laser target measured each time.

14. A model training device, characterized in that, The device includes: The feature extraction module is used to acquire multiple first target images and perform feature extraction processing on each first target image through a preset convolutional neural network to obtain a first feature image corresponding to each first target image; wherein, the first target image displays a visual target installed on a structure; The feature processing module is used to perform channel enhancement processing and spatial enhancement processing on the first feature image corresponding to each first target image to obtain a second feature image corresponding to each first target image, and to perform detail extraction processing on the second feature image corresponding to each first target image to obtain a third feature image corresponding to each first target image; wherein, the third feature image is used to enhance the display effect of the visual target and capture its detailed information to distinguish it from the background in the first target image; The model training module is used to obtain a target dataset based on the third feature image corresponding to each first target image, and to train a target recognition model based on the target dataset; wherein, the target recognition model is used to identify the position of the visual target from the second target image captured by the vibration monitoring device; the position of the visual target in the second target image is used to indicate the vibration change of the structure.

15. A vibration monitoring device, characterized in that, The device is located within the vibration monitoring equipment, and the device includes: The image capturing module is used to periodically capture images of visual targets mounted on the structure, obtaining a second target image each time. A location recognition module is used to identify the location of the visual target from each of the second target images using a pre-trained target recognition model; wherein the target recognition model is trained using the model training method as described in any one of claims 1 to 8; The first monitoring module is used to obtain a first monitoring result of the vibration change of the structure based on the position of the visual target in each of the second target images.

16. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; When the processor executes the computer execution instructions stored in the memory, it is used to implement the model training method as described in any one of claims 1 to 8, or the vibration monitoring method as described in any one of claims 9 to 13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the model training method as described in any one of claims 1 to 8, or the vibration monitoring method as described in any one of claims 9 to 13.

18. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, is used to implement the model training method as described in any one of claims 1 to 8, or the vibration monitoring method as described in any one of claims 9 to 13.

Images