Adversarial generation data enhancement method and system for road patrol maintenance image

Abstract

The invention relates to the technical field of image enhancement, and discloses a road patrol maintenance image confrontation generation data enhancement method and system, and the method comprises the steps: collecting the six-axis vibration signals of maintenance equipment in real time to construct a vibration frequency domain fingerprint spectrum, collecting and processing point cloud data and a visual image, constructing a multi-modal fusion feature vector, and carrying out the recognition of the multi-modal fusion feature vector. And constructing and training an adversarial generative network, and finally generating an enhanced road patrol maintenance image. The problem that traditional image enhancement only depends on visual modal data is effectively solved, the problems of equipment vibration and spatial-temporal dislocation of point cloud and visual images in maintenance operation can be simulated more truly through a multi-modal collaborative enhancement technology, the quality and analysis accuracy of road patrol maintenance images are improved, and a reliable basis is provided for road maintenance decisions.

Description

Technical Field

[0001] The present invention relates to the field of image enhancement technology, and more specifically, to an adversarial generation data enhancement method and system for highway inspection and maintenance images. Background Art

[0002] In highway inspection and maintenance, the quality of image data plays a key role in accurately assessing highway conditions. With the development of technology, image enhancement technology has been widely used in the field of highway inspection and maintenance, but traditional image enhancement methods have many limitations.

[0003] The Chinese patent with the authorization announcement number CN107153928B discloses a visualized highway maintenance decision-making system, which improves maintenance efficiency by collecting highway maintenance information and using indicator evaluation and visualization methods to assist maintenance decisions. However, the system mainly focuses on the visualization and indicator analysis of maintenance decisions, and does not involve image enhancement technology. It cannot solve the poor quality of highway inspection and maintenance images caused by equipment vibration, spatiotemporal misalignment between the LiDAR point cloud and the visual image, and other problems. In actual highway inspections, equipment vibration will cause the captured images to be blurred, and the spatiotemporal misalignment between the LiDAR point cloud and the visual image will lead to inaccurate information matching. These problems will seriously affect the judgment of highway diseases and facility conditions, and the patent does not provide solutions to these problems.

[0004] The patent application with publication number CN118333608A discloses a highway inspection and maintenance system, which mainly focuses on the processing of maintenance status data, model building and optimization of watering operations to improve the uniformity and utilization rate of watering. However, this prior art also does not take into account the impact of equipment vibration, laser radar point cloud and visual image time and space misalignment on image quality in highway inspection and maintenance images. Due to the lack of effective processing of image data, it is difficult to make accurate judgments based on high-quality images in analyzing highway pavement conditions, road facility integrity, etc., which is not conducive to timely discovery and treatment of highway diseases and facility damage problems.

[0005] Existing technologies cannot effectively simulate equipment vibration during maintenance operations, and the spatiotemporal misalignment of lidar point clouds and visual images in the processing of highway inspection and maintenance images. It is difficult to provide high-quality highway inspection and maintenance images, which affects the accurate assessment of highway conditions and the scientific nature of maintenance decisions. Summary of the invention

[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides an adversarial generation data enhancement method and system for highway inspection and maintenance images, aiming to innovatively integrate inertial navigation data, laser point clouds and visual images. By constructing a spatiotemporal aligned multimodal generation network, the problem that traditional methods cannot simulate equipment vibration and the spatiotemporal misalignment of point clouds and visual images is effectively solved. Through multimodal collaborative enhancement technology, a spatiotemporal aligned multimodal generation network is constructed to improve the quality of highway inspection and maintenance images, providing a more reliable basis for highway maintenance decisions.

[0007] The present invention is mainly used in highway inspection and maintenance work scenarios. In daily highway inspections, maintenance personnel use vehicles equipped with relevant equipment to inspect highways. During the inspection process, the equipment will collect a large amount of highway image data, but due to factors such as vibration during vehicle driving, differences in time and space between lidar point clouds and visual image acquisition, the collected images often have problems such as blur and inaccurate information. The method and system of the present invention can process these images, enhance image quality, and help maintenance personnel more clearly observe road surface diseases, road facility damage, etc., so as to more efficiently formulate maintenance plans and ensure the safety and smooth flow of highways.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] The adversarial generation data enhancement method for highway inspection and maintenance images includes:

[0010] Collect the six-axis vibration signals of the maintenance equipment in real time and construct a vibration frequency domain fingerprint. Perform frequency domain decomposition on the collected six-axis vibration signals based on a sliding time window, extract the characteristic frequency bands related to the movement speed of the maintenance equipment, and generate a vibration-speed mapping relationship matrix. Dynamically correct the vibration-speed mapping relationship matrix and output a calibrated vibration frequency domain fingerprint.

[0011] Collect point cloud data from the highway inspection area, perform entropy value weighted encoding on the point cloud data, and generate point cloud spatiotemporal encoding; obtain visual images of the highway inspection area, perform cross-modal correlation analysis on the six-axis vibration signal and the visual image, and obtain a weighted visual feature map; construct a multimodal fusion feature vector based on the calibrated vibration frequency domain fingerprint map, point cloud spatiotemporal encoding, and weighted visual feature map;

[0012] According to the multimodal fusion feature vector, a generative adversarial network including a generator and a discriminator is constructed and trained; a new multimodal fusion feature vector is obtained, and an enhanced highway inspection and maintenance image is generated according to the new multimodal fusion feature vector and the trained generative adversarial network.

[0013] Furthermore, the real-time acquisition of the six-axis vibration signal of the maintenance equipment includes: installing an inertial measurement unit on the highway inspection and maintenance equipment, and collecting the six-axis vibration signal of the maintenance equipment in real time during driving according to the installed inertial measurement unit; the six-axis vibration signal includes acceleration signals a in three directions of X, Y, and Z axes. x (t), a y (t), a z (t) and angular velocity signals ω in the three directions of X, Y, and Z axes x (t),ω y (t),ω z (t).

[0014] Furthermore, the construction of the vibration frequency domain fingerprint spectrum includes:

[0015] The collected six-axis vibration signals are converted into frequency domain to obtain the main frequency band energy distribution and harmonic distortion rate; the relationship between the acceleration signals in the three directions of X, Y, and Z axes and the angular velocity signals in the three directions of X, Y, and Z axes are analyzed to obtain the inter-axis coupling coefficient; the main frequency band energy distribution, harmonic distortion rate, and inter-axis coupling coefficient are integrated to construct a vibration frequency domain fingerprint.

[0016] Further, obtaining the inter-axis coupling coefficient includes:

[0017] The acceleration signal a in the three directions of X, Y and Z axis x (t), a y (t), a z (t) and angular velocity signals ω in the three directions of X, Y, and Z axes x (t),ω y (t),ω z (t) Perform timestamp alignment;

[0018] According to the aligned a x (t), a y (t), a z (t),ω x (t),ω y (t),ω z (t), construct the multivariate signal space matrix;

[0019] According to the multivariate signal space matrix, the correlation coefficient matrix R and the nonlinear correlation matrix N are obtained;

[0020] The correlation coefficient matrix and the nonlinear correlation matrix The fusion is performed to obtain a comprehensive correlation matrix C, from which the inter-axis coupling coefficient is extracted.

[0021] Furthermore, constructing a multivariate signal space matrix includes:

[0022] After alignment, a x (t), a y (t), a z (t),ω x (t),ω y (t),ω z (t) is combined into a six-dimensional signal vector ;

[0023] in a period of time Inside, collect The six-dimensional signal vector at each moment constructs a multivariate signal space matrix ,in, is the six-dimensional signal vector at the nth moment.

[0024] Further, obtaining the correlation coefficient matrix R includes:

[0025] For the multivariate signal space matrix Any two components in and ,calculate and The correlation coefficients between all different axes give a The correlation coefficient matrix , the correlation coefficient matrix Elements in Indicates ''Axis and ''Correlation between axis signals; where, is the six-dimensional signal vector at the i-th moment, is the six-dimensional signal vector at the jth moment, 1≤i≤n, 1≤j≤n, .

[0026] Furthermore, obtaining the nonlinear correlation matrix N includes: calculating and The mutual information value of , according to the mutual information value Construct the nonlinear correlation matrix N.

[0027] Further, generating a vibration-velocity mapping relationship matrix includes:

[0028] The six-axis vibration signal of each characteristic frequency band is represented in time-frequency form, and the amplitude statistics of each characteristic frequency band are extracted; the movement speed of the maintenance equipment is obtained, and the amplitude statistics of each characteristic frequency band are associated with the movement speed of the maintenance equipment to establish the vibration-speed mapping relationship matrix M vs ; Among them, the matrix element M vs (f i',v j' ) indicates the i'th characteristic frequency band f i' Next, the j'th velocity value v j' The corresponding amplitude statistics.

[0029] Furthermore, the output calibrated vibration frequency domain fingerprint spectrum includes:

[0030] The vibration-velocity mapping relationship matrix is ​​dynamically corrected by the Kalman filter to obtain the corrected M vs Matrix; according to the modified M vs The vibration frequency domain fingerprint spectrum is calibrated by the matrix to obtain the calibrated vibration frequency domain fingerprint spectrum.

[0031] Furthermore, generating point cloud spatiotemporal coding includes:

[0032] Collect point cloud data of the highway inspection area, and record the point cloud collection timestamp and spatial coordinates; calculate the spatiotemporal entropy weight coefficient of each point based on the point cloud collection timestamp and spatial coordinates; extract the surface features of the point cloud data, wherein the surface features include reflection intensity gradient, normal vector offset and local curvature mutation threshold; perform entropy value weighted encoding on the point cloud data according to the spatiotemporal entropy weight coefficient of each point and the surface features of the point cloud data, and generate point cloud spatiotemporal encoding.

[0033] Furthermore, the weighted visual feature map includes:

[0034] Calculate the gradient direction of each pixel point of the visual image to form the gradient direction of the visual image; calculate the mutual information between each characteristic frequency band of the six-axis vibration signal and the gradient direction of the visual image to obtain the pixel-frequency band mutual information matrix I pf ; Construction of maintenance equipment rigidity coefficient m 1 , road roughness n 1 , pixel-band mutual information matrix I pf The nonlinear mapping model w = F(m 1 ,n 1 ,I pf ), where F is a nonlinear function based on a deep neural network; m is determined by an analytical hierarchical process 1 、n 1 ,I pf The influence weight on w is substituted into the model w=F(m 1 ,n 1 ,I pf ), the pixel vibration sensitivity weight of each pixel is calculated; the visual image of the highway inspection area is feature extracted to obtain a visual feature map; the pixel vibration sensitivity weight of each pixel is element-by-element multiplied with the visual feature map to obtain a weighted visual feature map.

[0035] The adversarial generation data enhancement system for highway inspection and maintenance images is used to implement the adversarial generation data enhancement method for highway inspection and maintenance images, and the system includes:

[0036] Spectrum construction module: used to collect six-axis vibration signals of maintenance equipment in real time and construct vibration frequency domain fingerprint spectrum; perform frequency domain decomposition of the collected six-axis vibration signals based on the sliding time window, extract characteristic frequency bands related to the movement speed of the maintenance equipment, and generate a vibration-speed mapping relationship matrix; dynamically correct the vibration-speed mapping relationship matrix and output the calibrated vibration frequency domain fingerprint spectrum;

[0037] Feature fusion module: used to collect point cloud data in the highway inspection area, perform entropy value weighted encoding on the point cloud data, and generate point cloud spatiotemporal encoding; obtain visual images of the highway inspection area, perform cross-modal correlation analysis on the six-axis vibration signal and the visual image, and obtain weighted visual feature maps; construct a multimodal fusion feature vector based on the calibrated vibration frequency domain fingerprint map, point cloud spatiotemporal encoding, and weighted visual feature maps;

[0038] Adversarial generation module: Based on the multimodal fusion feature vector, a generative adversarial network consisting of a generator and a discriminator is constructed and trained; a new multimodal fusion feature vector is obtained, and enhanced highway inspection and maintenance images are generated based on the new multimodal fusion feature vector and the trained generative adversarial network.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention innovatively integrates multimodal data, effectively overcoming the limitations of traditional image enhancement methods. By real-time acquisition of the six-axis vibration signal of the maintenance equipment, the point cloud data and visual images of the highway inspection area, a multimodal fusion feature vector is constructed, and then an enhanced image is generated using a generative adversarial network. This not only solves the problem that traditional methods cannot simulate equipment vibration and the time and space misalignment between the lidar point cloud and the visual image, but also generates motion blur areas with physical reality, improving the authenticity and reliability of the image. In the actual application of highway inspection and maintenance, the enhanced image helps to more accurately identify road surface diseases, road facility damage, etc., providing a more accurate and comprehensive basis for highway maintenance decisions, improving the efficiency and quality of highway maintenance work, and ensuring the safe and stable operation of highways. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a principle flow chart of the adversarial generation data enhancement method of the highway inspection and maintenance image in the present invention;

[0043] Figure 2 A flow chart of a method for constructing a vibration frequency domain fingerprint spectrum in the adversarial generation data enhancement method for highway inspection and maintenance images of the present invention;

[0044] Figure 3 It is a schematic diagram of the principle of timestamp alignment of the present invention;

[0045] Figure 4 A flow chart of a method for generating a vibration-velocity mapping relationship matrix in the adversarial generation data enhancement method for highway inspection and maintenance images of the present invention;

[0046] Figure 5 A flow chart of a method for outputting a calibrated vibration frequency domain fingerprint spectrum in the adversarial generation data enhancement method for highway inspection and maintenance images of the present invention;

[0047] Figure 6 A flow chart of a method for generating spatiotemporal coding of point clouds in the adversarial generation data enhancement method for highway inspection and maintenance images of the present invention;

[0048] Figure 7 A flow chart of a method for constructing a multimodal fusion feature vector in the adversarial generation data enhancement method for highway inspection and maintenance images of the present invention;

[0049] Figure 8 This is a functional module diagram of the adversarial generation data enhancement system for highway inspection and maintenance images in the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Example 1

[0052] See also Figure 1 As shown, this embodiment provides an adversarial generation data enhancement method for highway inspection and maintenance images, including:

[0053] Step S1000, collecting six-axis vibration signals of the maintenance equipment in real time, constructing a vibration frequency domain fingerprint spectrum; performing frequency domain decomposition on the collected six-axis vibration signals based on a sliding time window, extracting characteristic frequency bands related to the movement speed of the maintenance equipment, and generating a vibration-speed mapping relationship matrix; dynamically correcting the vibration-speed mapping relationship matrix, and outputting a calibrated vibration frequency domain fingerprint spectrum;

[0054] Further, step S1000 includes:

[0055] Step S1100, collecting six-axis vibration signals of the maintenance equipment in real time and constructing a vibration frequency domain fingerprint spectrum;

[0056] Furthermore, if Figure 2 As shown, step S1100 includes:

[0057] Step S1110, installing an inertial measurement unit on the highway inspection and maintenance equipment, and collecting six-axis vibration signals of the maintenance equipment in real time during driving according to the installed inertial measurement unit; the six-axis vibration signals include acceleration signals a in three directions of X, Y, and Z axes. x (t), a y (t), a z (t) and angular velocity signals ω in the three directions of X, Y, and Z axes x (t),ω y (t),ω z (t);

[0058] Specifically, the purpose of step S1110 is to obtain the six-axis vibration signal of the highway inspection and maintenance equipment during driving, and provide the original data basis for the subsequent analysis of the equipment vibration state and the generation of relevant data. The inertial measurement unit (IMU) is a sensor combination that can measure the acceleration and angular velocity of an object. Its working principle is based on Newton's second law and the law of conservation of angular momentum. In this step, the inertial measurement unit is installed on the highway inspection and maintenance equipment, and the acceleration signals a in the three directions of the X, Y, and Z axes of the equipment are collected through its internal acceleration sensor and angular velocity sensor. x (t), a y (t), a z (t) and angular velocity signals ω in the three directions of X, Y, and Z axes x (t),ω y (t),ω z (t). The direction of the device is usually used as an important reference to establish the coordinate system. The forward direction of the device is usually set as the positive direction of the X-axis. In this way, during the driving process of the device, the acceleration signal a in the X-axis direction x(t) mainly reflects the information related to the change of driving speed, such as vehicle acceleration and deceleration. The acceleration signal a perpendicular to the driving direction and located in the horizontal direction of the device can be set as the Y axis. y (t) can reflect the lateral force and vibration of the vehicle during turning, avoiding and other operations. The Z axis is perpendicular to the plane determined by the X and Y axes, that is, perpendicular to the bottom surface of the device and upward. The acceleration signal a in the Z axis direction z (t) It can be used to monitor the vertical vibration of the vehicle caused by bumpy roads, ups and downs, etc.

[0059] Take the inspection of a highway that includes curves and straight roads as an example. When driving on a curve, the equipment will not only produce acceleration changes in the X and Y directions, but also in the Z axis direction due to the vehicle tilt; when driving on a straight road, the acceleration change in the X axis direction is more obvious, and the angular velocity signal will also change accordingly under different driving conditions. By collecting these signals in real time, the vibration of the equipment under various driving conditions can be accurately recorded. The beneficial effect of this real-time collection method is that it can obtain equipment vibration information comprehensively and dynamically. Because the driving environment of highway inspection and maintenance equipment is complex and changeable, different road conditions, driving speeds and other factors will cause different equipment vibration states. By continuously collecting six-axis vibration signals, these changes can be captured in time, providing rich data support for the subsequent accurate analysis of equipment vibration characteristics. For example, if a section of the road surface is uneven, the equipment will produce frequent and complex vibrations during driving. By collecting six-axis vibration signals, the performance of this vibration in each axis can be accurately reflected, thereby providing a basis for analyzing the operating status and potential failures of the equipment. From the perspective of data integrity, the six-axis vibration signal covers the acceleration and angular velocity information of the equipment in three dimensions of space. Compared with only collecting signals from a single or partial axis, it can more comprehensively describe the vibration state of the equipment, avoid missing important information, and improve the accuracy and reliability of subsequent analysis results.

[0060] Step S1120, performing frequency domain conversion on the collected six-axis vibration signal to obtain main frequency band energy distribution and harmonic distortion rate;

[0061] Specifically, frequency domain conversion is the process of converting a time domain signal (i.e., a vibration signal that changes over time) to the frequency domain. Its principle is based on Fourier transform. Through Fourier transform, complex time series can be decomposed into a combination of sine and cosine waves of different frequencies, thereby revealing the energy distribution of the signal on different frequency components. In the specific calculation process, the fast Fourier transform (FFT) algorithm can be used, which is a method for efficiently calculating discrete Fourier transform (DFT) and can quickly obtain the frequency domain representation of the vibration signal. After obtaining the frequency domain signal, the calculation of the main frequency band energy distribution is by determining the frequency range where the energy of the vibration signal is mainly concentrated, that is, the main frequency band, and then calculating the proportion of the energy in this frequency band to the total energy. For example, if it is found through analysis that the vibration energy of the equipment is more concentrated in the 5-15Hz frequency band, the energy distribution of the main frequency band can be obtained by calculating the sum of the energy of all frequency components in this frequency band and dividing it by the total energy of the entire frequency domain signal. Harmonic distortion rate reflects the distortion of the signal, and its calculation is based on the relationship between the fundamental wave and the harmonics. The fundamental wave is the basic frequency component of the signal, and the harmonics are the frequency components that are integer multiples of the fundamental frequency. When calculating the harmonic distortion rate, first determine the amplitude of the fundamental wave, then calculate the sum of the squares of all harmonic amplitudes, then divide the sum of the squares of the harmonic amplitudes by the square of the fundamental wave amplitude, and finally take the square root of the result and multiply it by 100% to obtain the harmonic distortion rate.

[0062] This step can understand the concentrated frequency range of the equipment vibration energy by obtaining the energy distribution of the main frequency band. For example, when it is found that the energy distribution of the equipment in a certain frequency band is abnormally increased, it may mean that there is a problem with the components or operating status of the equipment related to the frequency. If on a certain maintenance equipment, the main frequency band energy is mainly concentrated in 10-12Hz when the engine is operating normally, but the data collected at a certain time shows that the energy in the 15-18Hz frequency band has increased significantly, further inspection shows that a certain transmission component of the engine is worn, resulting in a change in the vibration energy distribution. The harmonic distortion rate can reflect the degree of signal distortion. If the harmonic distortion rate is too high, it means that there are more harmonic components in the signal, which may be caused by electrical faults inside the equipment, abnormal friction of mechanical parts, etc. By monitoring the harmonic distortion rate, potential fault hazards of the equipment can be discovered in time, providing a basis for preventive maintenance of the equipment. Moreover, these two features complement each other, can analyze the vibration characteristics of the equipment more comprehensively and deeply, help technicians more accurately judge the operating status of the equipment, improve the reliability and safety of the equipment, and reduce the interruption and loss of highway inspection and maintenance work caused by equipment failure.

[0063] Step S1130, analyzing the relationship between the acceleration signals in three directions and the angular velocity signals in three directions to obtain the inter-axis coupling coefficient;

[0064] The purpose of step S1130 is to deeply analyze the relationship between the acceleration signal and the angular velocity signal in the six-axis vibration signal of the highway inspection and maintenance equipment, obtain the inter-axis coupling coefficient, and provide key data support for constructing a vibration frequency domain fingerprint that can accurately reflect the vibration characteristics of the equipment. When the highway inspection and maintenance equipment is running, the acceleration signal a in the three directions of X, Y, and Z axes is x (t), a y (t), a z (t) and the angular velocity signals ω in the three directions of X, Y, and Z axes x (t),ω y (t),ω z (t), will be affected by factors such as the driving state of the equipment and the road conditions. The inter-axis coupling coefficient obtained by analyzing the relationship between these signals can reflect the degree of correlation between vibrations in different axis directions, which is of great significance for accurately judging the vibration state of the equipment. From a practical application perspective, if a highway inspection and maintenance equipment passes through a rugged road, the acceleration signal in the X-axis direction suddenly increases. At this time, if the inter-axis coupling coefficient shows that the X-axis and Y-axis vibrations are closely related, then the Y-axis vibration is also likely to be significantly affected. By obtaining the inter-axis coupling coefficient, the interaction between the vibrations of different axes can be captured, providing a basis for the subsequent comprehensive evaluation of the vibration of the equipment. The beneficial effects of this step are reflected in many aspects. First, it helps to understand the internal mechanism of equipment vibration more accurately. The vibrations of different axes do not exist in isolation, but are interrelated. The inter-axis coupling coefficient can quantify this correlation, so that technicians have a deeper understanding of equipment vibration. Secondly, in terms of equipment fault diagnosis, the change of the inter-axis coupling coefficient can be used as an important fault indication signal. For example, when an inter-axis coupling coefficient fluctuates abnormally, it may mean that a related component of the equipment has failed, such as loose connection components, worn bearings, etc. This helps to promptly discover and solve potential problems, ensure the normal operation of the equipment, and improve the efficiency and reliability of highway inspection and maintenance work. Furthermore, the accurate inter-axis coupling coefficient provides more accurate data for subsequent simulation of equipment motion blur. The simulated images generated based on this can more realistically reflect the actual operation of the equipment, thereby improving the accuracy of highway inspection and maintenance image analysis.

[0065] Further, step S1130 includes:

[0066] Step S1131: Acceleration signals a in the three directions of X, Y and Z axes x (t), a y (t), a z (t) and angular velocity signals ω in the three directions of X, Y, and Z axes x (t),ω y (t),ω z (t) Perform timestamp alignment;

[0067] Specifically, in the actual acquisition process, these signals may have time deviations due to factors such as the sampling frequency of the sensor and transmission delay. If the timestamps are not aligned, the subsequent analysis based on these signals will have errors, resulting in inaccurate results. The specific method of timestamp alignment is to record the exact time point of each signal sampling, and through the time synchronization algorithm, make all signals in the same reference frame on the time axis. For example, Figure 3 As shown, before the timestamp alignment, assuming that the acceleration signal a x The sampling time of (t) is t1, ω x The sampling time of (t) is t1+Δt (Δt is the time deviation). Through the timestamp alignment operation, ω x The time of (t) is adjusted to time t1, and the two are synchronized in time after the timestamps are aligned.

[0068] The beneficial effects of this step are significant. First, it ensures the consistency and accuracy of the data. When analyzing the relationship between signals, only data that is synchronized in time can truly reflect the intrinsic connection between the signals. If the signal time is not synchronized, the calculated correlation and other parameters will be biased, which will affect the judgment of the vibration state of the equipment. Second, it lays a solid foundation for the subsequent construction of multivariate signal space matrices and the calculation of correlation coefficient matrices and nonlinear correlation matrices. Only when time is aligned can the calculation results of these matrices be reliable and accurately reflect the distribution of signals in time and dimension and the relationship between signals. Third, in equipment status monitoring and fault diagnosis, time-aligned data can more accurately reflect the real-time changes in the equipment operation process, which helps to detect abnormal conditions of the equipment in a timely manner. For example, when a component of the equipment suddenly fails, the time-aligned signal can more clearly show the changes in the vibration signals of each axis at the moment of the failure, providing a more accurate basis for fault diagnosis.

[0069] Step S1132, according to the aligned a x (t), a y (t), a z (t),ω x (t),ω y (t),ω z (t), construct the multivariate signal space matrix;

[0070] Further, step S1132 includes:

[0071] Step S11321, the aligned a x (t), a y (t), a z (t),ω x (t),ωy (t),ω z (t) is combined into a six-dimensional signal vector ; Represents the transpose of a vector;

[0072] Step S11322, after a period of time Inside, collect The six-dimensional signal vector at each moment constructs a multivariate signal space matrix ,in, is the six-dimensional signal vector at the nth moment.

[0073] Specifically, the core of step S1132 is to use the timestamps to align Construct a multivariate signal space matrix to fully reflect the distribution of the six-axis vibration signal in time and dimension. In the specific implementation, first perform step S11321, and Combined into a six-dimensional signal vector This step is to integrate vibration signals of different dimensions into a vector for subsequent processing. For example, at a certain time t, the collected acceleration signal , , angular velocity signal , then the six-dimensional signal vector at this moment Then proceed to step S11322, after a period of time Inside, collect The six-dimensional signal vector at each moment constructs a multivariate signal space matrix ,in, It is For example, within 10 seconds (i.e. ), the signal is collected every 0.1 second, and the signal is collected at 100 moments in total (i.e. ), these 100 six-dimensional signal vectors are arranged in order to form a The multivariate signal space matrix .

[0074] The beneficial effects of this step are reflected in many aspects. From the perspective of data representation, the multivariate signal space matrix organizes a large amount of vibration signal data in an orderly manner, and intuitively shows the distribution of the signal in time and dimension. By observing the changes in the elements of the matrix, the changing trend of the vibration signals of each axis at different times can be clearly seen. In terms of data analysis, this matrix form provides convenience for the subsequent calculation of the correlation coefficient matrix and the nonlinear correlation matrix, and can use the matrix operation method to efficiently analyze the relationship between signals. For example, by operating on the matrix M, the correlation between the signals of different axes can be quickly obtained, thereby discovering the potential vibration law. In the equipment performance evaluation, the multivariate signal space matrix can comprehensively reflect the vibration state of the equipment over a period of time, and provide strong data support for evaluating the stability and reliability of the equipment. If an abnormal fluctuation occurs in a column of elements in the matrix (representing a six-dimensional signal vector at a certain moment), it may mean that the equipment has an abnormal situation at that moment. The technicians can further analyze the cause and take corresponding measures based on this.

[0075] Step S1133, obtaining a correlation coefficient matrix R and a nonlinear correlation matrix N according to the multivariate signal space matrix;

[0076] Further, step S1133 includes:

[0077] Step S11331, for the multivariate signal space matrix Any two components in and ,calculate and The correlation coefficients between all different axes give a The correlation coefficient matrix , the correlation coefficient matrix Elements in Indicates ''Axis and ''Correlation between axis signals; where, is the six-dimensional signal vector at the i-th moment, is the six-dimensional signal vector at the jth moment, 1≤i≤n, 1≤j≤n, ;

[0078] Step S11332, calculate and The mutual information value of , according to the mutual information value Construct the nonlinear correlation matrix N.

[0079] Specifically, in step S11331, the Pearson correlation coefficient is used to measure the linear correlation of signals between different axes. In step S11332, mutual information is a concept in information theory, which is used to measure the degree of mutual dependence between two random variables. By calculating the mutual information values ​​between all different axes, a 6×6 nonlinear correlation matrix N is constructed, and the matrix elements represent the nonlinear correlation between the corresponding axes. The beneficial effects of this step are very important. By obtaining the correlation coefficient matrix R and the nonlinear correlation matrix N, the correlation between signals between different axes can be comprehensively analyzed from both linear and nonlinear perspectives. In actual equipment operation, the relationship between signals often contains both linear and nonlinear components. The correlation coefficient matrix R can reveal the degree of linear correlation between signals. For example, if If it is close to 1 or -1, it means that there is a strong linear relationship between the signals of the i''th axis and the j''th axis; if it is close to 0, the linear relationship is weak. The nonlinear correlation matrix N supplements the nonlinear information. For some complex equipment vibration situations, nonlinear correlation may play a key role. For example, in the vibration process of some equipment, the vibrations of different axes may produce nonlinear coupling through complex mechanical structures. In this case, relying solely on linear correlation analysis may not accurately capture the relationship between signals, while the nonlinear correlation matrix N can discover these hidden nonlinear connections. In equipment fault diagnosis, comprehensive correlation analysis helps to more accurately determine the type and location of the fault. For example, when the vibration of a certain axis is abnormal, by analyzing the correlation coefficient matrix R and the nonlinear correlation matrix N, it can be determined which axes have a strong correlation with the abnormal axis, thereby narrowing the scope of fault investigation and improving the efficiency and accuracy of fault diagnosis.

[0080] Step S1134: Correlation coefficient matrix and the nonlinear correlation matrix The fusion is performed to obtain a comprehensive correlation matrix C, from which the inter-axis coupling coefficient is extracted.

[0081] Specifically, the formula To fuse the correlation coefficient matrix R and the nonlinear correlation matrix N, where α is the fusion coefficient, which is generally between 0.5-0.8. The value of α is determined based on a large number of experiments and practical application experience. When α is close to 0.5, it means that the importance of the linear correlation matrix R and the nonlinear correlation matrix N is relatively balanced; when α is close to 0.8, it places more emphasis on the role of the correlation coefficient matrix R. For example, in a vibration analysis scenario of a highway inspection and maintenance equipment, after many experiments, it was found that when α=0.6, it can better comprehensively reflect the relationship between the vibrations of different axes of the equipment. The element C in the fused comprehensive correlation matrix C i''j''It is the coupling coefficient between any two axes i'' and j''. This coupling coefficient combines linear and nonlinear correlation information and can more accurately describe the mutual influence of inter-axis vibration than using linear or nonlinear correlation indicators alone.

[0082] The beneficial effects of this step are reflected in multiple aspects. From the perspective of equipment vibration characteristics analysis, the inter-axis coupling coefficient integrates linear and nonlinear related information, and can more comprehensively and accurately reflect the interaction mechanism between vibrations in different axis directions. For example, during the operation of the equipment, the vibrations of different axes may affect each other through complex physical processes, including both linear transmission relationships and nonlinear resonance phenomena. The inter-axis coupling coefficient can quantify these complex relationships and help technicians deeply understand the nature of equipment vibration. In terms of equipment failure prediction, the change of the inter-axis coupling coefficient can be used as a sensitive indicator. When equipment parts gradually show potential failures such as wear and looseness, the inter-axis coupling coefficient will change accordingly. By monitoring the change trend of the inter-axis coupling coefficient, the abnormal situation of the equipment can be discovered in advance, providing a basis for preventive maintenance of the equipment, reducing the probability of equipment failure, and reducing the interruption and economic losses of highway inspection and maintenance work caused by equipment failure. When simulating equipment motion blur, accurate inter-axis coupling coefficients can provide more reliable data support for generating motion blur effects that are more in line with the actual situation, thereby improving the accuracy and reliability of subsequent highway inspection and maintenance image analysis.

[0083] Step S1140, integrating the main frequency band energy distribution, harmonic distortion rate and inter-axis coupling coefficient to construct a vibration frequency domain fingerprint.

[0084] Specifically, the energy distribution of the main frequency band reflects the concentration of the equipment's vibration energy in different frequency bands. As mentioned earlier, it can help determine the main distribution area of ​​the equipment's vibration energy. The harmonic distortion rate reflects the distortion of the signal. The higher the value, the greater the degree to which the signal deviates from the ideal sine wave, which means that there may be abnormal interference factors during the operation of the equipment. The inter-axis coupling coefficient shows the mutual influence between vibrations in different axial directions. The comprehensive correlation matrix C is obtained by calculating the correlation coefficient matrix R and the nonlinear correlation matrix N and fusing them. The elements C extracted from C are i''j'' It is the coupling coefficient between axis i'' and axis j'', which quantifies the degree of correlation between vibrations of different axes.

[0085] For example, suppose a highway inspection and maintenance equipment has a large vibration in the X-axis direction during operation. Through the inter-axis coupling coefficient, it is found that there is a strong coupling relationship between the X-axis and the Y-axis, which means that the Y-axis vibration may be significantly affected by the X-axis vibration. Combining these three features to construct a vibration frequency domain fingerprint is like generating a unique "fingerprint" for the vibration state of the equipment. Because different equipment will present different patterns in normal operation and fault state, just like each person's fingerprint is unique.

[0086] Its beneficial effects are reflected in many aspects. From the perspective of equipment status monitoring, the vibration frequency domain fingerprint spectrum can establish an accurate vibration characteristic model for the equipment. When the equipment fails or its performance deteriorates, its vibration frequency domain fingerprint spectrum will change accordingly. By comparing the current spectrum with the spectrum in the normal state, technicians can quickly and accurately determine whether the equipment is abnormal, as well as the location and cause of the abnormality. For example, if it is found that the harmonic distortion rate of a certain equipment suddenly increases, and the inter-axis coupling coefficient changes significantly between certain axis pairs, combined with the change in the main frequency band energy distribution, it can be inferred that the equipment may have a problem in a component that involves multi-axis motion and is prone to harmonic interference. In terms of simulating equipment motion blur, the vibration frequency domain fingerprint spectrum provides key input information for subsequent steps. Because it accurately reflects the vibration state of the equipment, the motion blur simulation generated based on this can be more consistent with the blur effect produced during the actual operation of the equipment, thereby laying the foundation for generating high-quality highway inspection and maintenance simulation images, improving the accuracy of subsequent image analysis, and helping to more accurately evaluate highway inspection and maintenance conditions.

[0087] Step S1200, performing frequency domain decomposition on the collected six-axis vibration signal based on a sliding time window, extracting characteristic frequency bands related to the movement speed of the maintenance equipment, and generating a vibration-speed mapping relationship matrix;

[0088] Furthermore, if Figure 4 As shown, step S1200 includes:

[0089] Step S1210, performing frequency domain decomposition on the collected six-axis vibration signal based on a sliding time window to extract characteristic frequency bands related to the movement speed of the maintenance equipment;

[0090] Specifically, sliding time windows are a common method used in time series data processing. By sliding a fixed-length window on the time axis, the data within the window is analyzed. and the sliding step length Δt' are crucial. For example, setting T windowis 1 second, and Δt' is 0.1 second, so that the vibration signals of different time periods can be analyzed in detail while ensuring data continuity. When processing the six-axis vibration signal in each sliding time window, the short-time Fourier transform (STFT) in Fourier transform is used to convert the time domain signal into a frequency domain signal. The principle of STFT is to slide a window function on the signal and perform Fourier transform on the signal in the window to obtain the distribution of the frequency components of the signal at different local times. The reason for choosing STFT is that it can simultaneously display the characteristics of the signal in both time and frequency dimensions, and is suitable for analyzing non-stationary signals, while the vibration signal of maintenance equipment is non-stationary during driving. The frequency band of 0.5-20Hz is selected as the characteristic frequency band related to the speed of the equipment, which is determined based on a large number of experiments and practical experience. When the equipment is moving, the signal components in this frequency band are closely related to the speed change. For example, when the equipment accelerates, the signal strength of some frequencies in this frequency band may increase; when it decelerates, it will weaken. By extracting the signal components in this characteristic frequency band from the frequency domain signal of each time window, the vibration characteristics related to speed can be accurately obtained.

[0091] The beneficial effects of this step are significant. First, the sliding time window combined with the STFT method can dynamically track the changes in the frequency components of the vibration signal over time. For example, when the equipment passes through a rough road, the vibration signal will mutate. This method can capture these changes in time and accurately extract the frequency characteristics corresponding to the current state of motion, avoiding ignoring local changes due to overall analysis of the signal. Secondly, the accurate extraction of the characteristic frequency bands related to speed improves the accuracy of subsequent analysis of the relationship between equipment vibration and speed. By analyzing the signals of these characteristic frequency bands, we can have a deeper understanding of the vibration laws of the equipment at different speeds and provide a reliable data basis for establishing an accurate vibration-speed mapping relationship matrix. Furthermore, in terms of equipment fault diagnosis, the changes in these characteristic frequency bands can serve as an important diagnostic basis. If the signal of a certain characteristic frequency band is abnormal, it may mean that there is a problem with a speed-related component of the equipment, which helps to quickly locate the source of the fault and improve the reliability and safety of the equipment.

[0092] Step S1220, performing time-frequency representation on the six-axis vibration signal of each characteristic frequency band, and extracting the amplitude statistics of each characteristic frequency band;

[0093] Specifically, the amplitude statistics include the amplitude mean and peak value, and the time-frequency representation uses short-time Fourier transform (STFT). This is because STFT can simultaneously display the characteristics of the signal in two dimensions, time and frequency, and is suitable for analyzing non-stationary vibration signals. After STFT processing of the six-axis vibration signal of each characteristic frequency band, a time-frequency diagram is obtained, from which the energy distribution of the signal at different times and frequencies can be clearly observed. When calculating the amplitude mean, first determine the time interval and frequency range corresponding to each characteristic frequency band in the time-frequency diagram, then accumulate the amplitude values ​​of all sampling points within the range, and then divide it by the total number of sampling points to obtain the amplitude mean. When calculating the amplitude peak, find the maximum value of the amplitude values ​​of all sampling points in the time-frequency diagram corresponding to each characteristic frequency band. This value is the amplitude peak.

[0094] The beneficial effects of this step are reflected in many aspects. In equipment status monitoring, the amplitude mean and peak can directly reflect the intensity characteristics of the vibration signal. The amplitude mean can reflect the average intensity of the vibration of the equipment over a period of time. If the amplitude mean exceeds the normal range, it may mean that the overall vibration of the equipment has intensified and there are potential problems. The amplitude peak can reflect the instantaneous maximum intensity in the vibration signal. When the amplitude peak is too high, it may indicate that the equipment has been hit hard at a certain moment, and it is necessary to pay attention to whether the relevant components are damaged. In terms of equipment fault diagnosis, the change of amplitude statistics can serve as an important basis for fault diagnosis. For example, when a part of the equipment is worn or loose, the amplitude mean and peak of its vibration signal may change significantly. By monitoring these changes, faults can be discovered in time and corresponding measures can be taken. In addition, when simulating equipment motion blur, amplitude statistics help to more accurately simulate the impact of equipment vibration on the image. Different amplitude statistics correspond to different degrees of vibration blur effects, which provide more accurate parameter support for generating motion blurred images that conform to actual conditions, and improve the authenticity and reliability of image simulation.

[0095] Step S1230, obtaining the movement speed of the maintenance equipment, associating the amplitude statistics of each characteristic frequency band with the movement speed of the maintenance equipment, and establishing a vibration-speed mapping relationship matrix M vs ; Among them, the matrix element M vs (f i' ,v j' ) indicates the i'th characteristic frequency band f i' Next, the j'th velocity value v j' The corresponding amplitude statistics.

[0096] Specifically, the movement speed of the maintenance equipment is obtained through GPS and other devices. GPS (Global Positioning System) is a positioning technology based on satellite navigation. It calculates the location information of the equipment by receiving signals transmitted by multiple satellites, and calculates the movement speed based on the position changes at different times. Establish the vibration-speed mapping relationship matrix M vs When the matrix element M vs (f i' ,v j' ) indicates the i'th characteristic frequency band f i' Next, the j'th velocity value v j' For example, assuming the characteristic frequency band for , speed value Through the previous steps, the mean amplitude at this characteristic frequency band and speed is , the peak amplitude is , then in the matrix In, M vs (f 1 ,v 1 ) can be used to include and The data structure is represented as .

[0097] The beneficial effect of this step is very significant. From the perspective of equipment performance analysis, the vibration-velocity mapping relationship matrix M vs The vibration of each characteristic frequency band of the equipment at different speeds is clearly displayed. By analyzing the data in the matrix, the vibration characteristics of the equipment at different operating speeds can be understood, and the stability and reliability of the equipment can be evaluated. For example, if the amplitude statistics of a specific characteristic frequency band are found to be abnormally increased within a certain speed range, it means that the equipment may have a vibration problem at this speed, and the relevant components need to be further checked. In terms of equipment failure prediction, the matrix provides a strong basis for predicting equipment failure. As the equipment runs, potential signs of failure can be discovered in a timely manner by continuously monitoring the changes in the vibration-speed mapping relationship. For example, when some elements in the matrix show an abnormal change trend, it may indicate that the equipment is about to fail, so that maintenance can be arranged in advance to avoid work interruptions and losses caused by equipment failure. In addition, in the simulation and analysis of highway inspection and maintenance images, the vibration-speed mapping relationship matrix provides key data support for generating more realistic motion blurred images. Based on this matrix, the blurring effect of the image can be accurately simulated according to the actual movement speed and vibration of the equipment, improving the accuracy of image analysis, which helps to more accurately evaluate the highway inspection and maintenance status.

[0098] Step S1200 aims to process the collected six-axis vibration signals, extract the characteristic frequency bands related to the movement speed of the maintenance equipment, and establish a vibration-speed mapping relationship matrix, so as to provide key data support for the subsequent analysis of the relationship between equipment vibration and speed, and also lay the foundation for generating a calibrated vibration frequency domain fingerprint spectrum, so that the spectrum can more accurately reflect the vibration characteristics of the equipment at different speeds. During the driving process of highway inspection and maintenance equipment, its vibration situation is complex and closely related to the movement speed. Through this series of operations, the intrinsic relationship between vibration signals and speed can be deeply explored, thereby providing a more accurate basis for the processing and analysis of highway inspection and maintenance images. Under different road conditions, such as highways, rural roads, etc., the driving speed of maintenance equipment is different, and the vibration signals generated will also be different. Through step S1200, these differences can be effectively captured, providing strong support for the subsequent accurate evaluation of the equipment operation status and the generation of high-quality simulation images. The beneficial effects of this step are reflected in many aspects. First, in the field of equipment status monitoring, the vibration-speed mapping relationship matrix can intuitively display the vibration characteristics of the equipment at different speeds. For example, when the amplitude of the vibration signal in a certain speed range is found to be abnormally increased, combined with the mapping relationship matrix, it can be judged that the equipment may have a potential failure risk at this speed, which is helpful for timely equipment maintenance and ensuring the smooth progress of highway inspection and maintenance work. Second, when simulating equipment motion blur, the accurate vibration-speed mapping relationship can provide key data for generating motion blur images that conform to the actual situation. The images generated based on these data are closer to the images taken in the real scene, which is conducive to the subsequent analysis and processing of highway inspection and maintenance images, and improves the accuracy and reliability of image analysis. Third, for optimizing highway inspection and maintenance strategies, by analyzing the vibration-speed mapping relationship, the vibration of the equipment at different speeds can be understood, and then the driving speed of the equipment can be reasonably adjusted to reduce equipment wear and extend the service life of the equipment, while improving the efficiency and quality of highway inspection and maintenance.

[0099] Step S1300, dynamically correct the vibration-velocity mapping relationship matrix through a Kalman filter, and output a calibrated vibration frequency domain fingerprint.

[0100] Furthermore, if Figure 5 As shown, step S1300 includes:

[0101] Step S1310, dynamically modify the vibration-velocity mapping relationship matrix through the Kalman filter to obtain the modified M vs matrix;

[0102] Step S1320, based on the modified M vs The vibration frequency domain fingerprint spectrum is calibrated by the matrix to obtain the calibrated vibration frequency domain fingerprint spectrum.

[0103] Specifically, the core purpose of step S1300 is to dynamically correct the vibration-velocity mapping relationship matrix through the Kalman filter, and calibrate the vibration frequency domain fingerprint spectrum based on the corrected matrix, so that the calibrated spectrum can more accurately reflect the actual vibration state of the equipment, and provide a reliable basis for the subsequent generation of motion blur images that conform to the actual situation. During the highway inspection and maintenance process, the operating environment of the equipment is complex and changeable. Factors such as road conditions and vehicle loads will cause changes in the vibration signal and movement speed of the equipment, so that the previously established vibration-velocity mapping relationship matrix may not accurately reflect the actual situation. Therefore, it is necessary to dynamically correct it to ensure that the subsequent analysis and application based on the matrix are accurate and reliable.

[0104] The main task of step S1310 is to use the Kalman filter to dynamically correct the vibration-velocity mapping relationship matrix, thereby obtaining the corrected M vs Matrix. Kalman filter is an algorithm that uses linear system state equations to optimally estimate system states through system input and output observation data. Its core principle is based on two steps: prediction and update.

[0105] In the prediction stage, the Kalman filter predicts the current state based on the state estimate at the previous moment and the state transfer equation of the system. For the vibration-velocity mapping relationship matrix, assume that the matrix at the previous moment is , according to the physical model and related parameters of the device motion, the state transfer equation can be established ,in is the state transfer matrix, which describes the change law of the matrix over time. For example, if the motion state of the device is relatively stable, The elements of the matrix may represent the linear change relationship between the elements in the matrix over time; if the motion state of the device is complex and changeable, The elements of the matrix need to be determined based on a more complex physical model.

[0106] In the update phase, the Kalman filter combines the latest vibration signal and speed measurement data to correct the prediction results. First, the Kalman gain is calculated , and its calculation formula is ,in is the forecast error covariance matrix, which reflects the uncertainty of the forecast value; is the observation matrix, which is used to relate the system state to the observation data; is the observation noise covariance matrix, Represents the transpose of the observation matrix. Then, the predicted matrix is ​​updated according to the Kalman gain to obtain the corrected matrix ,in It is the observation data at the current moment, that is, the part of the latest vibration signal and speed measurement data that is processed and related to the matrix.

[0107] In the correction process, it is of great significance to set an adaptive noise covariance matrix. According to changes in the equipment operating environment, such as changes in road surface flatness and vehicle load, the parameters of the noise covariance matrix are dynamically adjusted. When the road surface becomes rough, the noise of the equipment vibration signal increases and the uncertainty increases. At this time, the element value of the noise covariance matrix is ​​appropriately increased, so that the Kalman filter pays more attention to new observation data during the update process, thereby better adapting to the uncertainty in the signal; when the road surface is relatively flat, the signal noise is relatively small, and the element value of the noise covariance matrix is ​​reduced, the accuracy of the filter is improved, and the corrected matrix is ​​closer to the actual situation. For example, when driving on a rough road, the elements related to the vibration signal in the noise covariance matrix change from the initial value Q 1 Increase to Q 2 (Q 2 >Q 1 ), so that when calculating the Kalman gain, the weight of the new observation data increases, and the modified M vs The matrix can more accurately reflect the vibration-speed relationship of the equipment under the road condition.

[0108] The beneficial effects of this step are significant. Through the dynamic correction of the Kalman filter, the accuracy and adaptability of the vibration-velocity mapping relationship matrix can be effectively improved. An accurate matrix can more accurately describe the relationship between the vibration and speed of the equipment under different operating conditions, providing a more reliable data basis for subsequent analysis. In equipment fault diagnosis, an accurate matrix helps to more accurately determine the correlation between abnormal equipment vibration and speed changes, thereby more accurately locating the cause of the fault. For example, if the equipment vibrates abnormally at a certain speed, based on the corrected matrix, it can be more accurately analyzed whether it is normal vibration fluctuations caused by speed changes, or abnormal vibrations caused by faults in the equipment itself. In addition, in the generation of motion blurred images of simulated equipment, an accurate vibration-velocity mapping relationship matrix can provide more realistic parameters for the simulation process, so that the generated motion blurred images more realistically reflect the imaging conditions of the equipment under different speeds and vibration states, and improve the quality and reliability of image simulation.

[0109] The goal of step S1320 is to obtain the modified M according to step S1310. vs The vibration frequency domain fingerprint spectrum is calibrated by the matrix, and then the calibrated vibration frequency domain fingerprint spectrum is obtained. The vibration frequency domain fingerprint spectrum is composed of the main frequency band energy distribution, harmonic distortion rate and inter-axis coupling coefficient, which comprehensively reflect the vibration state of the equipment. vsThe matrix contains more accurate information about the relationship between device vibration and speed. Using this information to calibrate the fingerprint map can optimize the map's representation of the device's vibration state.

[0110] During the calibration operation, each feature of the vibration frequency domain fingerprint spectrum needs to be adjusted separately. vs The element M in the matrix vs (f i' ,v j' ) indicates the characteristic frequency band f in the i′th frequency band i′ Next, the j′th velocity value v j′ By analyzing the changes in the amplitude statistics of each characteristic frequency band at different speeds, the actual distribution of the equipment vibration energy in different frequency bands can be inferred. For example, if at a certain speed v 2 Next, M vs The matrix shows that the amplitude mean of the 0.5-10 Hz frequency band has increased significantly, which means that the vibration energy of this frequency band has increased. Therefore, in the energy distribution of the main frequency band, the energy proportion of this frequency band should be increased accordingly. In the specific calculation, assuming that the original energy proportion of this frequency band is E1, by vs The amplitude statistics of this frequency band and speed in the matrix are compared and analyzed with the amplitude statistics of other frequency bands and speeds. According to the relationship between energy and amplitude (generally speaking, energy is proportional to the square of amplitude), the energy proportion of this frequency band is recalculated as E2 (E2>E1), thereby completing the calibration of the energy distribution of the main frequency band.

[0111] For harmonic distortion, changes in the equipment's vibration-velocity relationship can affect the harmonic content of the signal. The changes in vibration of each characteristic frequency band at different speeds in the matrix can be used as a basis for judging the changes in harmonic distortion rate. For example, when the speed of the equipment changes, if The matrix shows that the vibration modes of certain characteristic frequency bands have changed, which may cause changes in harmonic content. Matrix analysis, combined with the calculation method of harmonic distortion rate (harmonic distortion rate refers to the ratio of harmonic content to fundamental content. When calculating, first determine the fundamental amplitude, then calculate the sum of the squares of all harmonic amplitudes, then divide the sum of the squares of the harmonic amplitudes by the square of the fundamental amplitude, and finally take the square root and multiply it by ), recalculate the harmonic distortion rate. Assume that the original harmonic distortion rate is , after analysis After recalculating the matrix, the new harmonic distortion rate is obtained , thereby completing the calibration of harmonic distortion rate.

[0112] In terms of the inter-axis coupling coefficient, changes in equipment vibration and speed may change the vibration correlation between different axes. The matrix reflects the statistical changes in the amplitude of each axis vibration at different speeds, which are related to the inter-axis coupling coefficient. The matrix shows that at a certain speed, Axis and The vibration amplitudes of the shafts vary in certain correlations, which may mean that the coupling coefficients between the shafts need to be adjusted. The information in the matrix is ​​combined with the previous method of calculating the inter-axis coupling coefficient (by calculating the correlation coefficient matrix and the nonlinear correlation matrix And fused to get the comprehensive correlation matrix ,from Extract the inter-axis coupling coefficient from ), and recalculate the inter-axis coupling coefficient. Assume that the original coupling coefficient between the X-axis and the Y-axis is , after recalculation, we get , complete the calibration of the inter-axis coupling coefficient.

[0113] Through the above calibration of the main frequency band energy distribution, harmonic distortion rate and inter-axis coupling coefficient, the vibration frequency domain fingerprint spectrum is calibrated to obtain the calibrated vibration frequency domain fingerprint spectrum. This step has many beneficial effects. In the field of equipment status monitoring and fault diagnosis, the calibrated vibration frequency domain fingerprint spectrum can provide more accurate equipment vibration information, which helps technicians to find potential problems of the equipment in time. For example, by comparing the spectrum before and after calibration, if it is found that the calibration result of a certain feature is greatly different from the normal range, such as the abnormal increase in the main frequency band energy distribution of a certain frequency band, combined with the equipment's operating speed information, it can be inferred that the corresponding parts of the equipment at this speed may have wear or failure hazards, so as to arrange inspections and maintenance in time, avoid equipment failures, reduce equipment maintenance costs, and improve equipment reliability and operation efficiency. In terms of the generation of simulated equipment motion blur images, the calibrated vibration frequency domain fingerprint spectrum is used as the reference input of the generator motion blur simulation, which can significantly improve the quality of the simulated image. Since the calibrated spectrum more accurately reflects the actual vibration state of the equipment, the motion blur image generated based on it can more realistically simulate the imaging of the equipment during highway inspection and maintenance. For example, when analyzing the condition of a highway surface, more realistic simulation images can help technicians more clearly observe subtle cracks, potholes and other defects on the road surface, improve the efficiency and accuracy of highway inspection and maintenance work, and provide a more reliable basis for highway maintenance decisions.

[0114] Step S2000, collecting point cloud data of the highway inspection area, performing entropy value weighted encoding on the point cloud data, and generating point cloud spatiotemporal encoding; obtaining a visual image of the highway inspection area, performing cross-modal correlation analysis on the six-axis vibration signal and the visual image, and obtaining a weighted visual feature map; constructing a multimodal fusion feature vector based on the calibrated vibration frequency domain fingerprint map, the point cloud spatiotemporal encoding, and the weighted visual feature map;

[0115] Furthermore, step S2000 includes:

[0116] Step S2100, collecting point cloud data of the highway inspection area, performing entropy value weighted coding on the point cloud data, and generating point cloud spatiotemporal coding;

[0117] Furthermore, if Figure 6 As shown, step S2100 includes:

[0118] Step S2110, collecting point cloud data of the highway inspection area, and recording the point cloud collection timestamp and spatial coordinates;

[0119] Specifically, point cloud data is acquired by laser scanning equipment, which uses the time difference between the emission and reception of laser beams to measure the distance information of the surface points of an object, thereby obtaining the three-dimensional spatial coordinates of the object. During the acquisition process, the scanning parameters are dynamically adjusted according to the complexity of the scene, in order to improve the acquisition efficiency while ensuring data quality. For example, in areas with curves or complex structures, such as road intersections and near bridge structures, the scanning resolution is increased. Because the shape and spatial layout of objects in these areas are more complex, higher resolution can obtain more detailed point cloud data and accurately describe the shape and position information of objects. Assuming that at a bend in a road, high-resolution scanning can clearly obtain detailed point cloud data of guardrails, roadside signs and road surface textures at the bend, which are of great significance for subsequent analysis of the safety status of the bend and the integrity of road facilities. In the open straight area, the resolution is appropriately reduced to improve the scanning efficiency. The scene in the open straight area is relatively simple, and reducing the resolution will not have a significant impact on the acquisition of key information, and it can also reduce the amount of data collection and processing time. For example, on a long straight highway, by lowering the scanning resolution, the laser scanning equipment can complete the scan of the area more quickly, thereby improving the overall inspection efficiency without losing key road information.

[0120] The point cloud acquisition timestamp and spatial coordinates are recorded for the needs of subsequent processing and analysis. The timestamp can record the specific moment of each point cloud data acquisition, which is very important for analyzing the changes of highway facilities over time. For example, by comparing the data of the same point cloud area collected at different times, the wear and deformation of road facilities can be monitored. The spatial coordinates clarify the position of each point in three-dimensional space, providing basic data for building accurate three-dimensional models and conducting spatial analysis. For example, when building a three-dimensional model of a highway, accurate spatial coordinates can ensure the accuracy of the model, allowing technicians to intuitively observe the terrain, slope and spatial layout of the highway facilities. The beneficial effect of this step is that it provides accurate and complete basic data for subsequent point cloud data processing, analysis and fusion with other modal data, which helps to improve the quality and efficiency of highway inspection and maintenance work, and provides strong data support for the maintenance and management of highway facilities.

[0121] Step S2120, calculating the spatiotemporal entropy weight coefficient of each point based on the point cloud acquisition timestamp and spatial coordinates;

[0122] Specifically, entropy is a concept in information theory that measures the uncertainty or confusion of information. The calculation of the spatiotemporal entropy weight coefficient combines the temporal and spatial characteristics of point cloud data, aiming to highlight the information contribution of key points and provide a basis for subsequent entropy value weighted encoding. First, for each point cloud data point, its entropy value in the time dimension and space dimension is calculated according to its timestamp and spatial coordinates. Assume that in the time period Collected within 2 point cloud data points, each point cloud data point has a timestamp of First, divide the time axis into several equally spaced time intervals. , count the frequency of point cloud data points in each time interval is the number of time intervals). Here the frequency The calculation method is, The number of point cloud data points in the time interval divided by the total number of points 2. According to the calculation formula of entropy in information theory , for the entropy value of point cloud data in the time dimension , substitute the frequency of each time interval into the formula for calculation. In the time interval, the frequencies of point cloud data points are , then the entropy value in the time dimension is for:

[0123]

[0124] For the calculation of entropy value in spatial dimension, it is necessary to consider the distribution of point cloud data points in space. Assume that the spatial coordinates of point cloud data points are , is the total number of point cloud data points. First, calculate the spatial distance between each point cloud data point and all other point cloud data points ; Next, determine a spatial neighborhood radius , count each point cloud data point with a radius of The number of points in the neighborhood of . Calculate the density of points in the neighborhood ,in is the volume of the neighborhood; according to the entropy calculation formula, the entropy value in the spatial dimension is .

[0125] The beneficial effects of this step are reflected in many aspects. In terms of data processing, by calculating the spatiotemporal entropy weight coefficient, key points that carry important information can be highlighted and redundant information can be suppressed. For example, in a large amount of point cloud data, some key points may represent key parts or abnormal conditions of road facilities. By giving these points higher weights, more attention can be paid to these points in subsequent processing, improving the efficiency and accuracy of data processing. In terms of highway condition analysis, the prominence of key points helps to more accurately identify problems such as road diseases and facility damage. For example, for a small pothole on the road surface, the corresponding point cloud data may account for a small proportion of the overall data, but through the calculation of the spatiotemporal entropy weight coefficient, the information of these points can be highlighted, making it easier for technicians to find and analyze the problem. In addition, in multimodal data fusion, accurate spatiotemporal entropy weight coefficients can better fuse point cloud data with other modal data (such as visual images and vibration signals), improve the fusion effect, and provide more reliable support for subsequent highway inspection and maintenance image analysis and decision-making.

[0126] Step S2130, extracting surface features of the point cloud data, wherein the surface features include a reflection intensity gradient, a normal vector offset, and a local curvature mutation threshold;

[0127] Specifically, the purpose of step S2130 is to extract the surface features of the point cloud data, which include reflection intensity gradient, normal vector offset and local curvature mutation threshold. These features can further describe the surface characteristics of the point cloud data, provide a basis for the subsequent generation of more targeted point cloud spatiotemporal coding, and help to more accurately reflect the actual situation in the highway inspection area.

[0128] The reflection intensity gradient is used to measure the degree of change in the reflection intensity of each point in the point cloud data. When calculating, for each point in the point cloud data, a suitable neighborhood (for example, with a radius of 1In this neighborhood, the gradient of the reflected intensity is calculated. Assume that point The reflection intensity is , other points in the neighborhood The reflection intensity is , then the reflection intensity gradient The calculation of can be achieved by differentiating the change of reflection intensity in space, generally using the method of numerical difference. For example, in the Cartesian coordinate system, the calculation point exist The gradient component of the reflected intensity in the direction , through the points in the neighborhood and in Direction offset The point (coordinates can be expressed as ,here The reflection intensity of the equal coordinate value is determined based on the coordinate information of the points in the neighborhood. and ; For point The distance in the x direction from the point is the reflection intensity of the point at Δx; For point The distance in the x direction from the point is the reflection intensity of the point at -Δx; calculated using the numerical difference method and The same is true for the direction, and finally the complete reflection intensity gradient is obtained ,in, For point exist The reflected intensity gradient component in the direction, For point exist The reflection intensity gradient component in the direction. The area with large reflection intensity gradient indicates that the reflection intensity changes dramatically, which may correspond to the boundary, material change or other important features of the object surface.

[0129] Normal vector offset is used to describe the difference between the normal vector of a point cloud data point and the normal vector of a reference plane. First, the normal vector of the point cloud data point needs to be determined. For each point, a local plane is formed by fitting the points in its neighborhood, and the normal vector of the plane is the normal vector of the point. Then, a reference plane (such as a horizontal plane or a specific reference plane) is selected to calculate the angle between the normal vector of the point and the normal vector of the reference plane. , normal vector offset Available through To calculate, Yes The normal vector of It is the normal vector of the reference plane. The normal vector offset can reflect the inclination and direction change of the surface where the point cloud data point is located, which helps to identify the concavity, convexity, slope change and other information on the surface of the object.

[0130] The local curvature mutation threshold is used to detect areas in point cloud data where the local curvature changes significantly. The local curvature reflects the curvature of the point cloud data surface. When calculating the local curvature, the neighborhood is also selected with the point as the center, and the points in the neighborhood are fitted to form a quadratic surface. The local curvature is calculated based on the parameters of the quadratic surface. Suppose a point The local curvature of , in its neighborhood, set a threshold ,when ( When the local curvature of other points in the neighborhood is equal to the local curvature of other points in the neighborhood, the point is considered to be in the local curvature mutation area. The determination of the local curvature mutation threshold needs to be adjusted according to the characteristics of the actual point cloud data and application requirements. Generally, the appropriate value is determined through multiple experiments and analysis.

[0131] The extraction of these surface features has many beneficial effects. In highway inspections, the reflection intensity gradient can help identify objects with obvious reflection intensity changes, such as road signs and traffic facilities, because the reflection intensity gradient of these objects is usually large. For example, the reflection intensity of the white markings on the road is different from that of the surrounding road surface, and they can be clearly distinguished by the reflection intensity gradient. The normal vector offset helps to analyze the slope and undulation of the road surface, which is of great significance for judging the safety of the road and the stability of vehicle driving. For example, in the inspection of mountainous roads, the normal vector offset can be used to find sections with large slopes and take corresponding safety measures in advance. The local curvature mutation threshold can detect potholes, bumps and other defects on the road surface, because these defects will cause local curvature mutations. By accurately extracting these surface features, the characteristics of point cloud data can be described more comprehensively and meticulously, providing key information for the subsequent generation of high-quality point cloud spatiotemporal coding, thereby improving the efficiency and accuracy of highway inspection and maintenance work.

[0132] Step S2140, performing entropy value weighted encoding on the point cloud data according to the spatiotemporal entropy weight coefficient of each point and the surface features of the point cloud data, to generate a point cloud spatiotemporal encoding.

[0133] Specifically, the core task of step S2140 is to perform entropy value weighted encoding on the point cloud data according to the spatiotemporal entropy weight coefficient of each point and the surface features of the point cloud data, and generate point cloud spatiotemporal coding, aiming to highlight the information contribution of key points, suppress redundant information, and provide more valuable point cloud data representation for subsequent multimodal fusion and highway inspection and maintenance image analysis.

[0134] When performing entropy weighted encoding, it is first necessary to clarify that the spatiotemporal entropy weight coefficient is calculated based on the point cloud acquisition timestamp and spatial coordinates, which reflects the uniqueness and importance of the information of each point in the time and space dimensions. The surface features of the point cloud data, such as the reflection intensity gradient, normal vector offset, and local curvature mutation threshold, describe the characteristics of the point cloud data surface from different angles. For each point cloud data point, its spatiotemporal entropy weight coefficient and surface features are comprehensively considered. Assume that point The spatiotemporal entropy weight coefficient is , and its reflection intensity gradient is , the normal vector offset is , the local curvature mutation threshold is In order to highlight the information contribution of key points, these features are weighted and fused. The comprehensive feature value is obtained After that, entropy value weighted encoding is performed on each point cloud data point according to its size. The encoding method can adopt a quantization-based method to divide the comprehensive feature value into different level intervals, and each interval corresponds to a specific encoding value. For example, the comprehensive feature value Divide into Interval , C min Represents the comprehensive eigenvalue The minimum value in the partition interval, that is, the comprehensive feature value of all point cloud data points The minimum value in C max Represents the comprehensive eigenvalue The maximum value in the partition interval, that is, the comprehensive feature value of all point cloud data points The maximum value of Falling in the range If the point cloud data point is within In this way, the larger the comprehensive eigenvalue of a point, the more its encoding value can reflect its importance, and it will receive more attention in subsequent processing, thereby achieving the purpose of highlighting key point information. In addition, for points with smaller comprehensive eigenvalues, since they carry relatively less or more redundant information, their importance in the data after encoding is reduced, which plays a role in suppressing redundant information. Through this entropy weighted encoding method, the generated point cloud spatiotemporal encoding can better retain key information, reduce the amount of data, and improve data processing efficiency.

[0135] The beneficial effects of this step are reflected in multiple aspects. In terms of data processing, point cloud spatiotemporal coding reduces the amount of data and reduces storage and transmission costs. For example, in large-scale highway inspection projects, a large amount of point cloud data is significantly reduced after entropy weighted coding, which facilitates data storage and transmission and improves the efficiency of data management. In multimodal fusion, point cloud spatiotemporal coding that highlights key points can be better integrated with other modal data (such as visual images and vibration signals). For example, when fused with visual images, the key point information in the point cloud spatiotemporal coding can better match the key features in the visual image, improve the accuracy and effect of fusion, and provide richer and more accurate information for subsequent highway inspection and maintenance image analysis. In terms of highway inspection and maintenance decision-making, the coded data is easier to analyze and understand, which helps technicians quickly and accurately identify road problems, such as road facility damage and road surface diseases, so as to formulate reasonable maintenance strategies in a timely manner, improve the quality and efficiency of highway inspection and maintenance work, and ensure the safety and normal operation of highways. For example, when analyzing the surface condition of a road, point cloud spatiotemporal coding can be used to quickly locate areas with sudden changes in local curvature, determine whether there are defects such as potholes in the road, and provide an accurate basis for road maintenance.

[0136] Step S2200, obtaining a visual image of the highway inspection area, performing cross-modal correlation analysis on the six-axis vibration signal and the visual image, and obtaining a weighted visual feature map;

[0137] Step S2200 aims to obtain the visual image of the highway inspection area and perform cross-modal correlation analysis with the six-axis vibration signal to obtain a weighted visual feature map, which provides key visual information support for constructing a multi-modal fusion feature vector, thereby improving the accuracy of subsequent highway inspection and maintenance image analysis. In the highway inspection and maintenance scenario, the visual image contains rich information such as road conditions and road facilities, while the six-axis vibration signal reflects the motion state of the equipment. Combining the two can more comprehensively understand the actual situation during the highway inspection process. Through cross-modal correlation analysis, the equipment vibration information can be linked to the specific scene in the visual image.

[0138] Further, step S2200 includes:

[0139] Step S2210, calculate the gradient direction of each pixel point of the visual image to form the gradient direction of the visual image; then calculate the mutual information between each characteristic frequency band of the six-axis vibration signal and the gradient direction of the visual image to obtain the pixel-frequency band mutual information matrix I pf ;

[0140] Specifically, mutual information is a concept in information theory that is used to measure the degree of mutual dependence between two random variables. In this step, by calculating the mutual information, the degree of correlation between the characteristic frequency band of the vibration signal and the gradient direction of the visual image can be quantified.

[0141] For the six-axis vibration signal, the characteristic frequency band related to the movement speed of the maintenance equipment (such as 0.5-20Hz) has been determined in the previous step. For the visual image, the gradient direction reflects the direction of the brightness change in the image, which can highlight the edge and texture information of the image. When calculating the mutual information, it is first necessary to process each characteristic frequency band of the six-axis vibration signal and convert it into a form corresponding to the pixel of the visual image. For example, the amplitude of the vibration signal at each time point can be mapped to the pixel position of the image (assuming that the vibration signal is synchronized with the image acquisition time). Then, for the visual image, the gradient direction of each pixel is calculated. When calculating the gradient direction, a common method is to use an edge detection operator such as the Sobel operator. Taking the Sobel operator as an example, it obtains the horizontal gradient by convolving the image with the convolution kernel in the horizontal and vertical directions. and vertical gradient , and then calculate the gradient amplitude and the gradient direction Through the above calculations, the gradient direction of each pixel of the visual image is obtained, forming the gradient direction of the visual image.

[0142] Next, the mutual information between the characteristic frequency band of the vibration signal and the gradient direction of the visual image is calculated. Assume that the amplitude sequence of a characteristic frequency band of the vibration signal within a certain period of time is , the gradient direction sequence of a pixel point in the visual image is (Here it is assumed that the two time lengths are the same and synchronized), the mutual information The calculation of is based on probability distribution. First, statistics and The joint probability distribution of and their respective marginal probability distributions and Then, the conventional mutual information formula is used according to and calculate The above calculation is performed for all pixels and characteristic frequency bands of vibration signals to obtain the pixel-frequency band mutual information matrix , the elements in the matrix Indicates Pixels and The mutual information between the characteristic frequency bands of the vibration signal.

[0143] The beneficial effect of this step is significant. The pixel-band mutual information matrix obtained by calculating the mutual information , which can accurately reflect the intrinsic connection between vibration signals and visual images. In the subsequent multimodal fusion process, It can serve as an important reference to help determine which pixels are more closely associated with which characteristic frequency bands of the vibration signal. For example, when analyzing road surface defects, if the mutual information between the vibration signal of a certain characteristic frequency band and the pixels in a certain area of ​​the image is large, it means that the area may be closely related to the vibration changes of the equipment, and there may be road surface defects or other abnormal conditions, thereby guiding technicians to analyze these areas more specifically and improve the accuracy and efficiency of highway inspection and maintenance image analysis. In addition, It provides key data for constructing a pixel vibration sensitivity weight model, which helps to achieve more reasonable multimodal fusion.

[0144] Step S2220, constructing the rigidity coefficient m of the maintenance equipment 1 , road roughness n 1 , pixel-band mutual information matrix I pf The nonlinear mapping model w = F(m 1 ,n 1 ,I pf ), where F is a nonlinear function based on a deep neural network;

[0145] Specifically, the rigidity coefficient of the maintenance equipment m 1 It reflects the equipment's ability to resist vibration. It is usually related to the equipment's structural material, mechanical design and other factors, and can be obtained through the equipment's technical parameters. 1 It reflects the unevenness of the road surface and can be measured by on-board sensors (such as laser roughness meters, etc.). Deep neural networks (DNNs) have powerful nonlinear modeling capabilities and can learn complex input-output relationships. 1 ,n 1 ,I pf ), m 1 、n 1 ,I pfAs the input of the network. The structure of the network can include multiple hidden layers, each hidden layer consists of multiple neurons. Neurons are connected by weights, and signals are transformed by nonlinear activation functions (such as ReLU functions) when transmitted between neurons, so that the network can learn the complex nonlinear relationship between input data. When training deep neural networks, a large amount of sample data is required. These sample data include highway inspection data under different equipment rigidity coefficients and road roughness conditions, as well as the corresponding pixel-band mutual information matrix and known pixel vibration sensitivity weights (which can be obtained through expert annotation or other reliable methods). Through the back-propagation algorithm, the weights between neurons in the network are continuously adjusted to minimize the error between the pixel vibration sensitivity weight w output by the network and the known true value. For example, using the mean square error (MSE) as the loss function, through multiple iterative training, the network gradually learns m 1 、n 1 and I pf The nonlinear mapping relationship between and w.

[0146] The beneficial effects of this step are reflected in many aspects. From the perspective of multimodal fusion, the pixel vibration sensitivity weight w calculated by this nonlinear mapping model can more reasonably reflect the impact of equipment, road conditions and the association between vibration and visual images on pixels. For example, when the equipment rigidity coefficient is large, the equipment has strong resistance to vibration, and the amplitude of road vibration transmitted to the equipment is small. Correspondingly, the image pixels are less affected by vibration and the weight w is lower; conversely, when the road roughness is large, the equipment vibration intensifies, the pixels are more affected by vibration, and the weight w is higher. In this way, the weights allocated according to the actual situation enable more accurate fusion of vibration signals and visual image information in the multimodal fusion process, highlighting the visual features related to vibration and improving the accuracy of multimodal fusion. In the analysis of highway inspection and maintenance images, accurate pixel vibration sensitivity weights help to more accurately identify road diseases and abnormal conditions. For example, in the areas of the image that are closely related to vibration (i.e., pixel areas with higher weights), there may be road potholes, cracks and other diseases. By focusing on the analysis of these areas, the accuracy and efficiency of disease detection can be improved, providing a more reliable basis for highway maintenance decisions.

[0147] Step S2230, determine m through the hierarchical analysis process 1 、n 1 ,I pf The influence weight on w is substituted into the model w=F(m 1 ,n 1 ,I pf ), calculating the pixel vibration sensitivity weight of each pixel;

[0148] Specifically, the Analytic Hierarchy Process (AHP) is a decision-making method that decomposes the elements related to decision-making into goals, criteria, plans, etc., and conducts qualitative and quantitative analysis on this basis. In this step, AHP is used to determine the rigidity coefficient of the maintenance equipment. , road roughness , pixel-band mutual information matrix Weighting of pixel vibration sensitivity the relative importance of .

[0149] First, a hierarchical model is constructed. The pixel vibration sensitivity weights are calculated As the target layer; As the criterion layer; and each pixel point as the solution layer. Next, construct the judgment matrix. The judgment matrix is ​​the key to AHP, which reflects the decision maker's judgment on the relative importance of each factor. For each two factors in the criterion layer, determine their relative importance to the target layer (calculate the pixel vibration sensitivity weight) by pairwise comparison. ) is the relative importance of and right If we think Compare right The influence of is slightly important. According to the 1-9 scale of AHP (where 1 means that the two factors are equally important, 3 means that one factor is slightly more important than the other, 5 means obviously important, 7 means strongly important, 9 means extremely important, and 2, 4, 6, and 8 represent intermediate values ​​between adjacent judgments), the corresponding element in the judgment matrix is ​​assigned a value of 3; otherwise, and When comparing, the corresponding elements are assigned This will give a The judgment matrix '',in Indicates The factor relative to The importance of each factor to the target layer. Then, the eigenvector and maximum eigenvalue of the judgment matrix are calculated. The eigenvector of can be used to obtain the relative weight vector of each factor. The specific calculation method is to solve the equation ,in is a matrix The maximum eigenvalue of . Generally, the characteristic root method or sum-product method is used for calculation. Finally, the calculated weight vector Substitute into the model w=F(m 1 ,n 1,I pf ). Assume that F is a specific function based on a deep neural network. For each pixel, The corresponding weight Perform weighted calculation (the calculation method here depends on The specific form of the AHP may be more complicated in practice and involve specific operations of the neural network), thereby obtaining the pixel vibration sensitivity weight of each pixel. The beneficial effects of this step are reflected in many aspects. From the perspective of model accuracy, the weight determined by AHP can more scientifically reflect the right The accuracy of the pixel vibration sensitivity weights can help to more accurately locate and analyze abnormal conditions such as pavement defects. For example, for defects such as pavement cracks, the pixel vibration sensitivity weights with reasonable weights can more accurately highlight these defect areas in the visual image, providing a more reliable basis for subsequent defect assessment and maintenance decisions.

[0150] Step S2240, extracting features from the visual image of the highway inspection area to obtain a visual feature map;

[0151] Specifically, the visual feature map contains rich semantic information in the image and is an important basis for subsequent multimodal fusion and image analysis. In this step, a convolutional neural network (CNN) is used to extract features from the visual image. A convolutional neural network is a deep learning model designed specifically for processing image data. It automatically extracts image features through a series of convolutional layers, pooling layers, and activation functions. Before feature extraction, the visual image needs to be preprocessed, including grayscale (if it is a color image) and normalization. Grayscale is to convert a color image into a grayscale image, reducing the amount of data and facilitating subsequent processing; normalization is to map the pixel values ​​of the image to a specific range (such as [0,1] or [−1,1]), so that different images are comparable, which helps to improve the training effect and stability of the model. The preprocessed image is input into the pre-built convolutional neural network. The convolution layer is one of the core components of CNN, which performs convolution operations by sliding the convolution kernel on the image. The convolution kernel is a small matrix, usually 3×3 or 5×5 in size. The process of convolution operation is to multiply the convolution kernel by the local area of ​​the image and sum them to obtain the feature map after convolution. For example, for a 3×3 convolution kernel and a 3×3 local area in the image, each element of the convolution kernel is multiplied by the element of the corresponding image area, and then these products are added to obtain a pixel value at the corresponding position in the feature map after convolution. Different types of features in the image, such as edges, textures, etc., can be extracted through different convolution kernels. The convolution layer usually has multiple convolution kernels, thereby generating multiple feature maps, each of which corresponds to a specific image feature.

[0152] After the convolution layer, there is usually an activation function layer. The activation function introduces nonlinear characteristics to the neural network, allowing the network to learn more complex functional relationships. Common activation functions include ReLU function and Sigmoid function. Taking the ReLU function as an example, it sets the values ​​less than 0 output by the convolution layer to 0, and the values ​​greater than 0 remain unchanged, which can effectively avoid the gradient vanishing problem and speed up the training of the network. The pooling layer is also an important part of CNN. It is mainly used to reduce the dimension of the feature map, reduce the amount of calculation and prevent overfitting. Common pooling methods include maximum pooling and average pooling. Maximum pooling selects the maximum value in a local area (such as a 2×2 area) as the output after pooling; average pooling calculates the average value of the local area as the output. For example, when performing maximum pooling in a 2×2 area, the maximum pixel value in the area is selected as the result after pooling. Through the alternating operation of multiple convolution layers, activation function layers and pooling layers, the convolutional neural network gradually extracts semantic information at different levels of the image. Shallow convolutional layers can extract low-level features of the image, such as edges and lines, while deep convolutional layers can extract higher-level semantic features, such as the shape and structure of objects. Ultimately, the network outputs visual feature maps with different levels of semantic information.

[0153] The beneficial effects of this step are reflected in multiple levels. In terms of multimodal fusion, the visual feature map provides rich visual information for subsequent fusion with other modal data such as vibration signals and point cloud data. For example, when fusing the visual feature map with the vibration frequency domain fingerprint map, the edges, textures and other features in the visual feature map can be associated with the relevant features in the vibration signal to enhance the effect of multimodal fusion. In the image analysis of highway inspection and maintenance, the visual feature map helps to more accurately identify targets such as pavement diseases and road facilities. For example, the features extracted by the convolutional neural network can more clearly distinguish the shape, size and location of pavement cracks, potholes and other diseases, thereby improving the accuracy of disease detection. In addition, the visual feature map can also be used as input for subsequent image classification, target recognition and other tasks, providing more comprehensive support for highway inspection and maintenance work, and improving the scientificity and accuracy of highway maintenance decisions.

[0154] Step S2250, multiplying the pixel vibration sensitivity weight of each pixel point by the visual feature map element by element to obtain a weighted visual feature map.

[0155] Specifically, the purpose of this step is to weight each pixel in the visual feature map according to the pixel vibration sensitivity weight, highlight the visual features related to device vibration, and suppress the information unrelated to vibration, so as to provide more targeted visual information for subsequent multimodal fusion and highway inspection and maintenance image analysis. Before performing the element-wise multiplication, the pixel vibration sensitivity weight w of each pixel point and the visual feature map have been calculated through the previous steps. The pixel vibration sensitivity weight w reflects the degree to which each pixel is affected by device vibration, and its value range is usually within a certain interval, such as [0, 1]. The larger the value, the greater the influence of the pixel by vibration and the closer the association with vibration. The visual feature map is obtained by the convolutional neural network extracting features from the visual image. It contains rich image semantic information, and each pixel value represents the intensity of the image features at the corresponding position.

[0156] This step has beneficial effects in many aspects. From the perspective of multimodal fusion, the weighted visual feature map can be better fused with other modal data (such as vibration frequency domain fingerprint spectrum, point cloud spatio-temporal encoding). Since the visual features related to vibration are highlighted, when constructing the multimodal fusion feature vector subsequently, these features can interact and fuse with other modal information more effectively, improving the accuracy and effectiveness of multimodal fusion. For example, when analyzing highway pavement diseases, areas such as pavement cracks that are closely related to vibration are strengthened in the weighted visual feature map. When fused with the vibration frequency domain fingerprint spectrum, it can more accurately reflect the relationship between device vibration and pavement diseases. In terms of highway inspection and maintenance image analysis, the weighted visual feature map helps to improve the recognition accuracy of pavement diseases and abnormalities. Because the visual features related to vibration are enhanced, technicians can more clearly observe the areas that may have problems when analyzing the image. For example, when detecting pavement potholes, since the pothole areas often cause device vibration, the corresponding pixels in the visual feature map are weighted, and the features are more obvious, facilitating technicians to quickly discover and locate these disease areas, providing a more reliable basis for highway maintenance decisions. In addition, the weighted visual feature map can also reduce the interference of background information unrelated to vibration, enabling subsequent image analysis to focus more on key areas, improving the analysis efficiency, helping to timely discover potential highway safety hazards, and ensuring the normal operation and traffic safety of highways.

[0157] Step S2300: Construct a multimodal fusion feature vector based on the calibrated vibration frequency domain fingerprint spectrum, point cloud spatio-temporal encoding, and weighted visual feature map.

[0158] Step S2300 aims to construct a multimodal fusion feature vector by fusing the calibrated vibration frequency domain fingerprint map, point cloud spatiotemporal coding and weighted visual feature map, so as to provide key data support for the subsequent generation of high-quality highway inspection and maintenance enhanced images. In the highway inspection and maintenance scenario, single-modal data often cannot fully and accurately reflect the actual situation. Multimodal fusion can integrate the advantages of different data sources and improve the accuracy and reliability of highway condition analysis. The vibration frequency domain fingerprint map reflects the vibration characteristics of the maintenance equipment, the point cloud spatiotemporal coding contains the spatial structure information of the highway inspection area, and the weighted visual feature map presents the visual details of the road surface and the surrounding environment. The fusion of these three modal data can provide richer and more comprehensive information for highway inspection and maintenance image analysis.

[0159] Furthermore, if Figure 7 As shown, step S2300 includes:

[0160] Step S2310, aligning the calibrated vibration frequency domain fingerprint, the point cloud spatiotemporal encoding, and the weighted visual feature map in a common reference coordinate system to form a three-dimensional heterogeneous modal feature tensor T; the three-dimensional heterogeneous modal feature tensor T has three channels, and the three channels are respectively a visual sub-tensor, a vibration sub-tensor, and a point cloud sub-tensor;

[0161] Specifically, this step is the basis of multimodal fusion. By integrating data of different modalities in the same coordinate system, the data can be processed and analyzed uniformly in the future. The common reference coordinate system is a unified coordinate framework used to determine the spatial position relationship of data of different modalities. In actual operation, it is first necessary to determine the coordinate transformation relationship of each modal data in the common reference coordinate system. For the vibration frequency domain fingerprint spectrum, it does not directly correspond to the spatial coordinates, but its position relationship in the common reference coordinate system can be indirectly determined by associating it with the motion state of the maintenance equipment and the acquisition position information. For example, assuming that the driving trajectory of the maintenance equipment on the road is known, the data in the vibration frequency domain fingerprint spectrum can be corresponded to the specific position on the road through the positioning information of the equipment (such as GPS data) and the time sequence of vibration signal acquisition. The point cloud spatiotemporal encoding contains the timestamp and spatial coordinate information of the point cloud acquisition. When converting it to the common reference coordinate system, it is necessary to perform coordinate transformation according to the parameters of the acquisition equipment and the characteristics of the acquisition scene. For example, if the installation position and posture of the laser scanning device are known, the original coordinates of the point cloud data can be converted to the common reference coordinate system through the corresponding rotation and translation transformation matrix. The weighted visual feature map is usually generated based on the image acquired by the image acquisition device, and its coordinate system is related to the image pixel position. During the alignment process, coordinate transformation is required based on the parameters of the image acquisition device (such as focal length, viewing angle, etc.) and the relative position relationship with the common reference coordinate system. For example, through technologies such as camera calibration, the mapping relationship between the image pixel coordinates and the common reference coordinate system can be determined, and each pixel point in the visual feature map can be mapped to the corresponding position in the common reference coordinate system. After completing the coordinate transformation, the calibrated vibration frequency domain fingerprint map, point cloud spatiotemporal encoding, and weighted visual feature map are combined according to the channel dimension to form a three-dimensional heterogeneous modal feature tensor T.

[0162] The beneficial effects of this step are significant. From the perspective of data processing, aligning data of different modes in a common reference coordinate system makes the data consistent in space, which is convenient for subsequent unified processing and analysis. For example, when performing feature extraction and fusion operations, calculations can be performed based on a unified coordinate system to improve calculation efficiency and accuracy. In terms of multimodal fusion, this alignment method provides a basis for the interaction and fusion of data of different modes. By using the same coordinate system, data of different modes can be more effectively associated and fused, potential connections between data can be mined, and the effect of multimodal fusion can be improved. For example, when analyzing road pavement diseases, the aligned point cloud data and visual feature maps can more accurately match the spatial structure and visual features of the road surface, thereby more clearly identifying the location and shape of the disease. In addition, in subsequent image generation and analysis, multimodal data in a unified coordinate system can provide strong support for generating more realistic and more realistic images, which helps to improve the accuracy and reliability of highway inspection and maintenance image analysis.

[0163] Step S2320, using an adaptive channel attention mechanism to dynamically assign weights to the visual sub-tensor, vibration sub-tensor, and point cloud sub-tensor of the three-dimensional heterogeneous modal feature tensor T to obtain a weighted feature tensor;

[0164] Specifically, the weighted feature tensor is used to improve the effect of multimodal fusion. The adaptive channel attention mechanism is a method that can automatically adjust the channel weight according to the importance of each channel feature, so that the model can pay more attention to important feature information and suppress unimportant information.

[0165] When implementing the adaptive channel attention mechanism, we first calculate the mean and variance of the features on each channel. For the visual subtensor, assume that its feature representation is , the size is ( Indicates height, Indicates width, Indicates the number of channels , then its mean The calculation method is: ;in, For height The index of For width The index of Indicates that at position ( , ) at the channel vector. Variance The calculation method is: For the vibration subtensor and point cloud subtensor, similar methods are used to calculate the mean and variance.

[0166] The importance of the channel feature is judged based on the size of the mean and variance. Generally speaking, a channel with a larger mean and variance means that the channel contains more important information, which may be more critical to accurately describe highway inspection and maintenance scenarios in multimodal fusion. For example, when analyzing road surface defects, if the mean and variance of a channel in the visual subtensor in the defect area are large, it means that the channel may contain key visual features of the defect, such as the edge information of cracks or the texture features of potholes. The weight of each channel is calculated based on the mean and variance. Assume that the weight vector of all channels is ,in , They correspond to the weights of the visual sub-tensor, vibration sub-tensor and point cloud sub-tensor respectively. Taking the weight of the visual sub-tensor as an example, the weight is calculated using the following formula: ;in, represents the oscillator tensor in the three-dimensional heterogeneous modal eigentensor T, Represents the point cloud subtensor in the three-dimensional heterogeneous modal feature tensor T, is a A set of three elements; yes The index used to traverse the collection The elements in . Is a follow index The mean of the variable changes, The value comes from the collection , for example, when = hour, Represents the visual subtensor The mean of The same is true, it is a random index After obtaining the weight of each channel, each sub-tensor of the three-dimensional heterogeneous modal feature tensor T is weighted to obtain the weighted feature tensor.

[0167] The beneficial effects of this step are reflected in multiple levels. In terms of multimodal fusion, through the adaptive channel attention mechanism, the model can automatically focus on important feature channels, enhance the use of key information, suppress redundant information, and thus improve the quality of multimodal fusion. For example, when fusing vibration, point cloud and visual information, for pavement disease analysis, it can highlight the features related to the disease, such as the features reflecting abnormal vibration of the equipment in the vibration sub-tensor, the spatial structural features of the disease area in the point cloud sub-tensor, and the texture features of the disease in the visual sub-tensor, making the fused features more representative. In the analysis of highway inspection and maintenance images, the weighted feature tensor helps to more accurately identify targets such as pavement diseases and road facilities. Because the important features are enhanced, technicians can more clearly observe the details and features of the target when analyzing the image, improving the accuracy and efficiency of disease detection. In addition, this dynamic weight distribution method can also improve the adaptability of the model, so that it can more effectively fuse multimodal data in different highway inspection scenarios, providing a more reliable basis for highway maintenance decisions.

[0168] Step S2330, superimpose the spatiotemporal entropy weight mask, perform pooling processing on the weighted feature tensor, and generate a multimodal fusion feature vector.

[0169] Specifically, this step aims to further optimize the results of multimodal fusion, by suppressing redundant information and highlighting key information, to generate a multimodal fusion feature vector with stronger representation ability, and to provide high-quality data input for subsequent adversarial generative network training and highway inspection and maintenance image analysis. The spatiotemporal entropy weight mask is generated based on the spatiotemporal entropy information of the point cloud data. In step S2120, the spatiotemporal entropy weight coefficients of each point cloud data point have been calculated, which reflect the information importance of the point cloud data in the time and space dimensions. The spatiotemporal entropy weight mask is to extend these weight coefficients to the same dimension and structure as the weighted feature tensor, so that the element at each position corresponds to a spatiotemporal entropy weight value. The operation of superimposing the spatiotemporal entropy weight mask is to multiply the weighted feature tensor by the spatiotemporal entropy weight mask element by element to obtain the feature tensor after superimposing the mask. In this way, positions with higher entropy values ​​(i.e., more important information) in time and space are enhanced, while positions with lower entropy values ​​(relatively redundant information) are suppressed. For example, in a highway inspection scenario, if the point cloud data of a certain area has a higher entropy value in time and space, it means that the area contains important information, such as key parts of road facilities or diseased areas. After superimposing the spatiotemporal entropy weight mask, the features of these areas in the feature tensor will be enhanced.

[0170] Pooling is a commonly used dimensionality reduction operation that aims to reduce the dimension of data while retaining important feature information. In this step, the feature tensor T after superimposing the spatiotemporal entropy weight mask is maskedPerform pooling. Common pooling methods include maximum pooling and average pooling. Through pooling, the dimension of the feature tensor is reduced to generate a multi-modal fusion feature vector.

[0171] The beneficial effects of this step are reflected in many aspects. In terms of data processing, superimposing spatiotemporal entropy weight masking and pooling processing can effectively suppress redundant information, reduce the amount of data, reduce computational complexity, and improve the efficiency of subsequent model training and analysis. For example, when processing large-scale highway inspection data, the amount of data after these operations is greatly reduced, while key information is retained, so that the subsequent adversarial generative network training can converge faster. In terms of multimodal fusion, by highlighting key information and suppressing redundant information, the generated multimodal fusion feature vector has a stronger representation ability and can more accurately reflect the characteristics of highway inspection and maintenance scenes. For example, when analyzing pavement diseases, the multimodal fusion feature vector can more concentratedly reflect the key features of the disease, which helps to improve the accuracy of disease identification. In highway inspection and maintenance image analysis, this high-quality multimodal fusion feature vector provides a solid foundation for subsequent image generation and analysis, can generate more realistic and detailed enhanced images, improve the accuracy of highway inspection and maintenance status analysis, and provide a more reliable basis for highway maintenance decision-making.

[0172] Step S3000: construct and train a generative adversarial network including a generator and a discriminator based on the multimodal fusion feature vector; obtain a new multimodal fusion feature vector, and generate an enhanced highway inspection and maintenance image based on the new multimodal fusion feature vector and the trained generative adversarial network.

[0173] Furthermore, step S3000 includes:

[0174] Step S3100, constructing and training a generative adversarial network including a generator and a discriminator;

[0175] The core purpose of step S3100 is to build and train a generative adversarial network containing a generator and a discriminator, which is a key link in generating high-quality highway inspection and maintenance enhanced images. The generative adversarial network (GAN) consists of a generator and a discriminator, which are mutually adversarial and trained collaboratively. Through this mechanism, the distribution of real data can be learned to generate more realistic image data. In the application scenario of highway inspection and maintenance images, since the actual collected images may have problems such as insufficient data volume and uneven image quality, the generative adversarial network can be used to enhance the existing image data and generate more images with rich details and features, providing more sufficient and high-quality data support for subsequent highway condition analysis.

[0176] Further, step S3100 includes:

[0177] Step S3110, constructing a generator of a generative adversarial network;

[0178] Specifically, step S3110 aims to construct a generator of a generative adversarial network, select a multi-layer convolutional neural network (CNN) structure, and optimize the design of multi-modal fusion feature vectors to achieve the transformation from multi-modal fusion feature vectors to high-quality highway inspection and maintenance images. Multi-layer convolutional neural networks have powerful feature extraction and image generation capabilities in the field of image processing. Through the combination of components such as convolutional layers, activation function layers, and pooling layers, they can automatically learn the feature representation of images.

[0179] When constructing the convolution layer of the generator, the DCGAN (Deep Convolution Generative Adversarial Network) architecture is referenced and adjusted according to the characteristics of the multimodal fusion feature vector. The DCGAN architecture is a convolutional neural network architecture that has been successfully applied to image generation. It maps low-dimensional vectors to high-resolution image space through a series of transposed convolution operations. For the multimodal fusion feature vector in this embodiment, it contains information such as the calibrated vibration frequency domain fingerprint spectrum, point cloud spatiotemporal encoding, and weighted visual feature map. These information have different modal characteristics, and the convolution kernel size and step size need to be set in a targeted manner. For the vibration frequency domain fingerprint spectrum features, because they contain fine frequency-related information, a smaller convolution kernel (such as 3×3) is used. A smaller convolution kernel can capture the changes in frequency features more finely in a local area, for example, it can accurately extract detailed information such as changes in energy distribution in a specific frequency band. Assuming that the energy change of the vibration frequency domain fingerprint spectrum in a certain frequency band presents a local peak feature, a smaller convolution kernel can more keenly perceive this change, so that when generating an image, it can more accurately reflect the image details related to the frequency feature, such as the performance of the equipment vibration in the image caused by the corresponding road condition. For point cloud spatiotemporal coding, since the spatial structural features of point cloud data are relatively macroscopic, a slightly larger convolution kernel (such as 5×5) is used. A larger convolution kernel can capture the structural information of point cloud data in a wider spatial range, which helps to better restore the spatial layout of the scene represented by the point cloud when generating images. For example, when processing point cloud data containing roads and surrounding facilities, a larger convolution kernel can integrate point cloud information in a wider area, making the spatial structural features such as the shape, width and relative position relationship of the road with surrounding facilities in the generated image more accurate. For the weighted visual feature map, a suitable convolution kernel (such as 4×4) is selected in combination with the texture and detail characteristics of the image. A convolution kernel of this size can not only capture the local texture details of the image, but also take into account the contextual information of the surrounding pixels to a certain extent. For example, when generating an image containing road surface texture, a 4×4 convolution kernel can effectively extract local features of the texture, such as the texture direction of road cracks, the edge features of potholes, etc., while taking into account the information of surrounding pixels, making the generated image texture more natural and coherent.

[0180] In this way, the input layer of the generator receives the multimodal fusion feature vector constructed from step S2000, and each convolution layer gradually processes and transforms these features, and finally generates a highway inspection and maintenance image with high resolution. The beneficial effect of this step is significant. In terms of image generation quality, the targeted convolution kernel design enables the generator to fully tap the information in the multimodal fusion feature vector, and the generated image is richer and more accurate in details and features. For example, the generated image can more realistically present the texture of the road surface, the morphology of the disease, and the details of the road facilities, improving the visualization and readability of the image. In terms of multimodal fusion applications, this design helps to better integrate data features of different modes, so that the generated image not only contains visual information, but also reflects the information reflected by vibration and point cloud data, enhancing the overall expression ability of the image for highway inspection and maintenance scenes, and providing a more comprehensive and accurate data basis for subsequent image analysis.

[0181] Step S3120, embedding a vibration-blur conversion layer in the front end of the generator of the adversarial generative network;

[0182] Specifically, the main task of step S3120 is to embed a vibration-fuzzy conversion layer in the front end of the generator of the adversarial generation network. This layer works according to the vibration frequency domain fingerprint spectrum calibrated in step S1000. The purpose is to generate a motion blur effect that conforms to the actual situation according to the vibration state of the equipment, so that the generated highway inspection and maintenance image is more realistic. In the working process, the motion state parameters of the equipment at different times, such as acceleration, angular velocity, etc., are first calculated according to the characteristic information in the vibration frequency domain fingerprint spectrum. The vibration frequency domain fingerprint spectrum contains rich information about the vibration of the equipment. By analyzing its main frequency band energy distribution, harmonic distortion rate, and inter-axis coupling coefficient, the motion state of the equipment at different times can be inferred. For example, according to the change of vibration energy in a specific frequency band, it can be judged whether the equipment is in an acceleration, deceleration or uniform motion state; through the inter-axis coupling coefficient, the vibration correlation of the equipment in different directions can be understood, and then the change of its motion posture can be inferred. After obtaining the motion state parameters of the equipment, these parameters are used as input, combined with the Newton-Euler kinematic model, and the fuzzy kernel function is dynamically generated. The Newton-Euler kinematics model is a classical mechanics model that describes the motion of an object. It can calculate the motion trajectory and posture changes of an object in space based on parameters such as the object's acceleration and angular velocity. In this step, the model is used to simulate the vibration and displacement of the device during motion, thereby generating a fuzzy kernel function that matches the actual motion state of the device.

[0183] The generation process of the blur kernel function takes into account the direction, speed and vibration of the device. When the device moves faster and the vibration amplitude is larger, the size of the generated blur kernel increases accordingly to simulate a more obvious motion blur effect. For example, when the device is traveling at high speed and the road is bumpy, resulting in large vibrations, the image will be blurred to a greater extent. At this time, the size of the generated blur kernel is larger, so that the generated image presents a more obvious blur effect in the corresponding area, more realistically reflecting the actual shooting situation. Conversely, when the device moves slower and the vibration is smaller, the blur kernel size decreases. In order to realize the linkage between the kernel size and the device speed error threshold, a speed error threshold range is set. When the calculated device speed error exceeds the threshold, the size of the blur kernel is adjusted according to a certain proportional relationship. Assuming that the speed error threshold range is [-ε,ε], when the calculated device speed error is greater than ε, according to the pre-set proportional coefficient k z , increase the size of the blur kernel; when the velocity error is less than -ε, also according to the proportional coefficient k z Reduce the size of the blur kernel. This ensures that the generated blur closely matches the actual motion of the device, improving the realism of the generated image.

[0184] The beneficial effects of this step are reflected in multiple aspects. In terms of image authenticity, the motion blur effect generated by the vibration-blur conversion layer makes the generated highway inspection and maintenance images more consistent with the actual shooting scene. For example, when analyzing images taken by maintenance equipment traveling on the road, the real image will produce a certain degree of blur due to the movement of the equipment. The blur effect simulated by the conversion layer can restore this real situation, which helps subsequent analysts to more accurately judge the information in the image. In terms of image analysis accuracy, reasonable motion blur simulation can avoid analysis errors caused by ignoring blur factors. For example, when detecting road surface diseases, if the image does not correctly simulate motion blur, it may cause deviations in the edges and details of the disease, affecting the accuracy of the detection. The real blurred image generated by this step can improve the accuracy of disease detection and other image analysis tasks, and provide a more reliable basis for highway maintenance decisions.

[0185] Step S3130, constructing a discriminator of the generative adversarial network, wherein the discriminator adopts a dual-stream verification architecture, and the dual-stream verification architecture includes a video stream part and a point cloud stream part.

[0186] Specifically, the core of step S3130 is to construct a discriminator of the adversarial generative network. The discriminator adopts a two-stream verification architecture, including a video stream part and a point cloud stream part. The purpose is to more accurately judge the quality of the generated image through a multi-dimensional evaluation method, and then guide the generator to generate a highway inspection and maintenance image that is closer to the real one. In the visual stream part of the discriminator, an asymmetric wavelet decomposition loss function is used to evaluate the texture similarity between the generated image and the real image. Wavelet decomposition is a mathematical method that decomposes a signal into sub-bands of different frequencies. Asymmetric wavelet decomposition can more effectively extract the texture features of the image. During operation, the generated image and the real image are first subjected to asymmetric wavelet decomposition respectively to decompose the image into sub-bands of different frequencies. It is very important to select a suitable wavelet basis function (such as db4 wavelet basis), which determines the effect of wavelet decomposition on image texture feature extraction. When processing image texture, the db4 wavelet basis can better capture the edge and texture information of different scales in the image. For each sub-band, the difference between the generated image and the corresponding sub-band of the real image is calculated. Here, the mean square error (MSE) is used to measure the pixel difference between sub-bands. The mean square error quantifies the degree of difference between two images by calculating the average of the sum of the squares of the differences between the corresponding pixel values ​​of the two images. Then, a weight is assigned to the difference of each sub-band according to its importance to the image texture. High-frequency sub-bands usually contain detailed texture information of the image, such as cracks on the road surface, details of the signs, etc. This information is crucial for judging the authenticity and quality of the image, so it is given a higher weight; low-frequency sub-bands mainly reflect the general outline of the image, such as the overall shape of the road, regional distribution, etc., and the weight is relatively low. Finally, the weighted differences of all sub-bands are summed to obtain the loss value of the visual flow, which is used to evaluate the texture similarity between the generated image and the real image.

[0187] In the point cloud flow part, a reflection-shadow joint probability model is constructed based on Markov random fields to verify the topological consistency between the generated image shadow area and the point cloud reflection intensity distribution. First, the reflection intensity information in the point cloud data is associated with the shadow area of ​​the generated image. For each pixel in the generated image, its corresponding position in the point cloud data is determined, and the reflection intensity value of the position is obtained. The point cloud data records the three-dimensional coordinates and reflection intensity of the surface of the object in the scene. By corresponding the generated image pixels with the point cloud data, the connection between the two can be established. Then, the Markov random field model is used to consider the spatial neighborhood relationship between the pixels to construct the reflection-shadow joint probability model. Markov random field is a probability-based model that assumes that the state of a pixel is only related to the state of its neighboring pixels. In the process of model construction, the parameters of the model, such as the potential function of the node and the weight of the edge, are determined to describe the mutual influence relationship between the pixels. The potential function defines the state energy of a single pixel, and the weight of the edge represents the interaction strength between adjacent pixels. The topological consistency of the two is evaluated by calculating the joint probability of the generated image shadow area and the point cloud reflection intensity distribution under the model. If the joint probability is high, it means that the topological consistency between the generated image shadow area and the point cloud reflection intensity distribution is good; otherwise, the consistency is poor.

[0188] The evaluation results of the visual flow and the point cloud flow are fused to obtain the final discriminant result of the discriminator. The fusion method adopts the weighted summation method, and different weights are assigned to the visual flow loss value and the point cloud flow joint probability according to the importance of visual texture and point cloud topological consistency in the actual application scenario. For example, in scenes that focus on image texture details, such as accurate detection of road surface diseases, the accuracy of road surface texture is crucial to determine the type and degree of disease. At this time, the weight of the visual flow loss value can be appropriately increased; in scenes with high requirements for scene structure accuracy, such as analyzing the spatial layout of road facilities, point cloud topological consistency is more critical, and the weight of the point cloud flow joint probability should be increased. By adjusting the weight, the discriminator can more accurately judge the quality of the generated image.

[0189] The beneficial effects of this step are significant. In terms of image quality assessment, through the dual evaluation of visual flow and point cloud flow, the discriminator can comprehensively judge the quality of the generated image from multiple angles, which is more accurate and reliable than a single evaluation method. For example, relying solely on the texture evaluation of visual images may ignore the spatial structural information of objects in the image, while combining the topological consistency evaluation of point cloud data can more comprehensively judge the authenticity of the image. In terms of generator training guidance, accurate discrimination results can provide more effective feedback for the training of the generator, prompting the generator to continuously adjust the strategy of generating images, generate highway inspection and maintenance images that are closer to the actual situation, and improve the quality and practicality of the generated images. In terms of highway inspection and maintenance image analysis, high-quality generated images help to more accurately identify targets such as pavement diseases and road facilities, improve the accuracy and efficiency of analysis, and provide a more reliable basis for highway maintenance decisions.

[0190] Step S3140: training the constructed generative adversarial network according to the multimodal fusion feature vector.

[0191] Specifically, the main task of step S3140 is to train the constructed adversarial generative network according to the multimodal fusion feature vector. Through the continuous confrontation and optimization of the generator and the discriminator, the image generated by the generator is closer and closer to the real highway inspection and maintenance image, so as to achieve the purpose of adversarial training. During the training process, the multimodal fusion feature vector is used as the input of the generator. The multimodal fusion feature vector contains rich information such as the calibrated vibration frequency domain fingerprint spectrum, the point cloud spatiotemporal coding and the weighted visual feature map, which provides the basis for the generator to generate high-quality images. The generator generates a simulated highway inspection and maintenance image based on the input feature vector. Since the generator is constructed based on a multi-layer convolutional neural network structure, it converts the abstract feature vector into an image with a certain resolution through the gradual processing and conversion of the multimodal fusion feature vector under the action of components such as the convolution layer and the activation function layer.

[0192] The discriminator receives the generated simulated images and real highway inspection and maintenance images. Among them, the video stream part analyzes the dynamic features of the image and evaluates the texture similarity between the generated image and the real image through the asymmetric wavelet decomposition loss function. As mentioned earlier, asymmetric wavelet decomposition decomposes the image into sub-bands of different frequencies. By calculating the mean square error between the sub-bands and combining the weights of different sub-bands, the loss value of the visual stream is obtained to measure the difference in texture between the generated image and the real image. The point cloud stream part judges the authenticity of the image from the perspective of the relevant features of the point cloud data. Based on the reflection-shadow joint probability model constructed based on the Markov random field, the topological consistency of the shadow area of ​​the generated image and the point cloud reflection intensity distribution is verified, and the joint probability is calculated to evaluate the consistency between the two.

[0193] By continuously adjusting the parameters of the generator and the discriminator, the images generated by the generator are getting closer and closer to the real images. In the early stages of training, the images generated by the generator may be quite different from the real images. The discriminator can easily identify these differences and give a lower evaluation score (for the judgment of the quality of the generated image, the evaluation results such as the visual flow loss value and the point cloud flow joint probability can be mapped to an evaluation score through a certain function). At this time, according to the feedback from the discriminator, the generator adjusts its own parameters through the back propagation algorithm. The back propagation algorithm is an optimization algorithm for training neural networks. It calculates the gradient of the loss function to the network parameters, propagates the gradient information from the output layer to the input layer, and gradually adjusts the weights and biases of each layer in the network, so that the images generated by the generator are closer to the real images in subsequent iterations.

[0194] At the same time, the discriminator is also constantly optimizing its own parameters to better identify the difference between the generated images and the real images. For example, in each training iteration, the discriminator updates its internal parameters based on the new generated images and real images to improve the accuracy of image quality judgment. As the training progresses, the generator gradually learns the characteristics and distribution patterns of the real images, the quality of the generated images continues to improve, and the difficulty of the discriminator's judgment gradually increases. When the images generated by the generator can deceive the discriminator, making it difficult to distinguish the difference between the generated images and the real images, the purpose of adversarial training is achieved.

[0195] The beneficial effects of this step are reflected in many aspects. In terms of improving the quality of image generation, through adversarial training, the generator can learn the rich features and distribution patterns of real images, and the generated highway inspection and maintenance images are getting closer and closer to the real situation in terms of details, texture, structure, etc. For example, the generated images can more accurately present the shape, size and location of road diseases, as well as the appearance and status of road facilities, improving the authenticity and usability of the images. In terms of model adaptability enhancement, the training process of the adversarial generative network enables the model to adapt to different highway inspection and maintenance scenarios and data characteristics. Since the actual collected data may be diverse and complex, through adversarial training, the generator and discriminator can be continuously adjusted and optimized to improve the model's processing ability and adaptability to various data. In the application of highway inspection and maintenance image analysis, high-quality generated images provide more reliable data support for subsequent analysis tasks. For example, in tasks such as image-based pavement disease detection and road facility evaluation, the generated enhanced images can help the detection model learn more comprehensive features, improve the accuracy and efficiency of detection, and provide a more accurate basis for highway maintenance decisions.

[0196] Step S3200: Use the trained generative adversarial network and input a new multimodal fusion feature vector to generate an enhanced highway inspection and maintenance image.

[0197] Specifically, the purpose of step S3200 is to use the trained adversarial generative network, input new multimodal fusion feature vectors, and generate enhanced highway inspection and maintenance images. These enhanced images are intended to provide richer and more accurate information for the subsequent analysis of highway inspection and maintenance conditions. When executing this step, first obtain new multimodal fusion feature vectors. These new feature vectors also contain information such as calibrated vibration frequency domain fingerprints, point cloud spatiotemporal coding, and weighted visual feature maps. They may come from highway inspection data at different times and different sections, or they may be obtained after different processing of existing data. For example, in a new highway inspection, the collected point cloud data, visual images, and corresponding equipment vibration data undergo the same processing flow as before, that is, the point cloud data is entropy weighted encoded, and the visual image is subjected to cross-modal correlation analysis and other operations to obtain a new multimodal fusion feature vector.

[0198] The new multimodal fusion feature vector is input into the generator of the trained adversarial generative network. Since the generator has learned how to generate patterns and rules close to the real highway inspection and maintenance images from the multimodal fusion feature vector in the previous training process, it can now generate enhanced images based on the new input. The multi-layer convolutional neural network structure inside the generator gradually processes the input feature vector according to the parameters and weights obtained by training. For example, for the vibration frequency domain fingerprint spectrum features, a smaller convolution kernel (such as 3×3) will finely capture its frequency-related features and convert these features into relevant details in the image; for point cloud spatiotemporal coding, a slightly larger convolution kernel (such as 5×5) will integrate its spatial structure information, making the spatial layout of the road and surrounding environment in the generated image more reasonable; for the weighted visual feature map, a suitable convolution kernel (such as 4×4) will highlight the texture and detail characteristics of the image, making the road surface texture, road signs, etc. of the generated image clearer. These enhanced images are richer than the original images in terms of details and features. For example, when detecting road surface defects, the enhanced image may more clearly show the direction and width of cracks, as well as the depth and edge details of potholes. In terms of road facility assessment, the text and color of road signs in the image, the shape and material of guardrails, and other features will also be more obvious. This is because during the training process of the adversarial generative network, the generator is continuously optimized and strives to generate images that are closer to reality. At the same time, the feedback from the discriminator prompts the generator to pay attention to various details and features of the image, which significantly improves the quality of the generated image.

[0199] From the perspective of beneficial effects, in the analysis of highway inspection and maintenance conditions, enhanced images help improve the accuracy of analysis. Taking pavement disease detection as an example, clearer image details enable the detection algorithm to more accurately identify the type, scope and severity of the disease. Traditional image analysis may be difficult to accurately judge some minor diseases or diseases under complex backgrounds due to image quality problems, while enhanced images can provide richer information and reduce misjudgments and missed judgments. For road facility assessment, rich image features help assessors to have a more comprehensive understanding of the status of the facilities, such as judging whether road signs are clear and identifiable, whether they need to be replaced, and whether guardrails are damaged or deformed, so as to formulate maintenance plans in a timely manner to ensure the safety and normal use of roads. In terms of data supplementation, enhanced images can expand the image data set of highway inspection and maintenance, provide more diverse data samples for subsequent model training, algorithm optimization, etc., further improve the overall level of highway inspection and maintenance image analysis technology, and promote the efficient implementation of highway maintenance work.

[0200] Example 2

[0201] This embodiment provides an adversarial generation data enhancement system for highway inspection and maintenance images based on embodiment 1. Figure 8 As shown, including:

[0202] Spectrum construction module: used to collect six-axis vibration signals of maintenance equipment in real time and construct vibration frequency domain fingerprint spectrum; perform frequency domain decomposition of the collected six-axis vibration signals based on the sliding time window, extract characteristic frequency bands related to the movement speed of the maintenance equipment, and generate a vibration-speed mapping relationship matrix; dynamically correct the vibration-speed mapping relationship matrix and output the calibrated vibration frequency domain fingerprint spectrum;

[0203] Feature fusion module: used to collect point cloud data in the highway inspection area, perform entropy value weighted encoding on the point cloud data, and generate point cloud spatiotemporal encoding; obtain visual images of the highway inspection area, perform cross-modal correlation analysis on the six-axis vibration signal and the visual image, and obtain weighted visual feature maps; construct a multimodal fusion feature vector based on the calibrated vibration frequency domain fingerprint map, point cloud spatiotemporal encoding, and weighted visual feature maps;

[0204] Adversarial generation module: Based on the multimodal fusion feature vector, a generative adversarial network consisting of a generator and a discriminator is constructed and trained; a new multimodal fusion feature vector is obtained, and enhanced highway inspection and maintenance images are generated based on the new multimodal fusion feature vector and the trained generative adversarial network.

[0205] The method and system of the present application may be implemented in many ways. For example, the method and system of the present application may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present application are not limited to the order specifically described above, unless otherwise specifically stated.

[0206] In addition, the parts of the above-mentioned technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive redundancy.

[0207] The specific implementation modes as described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation mode of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. The adversarial generation data enhancement method for highway inspection and maintenance images is characterized by: The method comprises: Collect the six-axis vibration signals of the maintenance equipment in real time and construct a vibration frequency domain fingerprint. Perform frequency domain decomposition on the collected six-axis vibration signals based on a sliding time window, extract the characteristic frequency bands related to the movement speed of the maintenance equipment, and generate a vibration-speed mapping relationship matrix. Dynamically correct the vibration-speed mapping relationship matrix and output a calibrated vibration frequency domain fingerprint. Collect point cloud data from the highway inspection area, perform entropy value weighted encoding on the point cloud data, and generate point cloud spatiotemporal encoding; obtain visual images of the highway inspection area, perform cross-modal correlation analysis on the six-axis vibration signal and the visual image, and obtain a weighted visual feature map; construct a multimodal fusion feature vector based on the calibrated vibration frequency domain fingerprint map, point cloud spatiotemporal encoding, and weighted visual feature map; According to the multimodal fusion feature vector, a generative adversarial network including a generator and a discriminator is constructed and trained; a new multimodal fusion feature vector is obtained, and an enhanced highway inspection and maintenance image is generated according to the new multimodal fusion feature vector and the trained generative adversarial network.

2. The adversarial generation data enhancement method for highway inspection and maintenance images according to claim 1 is characterized in that: The real-time acquisition of the six-axis vibration signal of the maintenance equipment includes: installing an inertial measurement unit on the highway inspection and maintenance equipment, and collecting the six-axis vibration signal of the maintenance equipment in real time during driving according to the installed inertial measurement unit; the six-axis vibration signal includes acceleration signals a in three directions of X, Y, and Z axes x (t), a y (t), a z (t) and angular velocity signals ω in the three directions of X, Y, and Z axes x (t),ω y (t),ω z (t).

3. The adversarial generation data enhancement method for highway inspection and maintenance images according to claim 2 is characterized in that: The construction of the vibration frequency domain fingerprint spectrum comprises: The collected six-axis vibration signals are converted into frequency domain to obtain the main frequency band energy distribution and harmonic distortion rate; the relationship between the acceleration signals in the three directions of X, Y, and Z axes and the angular velocity signals in the three directions of X, Y, and Z axes are analyzed to obtain the inter-axis coupling coefficient; the main frequency band energy distribution, harmonic distortion rate, and inter-axis coupling coefficient are integrated to construct a vibration frequency domain fingerprint.

4. The adversarial generation data enhancement method for highway inspection and maintenance images according to claim 3 is characterized in that: The inter-axis coupling coefficient is obtained by: The acceleration signal a in the three directions of X, Y and Z axis x (t), a y (t), a z (t) and angular velocity signals ω in the three directions of X, Y, and Z axes x (t),ω y (t),ω z (t) Perform timestamp alignment; According to the aligned a x (t), a y (t), a z (t),ω x (t),ω y (t),ω z (t), construct the multivariate signal space matrix; According to the multivariate signal space matrix, the correlation coefficient matrix R and the nonlinear correlation matrix N are obtained; The correlation coefficient matrix and the nonlinear correlation matrix The fusion is performed to obtain a comprehensive correlation matrix C, from which the inter-axis coupling coefficient is extracted.

5. The adversarial generation data enhancement method for highway inspection and maintenance images according to claim 4 is characterized in that: The constructing of a multivariate signal space matrix comprises: After alignment, a x (t), a y (t), a z (t),ω x (t),ω y (t),ω z (t) is combined into a six-dimensional signal vector ; in a period of time Inside, collect The six-dimensional signal vector at each moment constructs a multivariate signal space matrix ,in, is the six-dimensional signal vector at the nth moment.

6. The adversarial generation data enhancement method for highway inspection and maintenance images according to claim 5 is characterized in that: The obtaining of the correlation coefficient matrix R comprises: For the multivariate signal space matrix Any two components in and ,calculate and The correlation coefficients between all different axes give a The correlation coefficient matrix , the correlation coefficient matrix Elements in Indicates ''Axis and ''Correlation between axis signals; where, is the six-dimensional signal vector at the i-th moment, is the six-dimensional signal vector at the jth moment, 1≤i≤n, 1≤j≤n, .

7. The adversarial generation data enhancement method for highway inspection and maintenance images according to claim 6 is characterized in that: Obtaining the nonlinear correlation matrix N includes: calculating and The mutual information value of , according to the mutual information value Construct the nonlinear correlation matrix N.

8. The adversarial generation data enhancement method for highway inspection and maintenance images according to claim 1 is characterized in that: Generating a vibration-velocity mapping relationship matrix comprises: The six-axis vibration signal of each characteristic frequency band is represented in time-frequency form, and the amplitude statistics of each characteristic frequency band are extracted; the movement speed of the maintenance equipment is obtained, and the amplitude statistics of each characteristic frequency band are associated with the movement speed of the maintenance equipment to establish the vibration-speed mapping relationship matrix M vs ; Among them, the matrix element M vs (f i' ,v j' ) indicates the i'th characteristic frequency band f i' Next, the j'th velocity value v j' The corresponding amplitude statistics.

9. The adversarial generation data enhancement method for highway inspection and maintenance images according to claim 8 is characterized in that: The output calibrated vibration frequency domain fingerprint spectrum includes: The vibration-velocity mapping relationship matrix is ​​dynamically corrected by the Kalman filter to obtain the corrected M vs Matrix; according to the modified M vs The vibration frequency domain fingerprint spectrum is calibrated by the matrix to obtain the calibrated vibration frequency domain fingerprint spectrum.

10. The adversarial generation data enhancement method for highway inspection and maintenance images according to claim 1, characterized in that: Generating point cloud spatiotemporal coding comprises: Collect point cloud data of the highway inspection area, and record the point cloud collection timestamp and spatial coordinates; calculate the spatiotemporal entropy weight coefficient of each point based on the point cloud collection timestamp and spatial coordinates; extract the surface features of the point cloud data, wherein the surface features include reflection intensity gradient, normal vector offset and local curvature mutation threshold; perform entropy value weighted encoding on the point cloud data according to the spatiotemporal entropy weight coefficient of each point and the surface features of the point cloud data, and generate point cloud spatiotemporal encoding.

11. The adversarial generation data enhancement method for highway inspection and maintenance images according to claim 1, characterized in that: The weighted visual feature map includes: Calculate the gradient direction of each pixel point of the visual image to form the gradient direction of the visual image; calculate the mutual information between each characteristic frequency band of the six-axis vibration signal and the gradient direction of the visual image to obtain the pixel-frequency band mutual information matrix I pf ; Construct the maintenance equipment rigidity coefficient m1, road surface roughness n1, pixel-band mutual information matrix I pf The nonlinear mapping model w=F(m1,n1,I pf ), where F is a nonlinear function based on a deep neural network; m1, n1, I are determined by an analytical hierarchical process. pf The influence weight on w is substituted into the model w=F(m1,n1,I pf ), the pixel vibration sensitivity weight of each pixel is calculated; the visual image of the highway inspection area is feature extracted to obtain a visual feature map; the pixel vibration sensitivity weight of each pixel is element-by-element multiplied by the visual feature map to obtain a weighted visual feature map.

12. A system for enhancing adversarial generation data of highway inspection and maintenance images, which is used to implement the method for enhancing adversarial generation data of highway inspection and maintenance images as claimed in any one of claims 1 to 11, characterized in that: The system comprises: Spectrum construction module: used to collect six-axis vibration signals of maintenance equipment in real time and construct vibration frequency domain fingerprint spectrum; perform frequency domain decomposition of the collected six-axis vibration signals based on the sliding time window, extract characteristic frequency bands related to the movement speed of the maintenance equipment, and generate a vibration-speed mapping relationship matrix; dynamically correct the vibration-speed mapping relationship matrix and output the calibrated vibration frequency domain fingerprint spectrum; Feature fusion module: used to collect point cloud data in the highway inspection area, perform entropy value weighted encoding on the point cloud data, and generate point cloud spatiotemporal encoding; obtain visual images of the highway inspection area, perform cross-modal correlation analysis on the six-axis vibration signal and the visual image, and obtain weighted visual feature maps; construct a multimodal fusion feature vector based on the calibrated vibration frequency domain fingerprint map, point cloud spatiotemporal encoding, and weighted visual feature maps; Adversarial generation module: Based on the multimodal fusion feature vector, a generative adversarial network consisting of a generator and a discriminator is constructed and trained; a new multimodal fusion feature vector is obtained, and enhanced highway inspection and maintenance images are generated based on the new multimodal fusion feature vector and the trained generative adversarial network.

Images