Sound barrier corrosion state identification method fusing image and multi-sensor data

Abstract

The invention provides a sound barrier corrosion state identification method based on image and multi-sensor data fusion, and relates to the field of data processing. The method comprises the following steps: acquiring image data and multi-sensor data of a target area of a target sound barrier; extracting a first corrosion area feature from the image data; according to the multi-sensor data, constructing a second corrosion area feature by adopting GRU coding; based on the first corrosion region feature and the second corrosion region feature, calculating by adopting a space-time diagram neural network to obtain a corrosion risk score of the target region; and determining the corrosion state of the target sound barrier according to the corrosion risk score. By implementing the technical scheme provided by the invention, the accuracy of identifying the corrosion state of the sound barrier is improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a method for identifying the corrosion state of a sound barrier by integrating image and multi-sensor data. Background Art

[0002] With the acceleration of urbanization and the expansion of transportation infrastructure, as an important measure to reduce traffic noise pollution, sound barriers have been widely used in scenarios such as highways, railways, and urban viaducts. However, being exposed to the complex outdoor environment for a long time, sound barriers are inevitably affected by the external environment, resulting in the gradual corrosion, aging, and even damage of the structural materials. Therefore, it is necessary to detect the corrosion state of sound barriers.

[0003] Currently, the detection of the corrosion state of sound barriers still mainly relies on manual inspections, which require dispatching professional personnel to visually inspect the sound barriers regularly or use simple measuring tools for detection. However, due to the large number and wide distribution of sound barriers and the limited inspection frequency, it is difficult to detect corrosion problems in a timely manner. In addition, manual inspections are easily affected by personnel experience and environmental conditions, and the detection results are somewhat subjective. Therefore, the method of manually detecting the corrosion state has the problem of low accuracy.

[0004] Therefore, there is an urgent need for a method for identifying the corrosion state of a sound barrier by integrating image and multi-sensor data. Summary of the Invention

[0005] This application provides a method for identifying the corrosion state of a sound barrier by integrating image and multi-sensor data, which is convenient for improving the accuracy of identifying the corrosion state of a sound barrier.

[0006] In the first aspect of this application, a method for identifying the corrosion state of a sound barrier by integrating image and multi-sensor data is provided. The method includes: acquiring image data and multi-sensor data of a target area of a target sound barrier; extracting first corrosion area features from the image data; constructing second corrosion area features by using GRU encoding according to the multi-sensor data; calculating the corrosion risk score of the target area by using a spatio-temporal graph neural network based on the first corrosion area features and the second corrosion area features; and determining the corrosion state of the target sound barrier according to the corrosion risk score.

[0007] By adopting the above technical solutions, by simultaneously acquiring the image data and sensor data of the target area, the visual information and physical signals are fully utilized. The image data can capture the surface texture and color changes of corrosion, while the sensor data reflects the environmental changes and structural responses. The two complement each other, reducing the risk of environmental interference to a single data source. The image data is processed by a convolutional neural network to extract the first corrosion area features, effectively capturing local corrosion details. At the same time, the GRU encoding is used to perform temporal modeling on the sensor data to construct the second corrosion area features, which can fully reflect the dynamic evolution of the corrosion process, especially suitable for dealing with long-term dependencies in time series data. Based on the two extracted corrosion features, the ST-GNN can simultaneously capture spatio-temporal information, that is, consider the local features and global relationships of the corrosion state in space and the dynamic change trends in time. Such a model can more accurately fuse multi-modal data and generate a quantified corrosion risk score. By mapping the fused features to the corrosion risk score, a quantitative judgment of the corrosion state can be achieved, providing an objective basis for subsequent maintenance, early warning and decision-making, and reducing the error of manual subjective judgment. This method makes full use of automated data collection and deep learning models to realize real-time monitoring and high-precision evaluation of the corrosion state of the target area, effectively improving the detection efficiency, reducing the inspection cost, and helping to timely discover potential problems, thereby extending the service life of the facility. Therefore, it is convenient to improve the accuracy of the corrosion state recognition of the sound barrier.

[0008] Optionally, the acquiring the image data and multi-sensor data for the target area of the target sound barrier specifically includes: acquiring the original image of the target area sent by the high-definition camera; performing denoising on the original image by using median filtering, and using the pre-trained U-Net model to extract the areas with spots in the target area, outputting a binary segmentation map to obtain the image data; acquiring the temperature and humidity data, vibration data, electrochemical corrosion data, and ultrasonic data sent by the temperature and humidity sensor, vibration sensor, electrochemical corrosion sensor, and ultrasonic sensor; performing filtering and normalization processing on the temperature and humidity data, the vibration data, the electrochemical corrosion data, and the ultrasonic data to obtain the multi-sensor data.

[0009] By adopting the above technical solution, the original image of the target area is obtained by using a high-definition camera, which can intuitively capture the appearance features of corrosion, such as spots, cracks, color changes, etc. The physical environment and structural state data are collected by temperature and humidity sensors, vibration sensors, electrochemical corrosion sensors and ultrasonic sensors to supplement the deficiencies of the image information. This multi-source information fusion ensures that the corrosion detection not only depends on the apparent visual features, but also takes into account the deep physical, chemical and mechanical features, improving the comprehensiveness and reliability of the detection. Median filtering is used to remove the salt-and-pepper noise in the image, improve the image quality, and make the subsequent feature extraction more accurate. A deep learning segmentation network is adopted, which is specifically used to extract the corrosion area in the image, and can accurately identify damage features such as spots and corrosion pits. Through binaryzation processing, the corrosion area is clearly separated from the background, reducing the complexity of subsequent calculations and facilitating further feature extraction and analysis. This processing flow can effectively reduce the interference of external factors (such as light, shadow, stains, etc.) and ensure the accuracy of corrosion area recognition. For different types of sensor data, appropriate filtering methods (such as Kalman filtering, median filtering, low-pass filtering, etc.) are adopted to remove environmental noise, improve the signal quality, and make the data more stable. Since the measurement units and value ranges of different sensors are different, normalizing the data can eliminate the dimension difference, avoid individual data dominating the model calculation, and improve the effect of multi-sensor data fusion. These processing steps ensure the high-quality input of sensor data and make the subsequent data analysis and modeling more accurate and efficient.

[0010] Optionally, extracting the first corrosion area feature from the image data specifically includes: segmenting the image data by using a segmentation algorithm to obtain an abnormal area; extracting features of the abnormal area by using a pre-trained convolutional neural network to obtain the first corrosion area feature, where the first corrosion area feature includes texture, color change, and cracks.

[0011] By adopting the above technical solution, the image data is processed through a segmentation algorithm, which can distinguish the normal area from the abnormal area, so as to focus more on the parts where corrosion may occur rather than the whole image, improving the calculation efficiency and the pertinence of analysis. This method reduces unnecessary background interference, such as misjudgment caused by light changes, stains or other non-corrosive factors, making the subsequent feature extraction more accurate. Convolutional neural network (CNN) is good at capturing local patterns in images and can automatically learn and extract key features related to corrosion without relying on manually designed rules, enhancing the adaptability and generalization ability of the model. Pre-training the CNN means that the model has been trained on a large-scale dataset, has strong image recognition ability, and can be adapted to the corrosion detection task through transfer learning, reducing the training cost and improving the detection accuracy at the same time. Texture information can reflect the roughness changes on the material surface. For example, local metal peeling, corrosion pits, rust diffusion, etc. can all be distinguished through texture features. Using CNN can automatically extract local high and low frequency information and identify the texture differences between the corrosion area and the normal area. Corrosion is usually accompanied by color changes. For example, the metal surface changes from silver-gray to reddish-brown (rust), forms black or green due to oxidation, etc. Color information is an important basis for corrosion detection. CNN can extract the change patterns in the RGB channel or the HSV color space, improving the recognition ability for different types of corrosion (such as rust, oxidation, stains). Corrosion may lead to fractures, microcracks or even through-hole damages in the material structure, and cracks often have clear linear or branched morphologies. Using CNN to extract crack features can identify the position, length and width of the cracks through edge detection, morphological analysis, etc., helping to evaluate the severity of corrosion.

[0012] Optionally, the construction of the second corrosion area feature by using GRU encoding according to the multi-sensor data specifically includes: inputting the multi-sensor data into the GRU model according to a preset time window to obtain dynamic change features; performing dimension matching on the dynamic change features and the first corrosion area feature to obtain the second corrosion area feature.

[0013] By adopting the above technical solution, through time series modeling of multi-sensor data by GRU (Gated Recurrent Unit), the dynamic change features of the corrosion area are extracted, thus making up for the limitation that image features can only reflect static states. Inputting data with a preset time window enables the model to capture the trend of the corrosion process over time, such as the influence of temperature and humidity on the corrosion rate, and the structural deterioration caused by vibration, etc. In addition, through dimension matching, this method integrates image features (static information) and sensor features (dynamic information) to construct a more complete second corrosion area feature, improving the timeliness and accuracy of corrosion state assessment, and thus providing a more stable and reliable input for subsequent corrosion risk calculation.

[0014] Optionally, based on the first corrosion area feature and the second corrosion area feature, calculating the corrosion risk score of the target area by using a spatio-temporal graph neural network, specifically including: taking the first corrosion area feature as the first type of node; taking the second corrosion area feature as the second type of node; obtaining a first set of target nodes at different time steps within the target area from the first type of node and the second type of node; establishing a temporal edge between the first set of target nodes; obtaining a second set of target nodes corresponding to the same time step within the target area from the first type of node and the second type of node; performing weighted connection between the second set of target nodes to obtain a modal edge; constructing a spatio-temporal graph based on the first type of node, the second type of node, the temporal edge, and the modal edge.

[0015] By adopting the above technical solution, through a spatio-temporal graph neural network, the image feature (the first corrosion area feature) and the multi-sensor time series feature (the second corrosion area feature) are fused to construct a spatio-temporal correlation corrosion risk assessment model. The image feature is used as the first type of node, and the sensor feature is used as the second type of node. The corrosion features at different time steps are connected by a temporal edge to capture the long-term change trend of the corrosion state. At the same time, the modal edge is used to connect different modal features at the same time step to achieve information complementarity of multi-source data. Based on this structure, a spatio-temporal graph is constructed, enabling the model to not only understand the current state of corrosion but also combine historical information for trend prediction, thereby providing a more refined, more dynamic, and more accurate corrosion risk score and providing a more reliable basis for subsequent decision-making.

[0016] Optionally, based on the first corrosion area feature and the second corrosion area feature, calculating the corrosion risk score of the target area by using a spatio-temporal graph neural network specifically further includes: using the graph convolutional layer of the spatio-temporal graph neural network to perform feature aggregation on the spatio-temporal graph, fusing the information between different nodes to obtain a first data to be fused; using the temporal convolutional layer of the spatio-temporal graph neural network to capture the change trend of the spatio-temporal graph to obtain a second data to be fused; performing attention-weighted aggregation on the first data to be fused and the second data to be fused to generate the corrosion risk score of the target area.

[0017] By adopting the above technical solution, the graph convolution layer and the temporal convolution layer of the spatio-temporal graph neural network are utilized to deeply fuse the corrosion features in both the spatial and temporal dimensions, improving the accuracy of risk assessment. Through feature aggregation, the graph convolution layer enables information interaction of corrosion features of different modalities (images and sensor data) in the graph structure, enhancing the expression ability of local corrosion features. The temporal convolution layer further captures the evolution trend of the corrosion state and extracts the long-term dynamic change law. Finally, through the attention mechanism, the spatial features and the temporal features are weighted and fused to ensure that the corrosion risk score is more accurate and reasonable, which helps to improve the stability of corrosion prediction and supports the intelligent early warning and maintenance optimization of the sound barrier structure state.

[0018] Optionally, the method further includes: comparing the corrosion risk score with a preset threshold to determine the corrosion state, where the corrosion state includes normal, mild corrosion, moderate corrosion, and severe corrosion; generating an operation and maintenance warning for the target sound barrier according to the corrosion state.

[0019] By adopting the above technical solution, through the comparison between the corrosion risk score and the preset threshold, the intelligent grading of the corrosion state (normal, mild, moderate, severe corrosion) is realized, thus providing a clear and quantifiable corrosion assessment standard. Compared with the traditional method relying on manual experience, this method can automatically and objectively determine the corrosion degree and avoid human errors. In addition, this solution further generates an operation and maintenance warning in combination with the corrosion state, which can provide accurate maintenance suggestions for the management department, such as regular inspection, local repair, or emergency replacement, optimize the allocation of operation and maintenance resources, improve the safety and service life of the sound barrier, and ultimately reduce the long-term maintenance cost.

[0020] In the second aspect of the present application, a sound barrier corrosion state recognition device integrating images and multi-sensor data is provided. The device includes an acquisition module and a processing module. Among them, the acquisition module is used to acquire image data and multi-sensor data of a target area of a target sound barrier; the processing module is used to extract the first corrosion area feature from the image data; the processing module is further used to construct a second corrosion area feature by using GRU encoding according to the multi-sensor data; the processing module is further used to calculate the corrosion risk score of the target area by using a spatio-temporal graph neural network based on the first corrosion area feature and the second corrosion area feature; the processing module is further used to determine the corrosion state of the target sound barrier according to the corrosion risk score.

[0021] In a third aspect of the present application, an electronic device is provided. The electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions. Both the user interface and the network interface are used for communicating with other devices. The processor is used to execute the instructions stored in the memory so that the electronic device executes the method described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions that, when executed, perform the method described above.

[0023] In summary, one or more technical solutions provided in the present application have at least the following technical effects or advantages: By simultaneously acquiring the image data and sensor data of the target area, visual information and physical signals are fully utilized. The image data can capture the surface texture and color changes of corrosion, while the sensor data reflects environmental changes and structural responses. The two complement each other, reducing the risk of a single data source being interfered by the environment. The image data is used to extract the first corrosion area features through a convolutional neural network, effectively capturing local corrosion details. At the same time, the GRU encoding is used to perform temporal modeling on the sensor data to construct the second corrosion area features, which can fully reflect the dynamic evolution of the corrosion process, especially suitable for dealing with long-term dependencies in time series data. Based on the two extracted corrosion features, the ST-GNN can simultaneously capture spatio-temporal information, that is, consider the local features and global relationships of the corrosion state in space and the dynamic change trends in time. Such a model can more accurately fuse multi-modal data and generate a quantified corrosion risk score. By mapping the fused features to the corrosion risk score, a quantitative judgment of the corrosion state can be achieved, providing an objective basis for subsequent maintenance, early warning, and decision-making, and reducing manual subjective judgment errors. This method fully utilizes automated data collection and deep learning models to achieve real-time monitoring and high-precision evaluation of the corrosion state of the target area, effectively improving the detection efficiency, reducing the inspection cost, and helping to discover potential problems in a timely manner, thereby extending the service life of the facility. Therefore, it is convenient to improve the accuracy of sound barrier corrosion state recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic flowchart of a method for identifying the corrosion state of a sound barrier by fusing images and multi-sensor data provided by an embodiment of the present application; Figure 2 It is another schematic flowchart of a method for identifying the corrosion state of a sound barrier by fusing images and multi-sensor data provided by an embodiment of the present application; Figure 3 It is a schematic block diagram of a device for identifying the corrosion state of a sound barrier by fusing images and multi-sensor data provided by an embodiment of the present application; Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present application.

[0025] Explanation of reference numerals: 31, acquisition module; 32, processing module; 41, processor; 42, communication bus; 43, user interface; 44, network interface; 45, memory. Detailed implementation manners

[0026] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments.

[0027] In the description of the embodiments of the present application, words such as "for example" or "for illustration" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "for example" or "for illustration" in the embodiments of the present application should not be construed as being more preferred or more advantageous than other embodiments or design solutions. Exactly speaking, the use of words such as "for example" or "for illustration" aims to present relevant concepts in a specific manner.

[0028] In the description of the embodiments of the present application, the meaning of the term "a plurality" refers to two or more. For example, a plurality of systems refers to two or more systems, and a plurality of screen terminals refers to two or more screen terminals. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the technical features indicated. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. The terms "include", "comprise", "have" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0029] With the acceleration of the urbanization process and the continuous expansion of transportation infrastructure, sound barriers have become important facilities for reducing traffic noise pollution and are widely used in scenarios such as highways, railways and urban viaducts. However, being exposed to the complex outdoor environment for a long time, sound barriers are inevitably affected by factors such as climate change, air pollution, and mechanical vibration, resulting in the gradual corrosion, aging and even damage of the materials. Therefore, timely and accurately detecting the corrosion state of sound barriers is crucial for ensuring their service life and safety.

[0030] Currently, the corrosion detection of sound barriers mainly relies on manual inspections, that is, professional personnel conduct visual inspections or use simple measurement tools for evaluation on a regular basis. However, due to the large number and wide distribution of sound barriers, manual inspections not only involve huge workloads and high costs, but also, limited by the inspection frequency, it is difficult to detect corrosion problems in a timely manner. In addition, manual inspections are affected by factors such as the experience of inspection personnel, lighting conditions, and weather conditions, and the inspection results are highly subjective and less accurate, making it difficult to meet the needs of large-scale and refined monitoring.

[0031] To solve the above technical problems, this application provides a method for identifying the corrosion state of sound barriers by fusing image and multi-sensor data. Referring to Figure 1 , Figure 1 is a schematic flowchart of a method for identifying the corrosion state of sound barriers by fusing image and multi-sensor data provided by an embodiment of this application. This method is applied to a server and includes steps S110 to S150. The above steps are as follows: S110. Obtain image data and multi-sensor data for the target area of the target sound barrier.

[0032] Specifically, the server (i.e., the central data processing unit) collects image data and various sensor data from devices deployed in the target area of the sound barrier. The target sound barrier refers to a specific sound barrier for which the corrosion state needs to be monitored, and the target area is the part of the sound barrier that is under key monitoring, such as areas prone to corrosion (such as parts exposed to wind, rain, and salt spray erosion). The image data is captured by a high-definition camera installed on the sound barrier or a sound barrier inspection robot. The image data can intuitively show the appearance of the sound barrier surface, such as corrosion phenomena like spots, cracks, and discoloration. The camera may be set at a fixed position to capture images regularly or transmit real-time images to the server via a wireless network. The multi-sensor data refers to data collected by multiple sensors installed in the target area, such as temperature and humidity sensors, vibration sensors, electrochemical sensors, ultrasonic sensors, etc. These data can reflect the dynamic changes in the environment where the sound barrier is located and its internal state, such as environmental temperature, humidity, structural vibration conditions, and metal electrochemical reactions, which play an important auxiliary role in judging the corrosion state.

[0033] As the central processing system, the server is responsible for receiving, storing, and processing these multi-modal data. Through the fusion analysis of the image data and sensor data, the server can evaluate the corrosion state of the sound barrier and generate corresponding early warning information or maintenance suggestions.

[0034] For example, assume that there is a corrosion risk for a section of the sound barrier on a highway. High-definition cameras and various sensors such as temperature and humidity sensors, vibration sensors, and electrochemical sensors are installed in this area. Every morning, the camera automatically takes an image of the surface of the sound barrier, and at the same time, each sensor collects data on temperature and humidity, vibration intensity, and electrochemical reaction in this area. All this data is transmitted to the server in real time through the network. After receiving the data, the server first processes the image to detect whether there are spots or cracks, and then combines the sensor data to analyze the environmental and structural change trends, so as to comprehensively evaluate the corrosion risk in this area. If an anomaly is detected, the server will immediately generate a warning message to notify the maintenance personnel to conduct on-site inspections and repairs. This method can achieve automatic and real-time monitoring of the corrosion state of the sound barrier, effectively improving the maintenance efficiency and reducing potential risks.

[0035] In a possible implementation, obtaining image data and multi-sensor data for a target area of a target sound barrier specifically includes: obtaining the original image sent by the high-definition camera for the target area; using median filtering to denoise the original image, and using a pre-trained U-Net model to extract the areas with spots in the target area and output a binary segmentation map to obtain the image data; obtaining temperature and humidity data, vibration data, electrochemical corrosion data, and ultrasonic data sent by the temperature and humidity sensor, vibration sensor, electrochemical corrosion sensor, and ultrasonic sensor; filtering and normalizing the temperature and humidity data, vibration data, electrochemical corrosion data, and ultrasonic data to obtain multi-sensor data.

[0036] Specifically, the original image is obtained by using a high-definition camera installed in the target area to collect the original image, ensuring that detailed information is captured. Median filtering is used to denoise the original image to reduce noise interference and make the image clearer. The pre-trained U-Net model is used to segment the denoised image to identify the parts with spots (possibly corrosion marks) in the target area and generate a binary segmentation map to further extract key image feature data. Temperature and humidity data, vibration data, electrochemical corrosion data, and ultrasonic data are obtained from the temperature and humidity sensor, vibration sensor, electrochemical corrosion sensor, and ultrasonic sensor respectively. These data can reflect the physical state of the environment and the structure. Filtering is performed on the data of various sensors to remove noise, and the data is normalized so that the data of different sensors are unified in the numerical scale, laying a foundation for subsequent data fusion and feature extraction.

[0037] For example, assume that on a section of a sound barrier on a highway, it is necessary to monitor its corrosion condition. A high-definition camera, as well as temperature and humidity, vibration, electrochemical corrosion, and ultrasonic sensors, are installed at key areas on-site. The camera takes original images of the target area at regular intervals every day. The images may contain noise due to factors such as weather and lighting. The system first applies median filtering to the images to remove the noise generated by environmental interference. Then, the pre-trained U-Net model is used to segment the denoised images, identify the areas with rust spots or corrosion marks on the surface, and output these areas in the form of a binary image, thereby obtaining clear image data that centrally displays the key parts of the corrosion. At the same time, the temperature and humidity sensors record the temperature and humidity data of the area, the vibration sensor collects the vibration data generated by vehicle impacts or wind forces, the electrochemical sensor detects the potential changes during metal surface corrosion, and the ultrasonic sensor captures the possible internal crack information. After filtering these data, the abnormal data caused by equipment noise and environmental interference are removed, and then each data is normalized to make the subsequent multi-modal fusion processing more accurate. In this way, the system can not only obtain high-quality and clear image data to accurately identify the possible corrosion spots on the surface of the sound barrier, but also collect a variety of physical signal data reflecting the corrosion process, providing a comprehensive and reliable basis for further corrosion state assessment.

[0038] S120. Extract the first corrosion area feature from the image data.

[0039] Specifically, first, using the pre-processed image data (after denoising and segmentation), identify the possible corrosion spots or abnormal areas in the image. These areas may appear as parts that are different from the surrounding areas in terms of color, texture, or morphology. Then, the server uses a pre-trained convolutional neural network (such as ResNet or VGG) to extract features from these abnormal areas. The network will automatically extract the features related to corrosion, such as the size, shape, texture, color change of the spots, and the possible crack information. After convolutional operations, the extracted information will be integrated into a feature vector with a fixed dimension, and this vector represents the detailed features of the corrosion area in the image and is an important input data for subsequent corrosion state assessment and risk scoring.

[0040] For example, assume that a server receives a set of pre - processed image data. After each image is segmented by U - Net, the areas suspected of corrosion on the sound barrier have been marked (such as corrosion spots shown as white areas). The server then uses the pre - trained ResNet50 model to extract features from these marked areas. The model automatically captures the following feature information: The texture information reflects the roughness and non - uniformity of the surface of the corrosion area. Color changes, such as the change from the original metallic luster to dark red or brown, indicate rusting. If there are cracks in the area, the model will also extract this linear structure information.

[0041] Finally, the server converts each corrosion area into a 128 - dimensional feature vector, which is the first corrosion area feature and can be used for fusion with other modal data or directly for corrosion status assessment. This method ensures that the system can automatically and objectively extract key corrosion features from a large amount of image data, reduces the subjectivity of manual interpretation, and lays a solid foundation for subsequent multi - modal data fusion and risk assessment.

[0042] In a possible implementation, extracting the first corrosion area feature from the image data specifically includes: using a segmentation algorithm to segment the image data to obtain abnormal areas; using a pre - trained convolutional neural network to extract features from the abnormal areas to obtain the first corrosion area feature, and the first corrosion area feature includes texture, color change, and cracks.

[0043] Specifically, first, use a segmentation algorithm to process the input image data, segment the image into different regions, and thus identify the abnormal areas - that is, the parts that are significantly different from the surrounding normal areas. These areas usually may show signs of corrosion, such as spots or local color abnormalities. Then, use a pre - trained convolutional neural network (CNN, such as ResNet or VGG) to extract features from the identified abnormal areas. Through the processing of convolutional layers, activation layers, and pooling layers, the model can automatically extract the key information of these abnormal areas. These features include: The texture information reflects the roughness and non - uniformity of the surface of the corrosion area. Color changes, such as the change of the metal surface from silver - gray to reddish - brown or brown, indicate rusting. Crack information, if there are cracks in the area, the CNN can capture these linear structures or fracture edges. These information are integrated into a feature vector with a fixed dimension, which constitutes the "first corrosion area feature" and provides basic data for subsequent corrosion risk assessment or multi - modal data fusion.

[0044] For example, assume that in a monitoring system for a highway sound barrier, the system obtains an image captured by a high-definition camera. Due to environmental factors, some suspected rust areas appear on the surface of the sound barrier in this image, with a darker color and uneven local texture. The specific processing flow is as follows: The system first uses a trained U-Net model to perform segmentation on the image. After segmentation, the model separates each region in the image and marks those regions that are different from the normal metal surface (for example, the binarized result shown as bright white). These regions are the so-called abnormal regions and may contain corrosion spots. Next, the system inputs the image data of these abnormal regions into a pre-trained ResNet model. This model automatically extracts the features within the region and captures the texture features: the roughness change on the surface of the region, indicating peeling or corrosion of the metal surface. Color features: the color information changing from the normal silver-gray to dark red or brown, suggesting the presence of oxidation and rust. Crack features: If there are cracks in the region, the model will capture these linear structures through edge detection. Finally, ResNet outputs a 128-dimensional feature vector, which represents the "first corrosion region feature" of the abnormal region.

[0045] Through this method, the system can automatically convert the corrosion information in the image into numerical features, providing objective and quantitative data support for the subsequent automatic detection of the corrosion state and risk assessment.

[0046] S130. According to the multi-sensor data, use GRU encoding to construct the second corrosion region feature.

[0047] Specifically, the server uses the multi-sensor data to encode these time series data through a GRU (Gated Recurrent Unit) model, thereby constructing the "second corrosion region feature". This feature mainly reflects the dynamic information of the sensor data changing over time, providing additional temporal feature support for corrosion state detection.

[0048] Furthermore, the multi-sensor data may include temperature and humidity, vibration, electrochemical corrosion, and ultrasonic data. These data are all continuously recorded time series, which can reflect the physical state changes of the sound barrier at different time points. As a recurrent neural network suitable for processing time series data, GRU can capture the time dependence and dynamic change trends existing in these data, thereby converting the input time series into a feature vector with a fixed dimension, and this vector is the "second corrosion region feature".

[0049] For example, assume that in a certain target sound barrier area, sensors continuously record temperature, humidity, vibration, and electrochemical corrosion signals every day. The server takes a 5-minute time window and inputs the sensor data sequence during this period into a pre-trained GRU model. The GRU model analyzes the change patterns of these data, such as the fluctuations of temperature and humidity, the changes in vibration intensity, and the trends of electrochemical reactions, and finally outputs a 128-dimensional feature vector. This vector captures the dynamic information of the sensor data during this period and is used as the "second corrosion area feature" for subsequent fusion with image features and corrosion risk assessment.

[0050] In a possible implementation manner, according to multi-sensor data, a second corrosion area feature is constructed by using GRU encoding, specifically including: inputting the multi-sensor data into the GRU model according to a preset time window to obtain dynamic change features; performing dimension matching between the dynamic change features and the first corrosion area feature to obtain the second corrosion area feature.

[0051] Specifically, first, the continuously collected multi-sensor data (such as temperature, humidity, vibration, electrochemical corrosion, and ultrasonic data) are divided according to a preset time window. In this way, the data within each time window can reflect the changes in the environmental and structural states during this period. The multi-sensor data sequence within each time window is input into the GRU model. GRU (Gated Recurrent Unit) can capture the temporal characteristics and long-term dependencies of the data, and thus output a feature vector describing the dynamic changes. This vector contains the change trends and patterns of the corrosion-related physical quantities during this period. In order to perform subsequent fusion between the dynamic change features extracted from multi-sensors and the first corrosion area features extracted from images, it is necessary to perform dimension matching on them. That is, the dynamic features obtained by GRU are adjusted to the same dimension as the first corrosion area feature (for example, both are 128-dimensional), so as to obtain the so-called "second corrosion area feature". This feature synthesizes the dynamic information of multi-sensor data and provides supplementary information for further multi-modal fusion and corrosion risk assessment.

[0052] For example, assume that temperature, humidity, vibration, and electrochemical corrosion sensors are deployed in a certain target sound barrier area, and the data are sampled and recorded every second. In order to capture the dynamic changes of the corrosion process, the system sets a 5-minute time window and arranges the sensor data during this period (such as 300 seconds of data within 5 minutes) into a sequence and inputs it into a pre-trained GRU model. The GRU model analyzes the change trends of temperature, humidity, vibration intensity, and electrochemical signals within these 300 seconds and outputs a 128-dimensional dynamic feature vector, which describes the evolution of the corrosion-related physical characteristics over time during this period.

[0053] Meanwhile, the first corrosion area features (also 128-dimensional) have been extracted from the image data before, describing the static features such as the texture, color change, and cracks of the corrosion area. To facilitate subsequent data fusion, the system matches the dimensions of the dynamic features output by the GRU with the image features to obtain the unified "second corrosion area features". Such features not only contain the temporal dynamic information of multi-sensor data but also are aligned with the image features in terms of dimensions, facilitating the construction of a more comprehensive multi-modal fusion model to evaluate the corrosion risk.

[0054] S140. Based on the first corrosion area features and the second corrosion area features, the corrosion risk score of the target area is calculated using a spatio-temporal graph neural network.

[0055] Specifically, a spatio-temporal graph neural network is constructed to comprehensively analyze this multi-modal information and finally output the corrosion risk score of the target area. Specifically, first, the first corrosion area features (such as texture, color change, cracks) extracted from the image by the server and the second corrosion area features (reflecting dynamic changes) encoded from multi-sensor data through GRU are introduced as input nodes into the graph neural network. When constructing this network, not only the spatial relationships between different features are considered, but also the temporal information (through temporal edges or temporal convolutional layers) is used to capture the evolution trend of these features over time. Through the graph convolutional layer for local information aggregation and the temporal convolutional layer for capturing dynamic changes, and finally, the attention mechanism is used to weight the fused data to generate a comprehensive and quantified corrosion risk score.

[0056] For example, assume that in a target area of a highway sound barrier, the image collected by the camera is preprocessed and segmented by U-Net to extract the feature vector of local rust spots (the first corrosion area features); meanwhile, the temperature, humidity, vibration, and electrochemical data collected by multi-sensors are encoded by the GRU model to output the feature vector (the second corrosion area features) describing the dynamic changes within 5 minutes. The server takes these two feature vectors as the inputs of different nodes in the graph neural network respectively, and the network conducts feature fusion by constructing a spatio-temporal graph (including the connections between different modal nodes at the same time and the temporal connections between nodes at different time steps). Finally, the model outputs a corrosion risk score, such as 0.85, indicating that the corrosion in this area is severe. This score can be used to automatically trigger an early warning or guide maintenance personnel to conduct on-site inspections.

[0057] In a possible implementation, based on the first corrosion area feature and the second corrosion area feature, a corrosion risk score of the target area is calculated using a spatio-temporal graph neural network, which specifically includes: taking the first corrosion area feature as the first type of node; taking the second corrosion area feature as the second type of node; obtaining a first set of target nodes at different time steps within the target area from the first type of node and the second type of node; establishing a temporal edge between the first set of target nodes; obtaining a second set of target nodes corresponding to the same time step within the target area from the first type of node and the second type of node; performing weighted connection between the second set of target nodes to obtain a modal edge; constructing a spatio-temporal graph based on the first type of node, the second type of node, the temporal edge, and the modal edge.

[0058] Specifically, the first type of node takes the corrosion area features (such as texture, color change, crack) extracted from the image data as the first type of node. The second type of node takes the dynamic change features obtained by GRU encoding from multi-sensor data as the second type of node. From these two types of nodes, a first set of target nodes is constructed according to different time steps, representing the first type of nodes collected at different moments in the same target area. At the same time, a second set of target nodes is constructed from the two types of nodes according to the same time step, representing the corresponding features of the image and sensor information at the same moment.

[0059] A temporal edge is established between the first set of target nodes to connect nodes at different times to capture the temporal evolution trend of the corrosion state. The modal edge performs weighted connection between the second set of target nodes to associate the image features and sensor features at the same time step, fully integrating information from different modalities. A spatio-temporal graph is constructed using the above nodes (the first type and the second type of nodes), the temporal edge, and the modal edge. This graph structure not only reflects the temporal change trend of the target area but also embodies the complementary relationship between multi-modal data.

[0060] This method makes full use of the advantages of multi-modal data: the temporal edge captures the temporal evolution features, and the modal edge integrates information from different data sources, enabling the risk assessment to consider both static image features and dynamic sensor data, thereby improving the accuracy and real-time performance of corrosion risk detection.

[0061] In a possible implementation, based on the first corrosion area feature and the second corrosion area feature, calculating the corrosion risk score of the target area using a spatio-temporal graph neural network specifically further includes: performing feature aggregation on the spatio-temporal graph using the graph convolutional layer of the spatio-temporal graph neural network to fuse the information between different nodes to obtain a first data to be fused; using the temporal convolutional layer of the spatio-temporal graph neural network to capture the change trend of the spatio-temporal graph to obtain a second data to be fused; performing attention-weighted aggregation on the first data to be fused and the second data to be fused to generate the corrosion risk score of the target area.

[0062] Specifically, on the constructed spatio-temporal graph, graph convolutional network (GCN) is used to fuse the information between nodes and improve the feature expression ability. GCN takes into account the features of the nodes themselves and updates by combining the information of adjacent nodes, enabling the image features and sensor features in the corrosion area to complement each other and improving the accuracy of corrosion detection. For example, there is information flow between nodes at adjacent time steps, allowing the corrosion features at one time step to draw on the data from the previous and subsequent moments and reducing misjudgments. After graph convolution, the first data to be fused is obtained, which represents the feature vector that has fused the information of different nodes inside the spatio-temporal graph. Through temporal convolutional network (TCN), the changing trend of corrosion risk over time is analyzed to extract the dynamic pattern of the corrosion state. TCN adopts a one-dimensional convolutional structure and performs a sliding window operation on the time dimension of the spatio-temporal graph to analyze the changing trend of historical data. For example, if the corrosion features in a certain sound barrier area have gradually increased in the past few weeks (color change, crack deepening, abnormal sensor signals, etc.), TCN can capture this trend and improve the prediction accuracy.

[0063] After temporal convolution, the second data to be fused is obtained, which represents that the system has learned the temporal evolution pattern of the corrosion state. The data at different time steps and different modalities have different degrees of influence on the final risk score. Therefore, an attention mechanism needs to be used to assign higher weights to important features to generate the final corrosion risk score. Through the self-attention mechanism, the system automatically calculates which features are more important when predicting the current corrosion state and assigns different weights. For example, if the data collected in the recent few times indicates a sudden change in the corrosion state (crack propagation, soaring electrochemistry corrosion index), higher weights are assigned to these data, while the weights of the earlier data are reduced. By attentional weighted aggregation of the first data to be fused and the second data to be fused, the final corrosion risk score is generated. This score is usually between 0 and 1, or represents the severity of corrosion according to the set risk level (such as 0 - 100).

[0064] For example, assume that the sound barriers on a highway are long-term exposed to a high-humidity environment, and the operation and maintenance system monitors them through a deployed high-definition camera and sensor network. The camera takes a surface image of the sound barrier every 6 hours. The rusty areas are detected by the U-Net model, and the color change, texture abnormality, and crack information are extracted. For example, the image taken at 8:00 in the morning on July 1 shows slight spots (feature vector F_img_8:00), while the image taken at 8:00 in the morning on July 3 shows that the spots have expanded and there are slight cracks (feature vector F_img_8:00→F_img_8:48:00).

[0065] Temperature and humidity sensors, vibration sensors, electrochemical corrosion sensors, ultrasonic sensors, etc. continuously collect data and obtain the dynamic corrosion change characteristics through GRU encoding. For example, the corrosion potential of the electrochemical sensor has decreased significantly in the recent week (-0.2V → -0.5V), and the ultrasonic thickness detection results show that the material thickness has decreased (5mm → 4.6mm). The data at each time step is used as a node, and feature aggregation is performed through GCN. For example, a modal edge is established between F_img_8:00 and F_sensor_8:00 to fuse image and sensor data. In the time dimension, a temporal edge is established between F_img_8:00 and F_img_8:48:00 to capture the temporal evolution of the corrosion state.

[0066] Graph convolution is performed on the spatio-temporal graph, enabling data from different time steps and different modalities to complement each other. Time series modeling is performed on the corrosion characteristics at different time steps to learn the corrosion trend. Since the corrosion index has suddenly increased in recent days, the attention mechanism assigns higher weights to the most recent time steps (July 1st - 3rd) and lower weights to earlier time steps (data from mid-June). After multiple layers of calculations, the system outputs the corrosion risk scores: July 1st: score 0.35 (mild corrosion), July 3rd: score 0.75 (moderate corrosion), and it is predicted that in the next 5 days, the score may exceed 0.9 (severe corrosion). By combining image and sensor data, the limitations of a single data source are avoided, and the detection accuracy is improved. Time trend modeling: The temporal convolution network (TCN) is used to learn the variation law of the corrosion state over time, enhancing the prediction ability. The attention mechanism can automatically adjust the importance of data at different time steps, giving priority to abnormal signals in the recent period, and improving the sensitivity of prediction. An explicit corrosion risk score is output, supporting hierarchical early warning (such as normal, mild, moderate, severe), which is convenient for subsequent operation and maintenance decisions. This method can help operation and maintenance personnel detect the corrosion problems of the sound barrier in advance, and thus take corresponding maintenance measures (such as repairing the anti-corrosion coating, replacing severely corroded components), reducing safety risks and maintenance costs.

[0067] S150. Determine the corrosion state of the target sound barrier according to the corrosion risk score.

[0068] Specifically, the corrosion risk score calculated by the server using the spatio-temporal graph neural network usually ranges from 0 to 1 (or between 0 and 100). This score is based on image data (color changes, cracks, spots) and multi-sensor data (temperature and humidity, electrochemical corrosion, electromagnetic detection, ultrasonic thickness, etc.) to comprehensively evaluate the corrosion situation. The server pre-sets multiple thresholds to divide the corrosion score into different levels: 0.0 ≤ score < 0.3 → normal; 0.3 ≤ score < 0.5 → mild corrosion; 0.5 ≤ score < 0.8 → moderate corrosion; 0.8 ≤ score ≤ 1.0 → severe corrosion. These thresholds can be adjusted according to experimental data and engineering experience to ensure reasonable classification. The server compares the calculated corrosion risk score with the thresholds to determine the corrosion state of the current sound barrier.

[0069] In a possible implementation, referring to Figure 2 , Figure 2 is another process schematic diagram of a method for identifying the corrosion state of a sound barrier that integrates image and multi-sensor data provided by an embodiment of this application. It includes steps S210 to S220, and the above steps are as follows: S210. Compare the corrosion risk score with the preset threshold to determine the corrosion state, where the corrosion state includes normal, mild corrosion, moderate corrosion, and severe corrosion; S220. Generate an operation and maintenance warning for the target sound barrier according to the corrosion state.

[0070] Specifically, a score reflecting the corrosion degree of the sound barrier is calculated through the spatio-temporal graph neural network. This score usually ranges from 0 to 1 (or between 0 and 100), and the larger the value, the more severe the corrosion. The system compares the calculated corrosion risk score with the pre-set threshold. The threshold can be defined according to different corrosion states. Based on the comparison between the corrosion score and the threshold, the system determines the corrosion state of the target sound barrier. The classification of the corrosion state helps to quantify the corrosion degree, facilitating decision-makers to understand and take appropriate maintenance measures. Generate an operation and maintenance warning: According to the determined corrosion state, the system generates a corresponding operation and maintenance warning. The role of the warning is to remind the operation and maintenance personnel to conduct regular inspections or take repair measures to avoid more serious damage.

[0071] Illustrate with an example. Case: Sound barrier on an urban expressway Suppose there is a sound barrier on an urban expressway that is often exposed to a humid and polluted environment, and the operation and maintenance server needs to regularly evaluate its corrosion situation. The server collects data through high-definition cameras and sensors (such as electrochemical corrosion sensors, ultrasonic sensors).

[0072] After the calculation by the spatio-temporal graph neural network, the server obtained the corrosion risk score of the target sound barrier. For example: Score: 0.32 (representing mild corrosion). The server classifies the score according to the set thresholds (for example, 0 ≤ score < 0.3 → normal, 0.3 ≤ score < 0.5 → mild corrosion, 0.5 ≤ score < 0.8 → moderate corrosion, 0.8 ≤ score ≤ 1.0 → severe corrosion). In this example, the score of 0.32 falls within the range of mild corrosion, so the system determines the state of the sound barrier as "mild corrosion". According to the corrosion state, the server generated an operation and maintenance warning of "mild corrosion", suggesting that the maintenance team conduct an inspection and perform repair or reinforcement treatment within an appropriate time to prevent the corrosion situation from deteriorating.

[0073] Case: Sound barrier on the coastal road in a coastal city. On the coastal road in a coastal city, due to the salt mist in the air, the sound barrier is prone to corrosion. The server also collects data through cameras and sensors (such as temperature and humidity sensors, electrochemical corrosion sensors).

[0074] Suppose the calculated corrosion risk score is 0.78 (representing moderate corrosion). According to the set thresholds, the score of 0.78 belongs to the range of moderate corrosion, so the system determines that the sound barrier is in a moderate corrosion state. Since moderate corrosion may lead to a decrease in structural strength, the server generates an operation and maintenance warning of "moderate corrosion", prompting the maintenance team to perform repairs as soon as possible to avoid further corrosion.

[0075] This application also provides a sound barrier corrosion state recognition device that fuses image and multi-sensor data. Refer to Figure 3 , Figure 3 is a schematic diagram of the modules of a sound barrier corrosion state recognition device that fuses image and multi-sensor data provided in an embodiment of this application. The device is a server, and the server includes an acquisition module 31 and a processing module 32. Among them, the acquisition module 31 acquires image data and multi-sensor data of the target area for the target sound barrier; the processing module 32 extracts the first corrosion area feature from the image data; the processing module 32 constructs the second corrosion area feature by using GRU encoding according to the multi-sensor data; the processing module 32 calculates the corrosion risk score of the target area by using a spatio-temporal graph neural network based on the first corrosion area feature and the second corrosion area feature; the processing module 32 determines the corrosion state of the target sound barrier according to the corrosion risk score.

[0076] In a possible implementation, the acquisition module 31 acquires image data and multi-sensor data for a target area of a target sound barrier, specifically including: the acquisition module 31 acquires the original image for the target area sent by a high-definition camera; the processing module 32 performs denoising on the original image using median filtering, and extracts the areas with spots in the target area by using a pre-trained U-Net model, outputs a binary segmentation map, and obtains the image data; the acquisition module 31 acquires temperature and humidity data, vibration data, electrochemical corrosion data, and ultrasonic data sent by a temperature and humidity sensor, a vibration sensor, an electrochemical corrosion sensor, and an ultrasonic sensor; the processing module 32 performs filtering and normalization processing on the temperature and humidity data, vibration data, electrochemical corrosion data, and ultrasonic data to obtain multi-sensor data.

[0077] In a possible implementation, the processing module 32 extracts the first corrosion area features from the image data, specifically including: the processing module 32 segments the image data using a segmentation algorithm to obtain abnormal areas; the processing module 32 extracts features from the abnormal areas by using a pre-trained convolutional neural network to obtain the first corrosion area features, and the first corrosion area features include texture, color change, and cracks.

[0078] In a possible implementation, the processing module 32 constructs the second corrosion area features by using GRU encoding according to the multi-sensor data, specifically including: the processing module 32 inputs the multi-sensor data into the GRU model according to a preset time window to obtain dynamic change features; the processing module 32 performs dimension matching on the dynamic change features and the first corrosion area features to obtain the second corrosion area features.

[0079] In a possible implementation, the processing module 32 calculates the corrosion risk score of the target area by using a spatio-temporal graph neural network based on the first corrosion area features and the second corrosion area features, specifically including: the processing module 32 uses the first corrosion area features as the first type of nodes; the processing module 32 uses the second corrosion area features as the second type of nodes; the processing module 32 obtains a first target node set at different time steps in the target area from the first type of nodes and the second type of nodes; the processing module 32 establishes a temporal edge between the first target node sets; the processing module 32 obtains a second target node set corresponding to the same time step in the target area from the first type of nodes and the second type of nodes; the processing module 32 performs weighted connection between the second target node sets to obtain a modal edge; the processing module 32 constructs a spatio-temporal graph based on the first type of nodes, the second type of nodes, the temporal edge, and the modal edge.

[0080] In a possible implementation, the processing module 32 calculates the corrosion risk score of the target area by using a spatio-temporal graph neural network based on the first corrosion area feature and the second corrosion area feature. Specifically, it further includes: the processing module 32 uses the graph convolutional layer of the spatio-temporal graph neural network to perform feature aggregation on the spatio-temporal graph, fuses the information between different nodes, and obtains the first data to be fused; the processing module 32 uses the temporal convolutional layer of the spatio-temporal graph neural network to capture the change trend of the spatio-temporal graph and obtains the second data to be fused; the processing module 32 performs attention-weighted aggregation on the first data to be fused and the second data to be fused to generate the corrosion risk score of the target area.

[0081] In a possible implementation, the processing module 32 compares the corrosion risk score with a preset threshold to determine the corrosion state. The corrosion state includes normal, mild corrosion, moderate corrosion, and severe corrosion; the processing module 32 generates an operation and maintenance warning for the target sound barrier according to the corrosion state.

[0082] It should be noted that: when the device provided in the above embodiment realizes its functions, only the division of the above functional modules is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be seen in the method embodiment, which will not be repeated here.

[0083] This application also provides an electronic device. Refer to Figure 4 , Figure 4 which is a schematic structural diagram of an electronic device provided in an embodiment of this application. The electronic device may include: at least one processor 41, at least one network interface 44, a user interface 43, a memory 45, and at least one communication bus 42.

[0084] Among them, the communication bus 42 is used to realize the connection and communication between these components.

[0085] Among them, the user interface 43 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 43 may further include a standard wired interface and a wireless interface.

[0086] Among them, the network interface 44 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface).

[0087] Among them, the processor 41 may include one or more processing cores. The processor 41 connects various parts within the entire server through various interfaces and lines. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 45, and by calling the data stored in the memory 45, it executes various functions of the server and processes data. Optionally, the processor 41 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 41 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 41 and may be implemented separately through a single chip.

[0088] Among them, the memory 45 may include random access memory (RAM) and may also include read-only memory. Optionally, the memory 45 includes a non-transitory computer-readable storage medium. The memory 45 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 45 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store the data involved in the above-mentioned various method embodiments. Optionally, the memory 45 may also be at least one storage device located far from the aforementioned processor 41. As Figure 4 shown, the memory 45, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for a method of identifying the corrosion state of a sound barrier by fusing images and multi-sensor data.

[0089] In Figure 4In the electronic device shown, the user interface 43 is mainly used to provide an interface for the user to input and obtain the data input by the user; and the processor 41 can be used to call an application program stored in the memory 45 for a method of identifying the corrosion state of a sound barrier by fusing an image and multi-sensor data. When executed by one or more processors, the electronic device executes the method as described in one or more of the above embodiments.

[0090] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0091] The present application also provides a computer-readable storage medium storing instructions. When executed by one or more processors, the electronic device executes the method as described in one or more of the above embodiments.

[0092] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0093] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some service interfaces. The indirect couplings or communication connections of the devices or units can be in electrical or other forms.

[0094] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0095] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0096] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned memory includes: various media such as USB flash drives, mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0097] The above are only exemplary embodiments of the present disclosure, and the scope of the present disclosure cannot be limited thereby. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure still fall within the scope covered by the present disclosure. Those skilled in the art will easily think of other implementation manners of the present disclosure after considering the specification and the practice of the present disclosure. The present application aims to cover any variations, uses, or adaptive changes of the present disclosure, and these variations, uses, or adaptive changes follow the general principles of the present disclosure and include the common general knowledge or conventional technical means in the technical field not recorded in the present disclosure. The specification and the embodiments are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for identifying the corrosion state of a sound barrier by fusing images and multi-sensor data, characterized in that, The method includes: Obtaining image data and multi-sensor data for a target area of a target sound barrier; Extracting first corrosion area features from the image data; Constructing second corrosion area features by using GRU encoding according to the multi-sensor data; Calculating a corrosion risk score for the target area by using a spatio-temporal graph neural network based on the first corrosion area features and the second corrosion area features; Determining the corrosion state of the target sound barrier according to the corrosion risk score.

2. The method for identifying the corrosion state of a sound barrier by fusing an image and multi-sensor data according to claim 1, characterized in that, The obtaining of the image data and multi-sensor data for the target area of the target sound barrier specifically includes: Obtaining an original image for the target area sent by a high-definition camera; Performing denoising on the original image by using median filtering, and extracting the areas with spots in the target area by using a pre-trained U-Net model, outputting a binary segmentation map to obtain the image data; Obtaining temperature and humidity data, vibration data, electrochemical corrosion data, and ultrasonic data sent by a temperature and humidity sensor, a vibration sensor, an electrochemical corrosion sensor, and an ultrasonic sensor; Filtering and normalizing the temperature and humidity data, the vibration data, the electrochemical corrosion data, and the ultrasonic data to obtain the multi-sensor data.

3. The method for identifying the corrosion state of a sound barrier by fusing an image and multi-sensor data according to claim 1, wherein The extracting of the first corrosion area features from the image data specifically includes: Segmenting the image data by using a segmentation algorithm to obtain abnormal areas; Extracting features of the abnormal areas by using a pre-trained convolutional neural network to obtain the first corrosion area features, where the first corrosion area features include texture, color change, and cracks.

4. The method for identifying the corrosion state of the sound barrier by fusing the image and multi-sensor data according to claim 1, wherein The constructing of the second corrosion area features by using GRU encoding according to the multi-sensor data specifically includes: Inputting the multi-sensor data into a GRU model according to a preset time window to obtain dynamic change features; Performing dimension matching on the dynamic change features and the first corrosion area features to obtain the second corrosion area features.

5. The method for identifying the corrosion state of a sound barrier by fusing an image and multi-sensor data according to claim 1, characterized in that The calculating of the corrosion risk score for the target area by using a spatio-temporal graph neural network based on the first corrosion area features and the second corrosion area features specifically includes: Taking the first corrosion area features as the first type of nodes; Taking the second corrosion area features as the second type of nodes; Obtaining a first target node set at different time steps within the target area from the first type of nodes and the second type of nodes; Establishing a time series edge between the first target node sets; Obtaining a second target node set corresponding to the same time step within the target area from the first type of nodes and the second type of nodes; Performing weighted connection between the second target node sets to obtain a modal edge; Constructing a spatio-temporal graph based on the first type of nodes, the second type of nodes, the time series edge, and the modal edge.

6. The method for identifying the corrosion state of a sound barrier by fusing an image and multi-sensor data according to claim 5, wherein The calculating of the corrosion risk score for the target area by using a spatio-temporal graph neural network based on the first corrosion area features and the second corrosion area features specifically further includes: Use the graph convolutional layer of the spatio-temporal graph neural network to perform feature aggregation on the spatio-temporal graph, fuse the information between different nodes, and obtain the first data to be fused; Use the temporal convolutional layer of the spatio-temporal graph neural network to capture the change trend of the spatio-temporal graph and obtain the second data to be fused; Perform attention-weighted aggregation on the first data to be fused and the second data to be fused to generate the corrosion risk score of the target area.

7. The method for identifying the corrosion state of the sound barrier by fusing the image and multi-sensor data according to claim 1, characterized in that, The method further includes: Compare the corrosion risk score with a preset threshold to determine the corrosion state, where the corrosion state includes normal, mild corrosion, moderate corrosion, and severe corrosion; Generate an operation and maintenance warning for the target sound barrier according to the corrosion state.

8. An acoustic barrier corrosion state recognition device that integrates images and multi-sensor data, characterized in that, The device includes an acquisition module (31) and a processing module (32), where The acquisition module (31) is used to acquire image data and multi-sensor data of a target area of a target sound barrier; The processing module (32) is used to extract the first corrosion area feature from the image data; The processing module (32) is further used to construct a second corrosion area feature by using GRU encoding according to the multi-sensor data; The processing module (32) is further used to calculate the corrosion risk score of the target area by using a spatio-temporal graph neural network based on the first corrosion area feature and the second corrosion area feature; The processing module (32) is further used to determine the corrosion state of the target sound barrier according to the corrosion risk score.

9. An electronic device, characterized in that, The electronic device includes a processor (41), a memory (45), a user interface (43), and a network interface (44). The memory (45) is used to store instructions. Both the user interface (43) and the network interface (44) are used to communicate with other devices. The processor (41) is used to execute the instructions stored in the memory (45) so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.

Images