A machine vision-based spraying robot crack identification and adaptive spraying method, device and medium

Abstract

The application discloses a kind of based on machine vision's spraying robot crack identification and adaptive spraying method, equipment and medium, it is related to the field of spraying, the method includes the following steps: S1, real-time acquisition multimodal original data stream, and generate alignment, denoising and enhanced multimodal data;S2, from multimodal data extraction and transform out fusion feature pyramid;S3, to fusion feature pyramid is segmented, extracts the morphological parameter and position coordinate of crack region;S4, output optimization spraying parameter and local trajectory adjustment instruction, carries out repair spraying operation;S5, carries out real-time visual feedback and quality verification, if spraying quality does not reach preset standard, strategy adjustment is carried out again.The application can effectively overcome the limitation caused by noise propagation when multi-scale feature alignment under complex bridge environment and feature drift in dynamic spraying process, realize the improvement of crack identification accuracy and spraying repair quality and efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of spraying technology, in particular to image data processing, and specifically to a crack identification and adaptive spraying method, device and medium for a spraying robot based on machine vision. BACKGROUND

[0002] As an important part of modern transportation infrastructure, steel structure bridges are prone to surface cracks during long-term service due to the influence of load, environment and material aging. If not detected and repaired in time, crack propagation will threaten the safety of the structure and shorten the service life of the bridge. Traditional manual inspection methods have been difficult to meet multiple indicators such as efficiency, accuracy and safety. In this context, intelligent spraying robot technology based on machine vision has emerged and rapidly become a research hotspot and development direction in the field. This technology integrates advanced image acquisition, intelligent identification and precision control systems to achieve automated and accurate identification of structural surface defects such as cracks, and on this basis, performs adaptive repair spraying operations, thereby significantly improving the intelligent level and overall efficiency of maintenance operations.

[0003] In the prior art, convolutional neural networks (CNN) are often used for image recognition of spraying areas, and the spraying path, nozzle movement speed, spraying pressure and coating flow are determined according to the morphological characteristics of the cracks. For example, the Chinese invention patent CN109146849A discloses a road crack detection method based on convolutional neural networks and image recognition. This method is safe and reliable, fast in detection speed, high in detection efficiency, low in false alarm rate, and does not have night work fatigue phenomenon. It can accurately determine the type of highway defects, and relevant departments can take reasonable measures to repair the road defects in time.

[0004] However, although CNN has strong feature extraction and classification capabilities in image recognition tasks, its direct application in real bridge maintenance scenarios is not ideal due to the complex and variable bridge surface environment, strong light changes, rain and fog weather interference, rust stains, welds, and various noise sources such as coating peeling. A CNN model trained under specific lighting and background conditions often overfits to non-critical features such as background texture and shadows in the training data, i.e., the model learns the "shortcut" in the data rather than the essential features of the cracks. When the environmental conditions change (such as the difference in morning and evening light, new rust morphology), the recognition performance of the model will decrease significantly, leading to missed or false detection of cracks, and thus affecting the accuracy of subsequent spraying path planning and spraying quality. SUMMARY

[0005] The present application overcomes the shortcomings of the prior art and provides a crack identification and adaptive spraying method, device and medium for a spraying robot based on machine vision.

[0006] To achieve the above object, the technical scheme adopted by the present application is as follows: In a first aspect, the present application provides a crack identification and adaptive spraying method for a spraying robot based on machine vision, comprising the following steps:

[0007] S1, real-time acquisition of multi-modal original data stream of the bridge structure surface to be detected, and generation of multi-modal data after time-space alignment, noise suppression and feature enhancement; the multi-modal original data stream comprises high-resolution color images, high-precision depth images and posture and motion information of the spraying robot;

[0008] S2, from the multi-modal data, a multi-modal feature transformation unit is used to extract and transform a fusion feature pyramid with multi-scale time dynamic changes; the multi-modal feature transformation unit comprises a parallel feature extraction network, a group of intra-modal time sequence feature transformers and a cross-modal spatial attention fusion module;

[0009] S3, the fusion feature pyramid is segmented by a feature extraction and segmentation unit to identify and extract the morphological parameters and position coordinates of the crack region in the three-dimensional space;

[0010] S4, based on the morphological parameters, position coordinates and current posture information of the spraying robot, output of optimized spraying parameters and local trajectory adjustment instructions for repair spraying operation.

[0011] In a preferred embodiment of the present application, in the step of S1, the time-space alignment of the multi-modal original data stream comprises:

[0012] acquisition of each group of color images and depth images at the same time;

[0013] registration of the depth images and color images to the same coordinate system by using the pre-calibrated external parameters of each sensor to generate pixel-level aligned RGB-D image pairs; and

[0014] timestamp alignment of the posture and motion information data of the spraying robot and the RGB-D image pairs to generate uniformly sequenced data frames.

[0015] In a preferred embodiment of the present application, in the step of S1, the noise suppression of the multi-modal original data stream comprises:

[0016] application of a non-local mean filter-based image denoising algorithm to the color images;

[0017] application of a bilateral filter or median filter-based denoising algorithm to the depth images; and

[0018] application of Kalman filtering or complementary filtering to the posture and motion information data of the spraying robot.

[0019] In a preferred embodiment of the present application, in the step of S1, the feature enhancement of the multi-modal raw data stream comprises:

[0020] Applying adaptive histogram equalization to color images to enhance image contrast, and extracting gradient information of the images;

[0021] Applying depth gradient calculation to depth images to highlight the small height changes of the surface; and

[0022] Based on the pose and motion information data of the spraying robot, the instantaneous speed and acceleration vector of the robot are calculated as the context input of the subsequent feature transformation.

[0023] In a preferred embodiment of the present application, in the step of S2, the parallel feature extraction network comprises independent feature extraction paths for color images and depth images, and each path adopts a lightweight convolutional neural network as a backbone network for extracting a multi-scale initial feature map set from a single frame of image.

[0024] The intra-modal temporal feature transformer group comprises at least two parallel temporal transformer modules corresponding to the color image modal and the depth image modal, respectively, each temporal transformer module receives a multi-frame, multi-scale feature map sequence of its corresponding modal from the parallel feature extraction network, and captures the dynamic change pattern of the crack feature in the time dimension through an encoder structure based on the Transformer architecture, including a multi-head self-attention mechanism and a feedforward neural network, to generate a multi-scale temporal feature map set in the time dimension which is stable and robust to feature drift.

[0025] In a preferred embodiment of the present application, in the multi-modal feature transformation unit, the pose and motion information of the spraying robot is further integrated into the input of the intra-modal temporal feature transformer group.

[0026] Specifically, the pose and motion information of the spraying robot is processed through a one-dimensional convolutional neural network to extract its temporal features, and the temporal features are encoded as conditions and spliced or fused through a gating mechanism with the feature vectors of the color image modal and the depth image modal, which are jointly input into the temporal transformer module to provide additional context information.

[0027] In a preferred embodiment of the present application, the cross-modal spatial attention fusion module is used to receive the multi-scale feature map set of the color image modal and the multi-scale feature map set of the depth image modal from the intra-modal temporal feature transformer group, and for each feature pyramid level, the following operations are performed:

[0028] The feature map of the color image modal and the feature map of the depth image modal are upsampled or downsampled to align their spatial resolutions.

[0029] A query, key and value matrix is constructed, and a cross-modal spatial attention mechanism is adopted to assign attention weights to the value matrix by calculating the similarity between the query and the key, so as to realize feature fusion from different modalities and suppress the influence of structured noise through depth information.

[0030] The cross-modal spatial attention fusion module outputs a set of multi-scale feature pyramids that fuse color and depth information and are accurately aligned in the spatial dimension and noise suppressed.

[0031] In a preferred embodiment of the present application, in the step of S3, the feature extraction and segmentation unit adopts a segmentation network based on an encoder-decoder architecture, the encoder part of the segmentation network is composed of or connected to the fusion feature pyramid, and the decoder part receives the multi-scale fusion features output by the encoder and gradually restores the spatial resolution through upsampling, skip connection and convolutional layers, and finally outputs a pixel-level crack probability map or a binary segmentation mask.

[0032] The morphological parameters of the crack are accurately extracted through morphological processing and geometric analysis algorithms, including the length, average width, maximum width, direction, connectivity, area and position coordinates in three-dimensional space of the crack.

[0033] By matching the segmented crack pixels with the corresponding depth image data, the point cloud representation of the crack in three-dimensional space is obtained, and based on this point cloud data, the actual three-dimensional shape, depth profile information and surface roughness of the crack and other advanced geometric features are calculated.

[0034] Second aspect: the present application provides an electronic device, comprising: at least one processor;And the memory connected with at least one processor in communication;

[0035] Wherein, the memory stores the computer program executed by at least one processor, and the computer program is executed by at least one processor to enable at least one processor to execute the machine vision based spraying robot crack identification and adaptive spraying method in any one of the above.

[0036] Third aspect: the present application provides a computer readable storage medium, the computer readable storage medium stores computer instructions, the computer instructions are used to make processor execute and realize the machine vision based spraying robot crack identification and adaptive spraying method in any one of the above.

[0037] The present application solves the defects in the background art, and has the following beneficial effects:

[0038] (1) The application provides a spraying robot crack identification and adaptive spraying method, equipment and medium based on machine vision, through the introduction of a multi-modal data fusion mechanism, synchronous acquisition and processing of color images, depth images and robot posture and motion information, construction of a multi-modal data base with time and space alignment, denoising and enhancement, comprehensive utilization of visual texture and three-dimensional geometric information, significant enhancement of crack feature perception ability, especially in complex backgrounds such as light change, rust and weld, effective differentiation between real cracks and structured noise, thereby improving the accuracy of crack detection and further enhancing the accuracy of subsequent spraying path planning and spraying quality.

[0039] (2) In the application, a group of intra-modal time sequence feature transformers is designed in the feature extraction stage, a Transformer-based encoder structure is used to process continuous multi-frame feature sequences, and a multi-head self-attention mechanism is used to model the dynamic changes of crack features in the time dimension, which can effectively suppress feature drift caused by robot motion, light fluctuation and the like, enhance the stability and consistency of features in time sequence, and compared with a static FPN structure, the adaptability to dynamic working environment can be significantly improved, thereby reducing missed detection and false detection and ensuring the reliability of crack identification results in continuous operation.

[0040] (3) In the application, through the cooperation of the cross-modal spatial attention fusion module, the aligned RGB and depth features are used to construct query, key and value mapping, and intelligent feature fusion and noise suppression are realized through attention weight, which can effectively distinguish texture similar interference areas (such as rust spots and welds) according to depth information, suppress the response of non-crack structures, and further improve the signal-to-noise ratio of the multi-scale feature pyramid, compared with the traditional multi-scale fusion method, the blind propagation of noise in the traditional FPN can be avoided, and it has stronger anti-interference ability in complex bridge background, especially suitable for accurate identification and geometric parameter extraction of small cracks, thereby improving the accuracy of crack identification and the quality and efficiency of spraying repair. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, brief descriptions will be given to the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments described in the application, and for those skilled in the art, other drawings can also be obtained without creative labor;

[0042] Figure 1 is the flow chart of the spraying robot crack identification and adaptive spraying method based on machine vision of the embodiment 1 of the application;

[0043] Figure 2is a structure block diagram of a machine vision-based spraying robot crack identification and adaptive spraying method of embodiment 1 of the present application;

[0044] Figure 3 is a structure block diagram of a multi-modal feature transformation unit of embodiment 1 of the present application;

[0045] Figure 4 is a structure schematic diagram of an electronic device that can be used to implement embodiment 1 of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0047] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in a variety of ways beyond the specific embodiments described herein, and the scope of the present application is not limited to the specific embodiments disclosed below.

[0048] SUMMARY

[0049] To improve the generalization ability of traditional convolutional neural networks in bridge crack identification due to environmental complexity and multi-scale interference, some studies attempt to introduce a feature pyramid network (FPN) structure. FPN can extract deep semantic features and shallow detail features of an image in parallel, and realize multi-scale feature fusion through horizontal connection, which can theoretically enhance the model's perception ability for cracks of different scales, while suppressing background interference using attention mechanisms to improve recognition accuracy in complex backgrounds such as rust and welds.

[0050] However, the applicant found that when FPN is directly applied to the bridge crack identification and spraying robot operation scene, there are still two outstanding problems due to the limitations of its structural principle: first, the noise propagation mechanism. A large number of structured disturbances on the bridge surface (such as rust spots and weld textures) enter the FPN through shallow features and are propagated and even semantized in the feature fusion process, which is difficult to be effectively suppressed, but instead pollutes the deep features and reduces the purity of crack features. Second, the feature drift problem. In dynamic operation of the spraying robot, factors such as camera pose, paint surface reflection, and environmental lighting can cause image features to drift in time sequence, and the FPN trained statically lacks the ability to adapt to such dynamic changes, resulting in unstable recognition results.

[0051] To solve the above problems, the application provides a spraying robot crack identification and adaptive spraying method based on machine vision, which introduces multi-modal perception, dynamic timing feature transformation, adaptive gated cross-modal attention fusion and reinforcement learning driven spraying parameter optimization mechanism, can effectively overcome the limitations caused by noise propagation in multi-scale feature alignment and feature drift in dynamic spraying process in complex bridge environment, and realize the improvement of crack identification accuracy and spraying repair quality and efficiency.

[0052] Embodiment 1: as shown in Figure 1 and Figure 2 A spraying robot crack identification and adaptive spraying method based on machine vision, comprising the following steps:

[0053] S1, real-time acquisition of multi-modal original data stream of the bridge structure surface to be detected, and generation of multi-modal data after time-space alignment, noise suppression and feature enhancement;

[0054] S2, from the multi-modal data, a multi-modal feature transformation unit is used to extract and transform a fusion feature pyramid with multi-scale dynamic changes over time;

[0055] S3, the fusion feature pyramid is segmented by a feature extraction and segmentation unit to identify and extract the morphological parameters of the crack area and the position coordinates in the three-dimensional space;

[0056] S4, based on the morphological parameters, position coordinates and current posture information of the spraying robot, output the optimized spraying parameters and local trajectory adjustment instructions for repair spraying operation;

[0057] S5, during or after the spraying operation, real-time visual feedback and quality verification are performed, if the spraying quality does not meet the preset standard, the quality evaluation result is fed back as new state information, and strategy adjustment is performed again.

[0058] In one specific embodiment, in the step of S1, the multi-modal original data stream is collected by a multi-modal perception unit configured on the spraying robot body, which is used to collect the multi-modal original data stream of the bridge structure surface to be detected.

[0059] In this embodiment, the data stream covers high-resolution color images, high-precision depth images and posture and motion information sequence of the spraying robot; the multi-modal perception unit is composed of an RGB color camera, a depth camera and an inertial measurement unit (IMU).

[0060] The RGB color camera is preferably an industrial-grade color camera of Basler acA1920-40gc model, configured to capture image data in the visible light spectrum, with an optical resolution of 1920x1080 pixels and a frame rate of 30 frames per second; to minimize motion blur, the camera is integrated with a global shutter function to ensure image clarity during high-speed robot motion; the depth camera is preferably a structured light depth sensor of Intel RealSense D435i model, configured to obtain distance information from the sensor to the surface to be measured, with a depth measurement range of 0.1-5 m and a depth measurement accuracy of better than ±1 mm under ideal conditions.

[0061] It should be noted that the depth camera and the RGB color camera are time-synchronized at the millisecond level through a hardware triggering mechanism, ensuring high consistency in the acquisition time of each set of RGB-D image pairs.

[0062] Further, the IMU is preferably a six-axis inertial sensor of Analog Devices ADIS16470 model, configured to collect real-time three-axis acceleration data and three-axis angular velocity data around the X, Y, and Z axes of the robot, with a data update rate of 200 Hz, for accurately representing the instantaneous motion state of the robot, including linear velocity, angular velocity, and attitude including pitch, yaw, and roll.

[0063] Specifically, all sensors of the multi-modal perception unit are fixedly installed on the end effector of the spraying robot and are calibrated for internal and external parameters, wherein the camera internal parameter calibration uses Zhang's calibration method, and the external parameter calibration uses multi-camera joint calibration based on a checkerboard or a custom calibration board, to ensure high consistency of sensor data in space and time, laying a foundation for subsequent data fusion.

[0064] In this embodiment, the specific steps for spatio-temporal alignment of the multi-modal raw data stream are as follows:

[0065] 1. Start.

[0066] 2. Based on the timestamp information of the RGB color camera and the depth camera, perform time synchronization at the hardware or software level to ensure that each set of color images and depth images is acquired within 5 milliseconds;

[0067] 3. Use the pre-calibrated external parameters of each sensor, including rotation matrix and translation vector, to register the depth image to the coordinate system of the color image, generating a pixel-level aligned RGB-D image pair;

[0068] 4. Align the timestamps of the IMU collected pose and motion information data and the RGB-D image pairs, resample the IMU data using linear interpolation or nearest neighbor interpolation method to generate uniformly timed data frames, and ensure the synchronization of all modal information in time.

[0069] 5. End.

[0070] The specific steps of noise suppression processing of multi-modal raw data stream are:

[0071] 1. Start.

[0072] 2. Apply non-local mean filter-based image denoising algorithm to color image;

[0073] Wherein, the search window size is set to 21x21 pixels, the similarity calculation window size is set to 7x7 pixels, and the Gaussian weighting function standard deviation is set to 10, to effectively remove high-frequency random noise in the image while maximizing the clarity of edge structures such as cracks.

[0074] 3. Apply bilateral filter-based denoising algorithm to depth image;

[0075] Wherein, the kernel function size is set to 9x9 pixels, the color space standard deviation sigma_color is set to 75, and the spatial domain standard deviation sigma_space is set to 75, to smooth random fluctuations in depth data and effectively preserve depth edge information, avoiding crack boundary blurring.

[0076] 4. Apply extended Kalman filter to IMU data;

[0077] Wherein, the state vector contains robot position, velocity, attitude (quaternion) and gyroscope and accelerometer bias, by fusing accelerometer and gyroscope measurement values, more accurate and stable robot attitude and motion information is estimated, to suppress the inherent Gaussian white noise and drift error of the sensor.

[0078] 5. End.

[0079] The specific steps of feature enhancement processing of multi-modal raw data stream are:

[0080] 1. Start.

[0081] 2. Apply CLAHE algorithm adaptive histogram equalization to color image;

[0082] Wherein, the contrast limit is set to 2.0, and the block grid size is set to 8x8, to enhance the local contrast of the image and make the cracks more obvious under different lighting conditions;

[0083] At the same time, the gradient information of the image is extracted, and the edge features are extracted through a Canny operator, wherein a low threshold is set as 50 and a high threshold is set as 150, so as to highlight the boundaries of the cracks.

[0084] 3. A depth gradient calculation is applied to the depth image, and a Sobel operator or a Scharr operator is used to calculate the gradient of the depth image in the x and y directions to highlight the small surface concave or convex changes that usually indicate the presence of cracks;

[0085] 4. Based on the IMU data processed by Kalman filtering, the instantaneous linear velocity, angular velocity and linear acceleration vector of the robot are calculated, and the motion state parameters are input as the context of subsequent feature transformation, so as to provide the model with the motion information of the spraying robot.

[0086] 5. End.

[0087] As Figure 3 shown, in one specific embodiment, in the step of S2, the multi-modal feature transformation unit is used to extract and transform the multi-scale feature representation with time dynamic changes from the continuous multi-modal data frames, and solve the noise propagation and feature drift problems in multi-scale feature alignment, including: a parallel feature extraction network, a group of intra-modal temporal feature transformers and a cross-modal spatial attention fusion module.

[0088] In this embodiment, the parallel feature extraction network contains independent feature extraction paths for different modal data: one path processes the RGB color image sequence, and the other path processes the depth image sequence; each path uses a lightweight and efficient convolutional neural network as the backbone network.

[0089] For example, based on the MobileNetV3-Small architecture, a version optimized and pruned for specific tasks is used to extract rich features while ensuring computational efficiency; for the RGB image path, the input is a three-channel color image frame; for the depth image path, the input is a single-channel depth image frame; wherein the backbone network outputs a set of multi-scale initial feature maps, including 4 feature levels with decreasing resolution and increasing semantic information (for example, feature maps with resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 relative to the input resolution); for an input image of 1920x1080, the output feature map sizes are 480x270, 240x135, 120x67, and 60x33, respectively.

[0090] Specifically, the output feature maps capture low-level texture features containing edges and corner points, mid-level semantic features containing local structures of cracks, and high-level abstract features containing overall morphology and context of cracks, respectively; the parallel feature extraction network can generate a series of multi-scale feature maps with rich semantic information for each input frame, providing a basis for subsequent temporal transformation and modal fusion.

[0091] In this embodiment, the intra-modal temporal feature transformer group includes two parallel temporal transformer modules corresponding to the RGB color image modality and the depth image modality, respectively, and each temporal transformer module receives a sequence of continuous multi-scale feature maps of its corresponding modality from the parallel feature extraction network.

[0092] Further, for each RGB or depth modality, the input is a sequence of multi-scale feature maps with a length of T (e.g., T = 8 frames), and each feature map is first flattened into a series of feature vectors.

[0093] For example, for a feature map with a size of H x W x C, it will be flattened into H x W C-dimensional feature vectors, and in order to preserve the temporal information, each feature vector is also added with a learnable position encoding that can indicate the temporal position of the feature vector in the sequence; the core of the temporal transformer module is an encoder structure based on the Transformer architecture, which contains a multi-head self-attention mechanism and a feedforward neural network.

[0094] Specifically, the multi-head self-attention mechanism allows the model to simultaneously focus on the features at different positions in the sequence in the time dimension, thereby capturing the dynamic change pattern of crack features between consecutive frames; for example, when the spraying robot moves or the lighting conditions change, the visual appearance of the crack may appear a slight drift or local occlusion, and the self-attention mechanism can effectively suppress the influence of noisy frames or unstable frames (such as image blur frames caused by transient occlusion or violent motion) by calculating the query (Query), key (Key) and value (Value) matrices and assigning dynamic weights to the value matrix according to the similarity between the query and the key, and enhancing the crack features that appear continuously and stably in the time dimension; after processing by the temporal transformer module, each modality generates a set of multi-scale temporal feature maps that are more stable in the time dimension and have robustness to feature drift, which can effectively alleviate the problem of feature drift faced by the static FPN in the dynamic spraying process and ensure the continuity and consistency of the crack features.

[0095] As a preferred embodiment, in the extraction and transformation of multi-modal data, the sequence of spraying robot pose and motion information collected by the IMU can be further integrated into the input of the intra-modal temporal feature transformer group.

[0096] Specifically, the IMU data sequence including three-axis acceleration and three-axis angular velocity is first processed by a small one-dimensional convolutional neural network (1D CNN) containing three convolutional layers, each followed by a ReLU activation function and batch normalization, for extracting high-level temporal features thereof;

[0097] The extracted IMU temporal features are used as conditional encoding, which can be concatenated with the visual feature vector or fused through a gating mechanism for each visual feature vector in the intra-modal temporal transformer.

[0098] For example, in the gating mechanism, the IMU features first generate a gating vector through a small multi-layer perceptron, which is element-wise multiplied with the visual feature vector to dynamically adjust the influence of IMU information on the transformation of visual features. This conditional encoding can provide additional context information for the self-attention mechanism, such as the instantaneous speed, acceleration, and rotation direction of the robot, which helps the model better understand and compensate for the visual feature drift caused by the dynamic motion of the robot, especially in high-speed or non-uniform motion scenarios, improving the accuracy and stability of feature representation.

[0099] In this embodiment, the cross-modal spatial attention fusion module is used to fuse the color image modal multi-scale feature map set and the depth image modal multi-scale feature map set that have been optimized in the time dimension from the intra-modal temporal feature transformer group, through an intelligent, spatially aware attention mechanism to fuse features from different modalities to address the noise propagation problem when aligning multi-scale features in complex bridge environments.

[0100] Further, the cross-modal spatial attention fusion module performs the following operations for each feature pyramid level:

[0101] Bilinear interpolation or nearest neighbor interpolation is performed on the feature maps of the color image modal and the feature maps of the depth image modal to accurately align their spatial resolutions for pixel-level fusion.

[0102] A query, key, and value matrix is constructed; in a specific design, the processed RGB feature map is used as the query Q , the depth feature map is used as the key K , and the value V is the RGB feature map itself. The specific operation is as follows: the RGB modal query matrix Q rgb = FFN Q ( F rgb，aligned ), the depth modal key matrix K depth = FFN K ​F depth，aligned ), a depth modality value matrix V depth = FFN V ( F depth，aligned ), wherein the FFN represents a feed-forward network; F rgb，aligned represents an aligned RGB feature map; F depth，aligned represents an aligned depth feature map.

[0103] The cross-modality spatial attention mechanism is adopted to calculate the similarity between the query Q and the key K through a dot product operation, and the result is scaled and Softmax normalized to obtain an attention weight map; wherein the weight map then acts on the value V to generate a fused feature representation.

[0104] Exemplarily, the depth information can provide height information for structured noise such as welds, rust patches, and paint peeling on the surface of a bridge, while a real crack usually appears as a small depression or gap. When a certain area in the RGB feature map contains texture features similar to cracks, but the corresponding depth information shows that the area has no obvious depression, the depth modality attention weight will be given a lower value, thereby inhibiting the area from being misidentified as a crack in the RGB modality. Conversely, when both RGB and depth information strongly indicate the presence of a crack (e.g., the RGB texture shows a fine line structure, and the depth map presents a clear linear depression in the area), the attention weight will be amplified, achieving effective fusion and enhancement of the features of the two modalities.

[0105] It can be understood that the cross-modality spatial attention fusion module outputs a multi-scale feature pyramid that fuses RGB and depth information and is precisely aligned and noise-suppressed in the spatial dimension. The features of this pyramid have higher discriminability for real cracks.

[0106] As a preferred embodiment, in the cross-modality spatial attention fusion module, the attention mechanism can further adopt a learnable gating network to dynamically adjust the contribution weights of different modality features.

[0107] Specifically, the gating network receives feature maps (such as feature summary representations obtained by global average pooling) from each modality, and predicts a set of gating coefficients between 0 and 1 through a small multi-layer perceptron (MLP).

[0108] Exemplarily, for the RGB modality feature map F rgb，aligned and the depth modality feature map Fdepth，aligned , concatenated as the input vector of the gating network Then the concatenated features are processed by an MLP z Output two scalar weights for RGB modalities and depth modality in the fused feature g rgb and depth modality in the fused feature g depth And pass through a sigmoid activation function to map it to the range [0, 1], followed by element-wise or channel-wise multiplication operation with the feature maps of each modality to achieve the weighting of feature information.

[0109] For example, when the ambient light condition is good, the RGB texture information is rich and reliable, but the depth information may be limited or noisy due to uneven surface material reflectivity, the gating network can learn and increase the weight of g rgb , and reduce the weight of g depth ; while in the environment with complex light, serious shadow interference, but more reliable depth information, the weight of g depth is increased, and the gating network automatically learns how to allocate the optimal modality weight according to the input feature content and environmental conditions during the training process, further enhancing the adaptability of feature fusion.

[0110] In a specific embodiment, in the step of S3, the feature extraction and segmentation unit adopts a segmentation network based on an encoder-decoder architecture, preferably a DeepLabV3+ model; wherein the encoder part of the segmentation network is composed of or connected to the fusion feature pyramid, such as receiving features from each level of the pyramid through a skip connection.

[0111] In this embodiment, the encoder uses a pre-trained backbone network and integrates a dilated spatial pyramid pooling module to capture multi-scale context information; the decoder part receives the multi-scale fusion features output by the encoder, and gradually restores the spatial resolution of the feature map through upsampling, skip connection and convolution layer, finally outputs a pixel-level crack probability map or a binary segmentation mask.

[0112] Wherein the backbone network is selected from one of ResNet-101 or EfficientNet-B4; the pixel-level crack probability map, for example, each pixel value is between 0 and 1, representing the probability that the pixel belongs to the crack; the binary segmentation mask is converted from the probability map to a binary map by setting a threshold, such as 0.5, where 1 represents a crack and 0 represents a background.

[0113] It can be understood that the feature extraction and segmentation unit can accurately identify and segment the crack area on the surface of the bridge, and on this basis, through a series of morphological processing and geometric analysis algorithms such as opening operation, closing operation, thinning algorithm, the morphological parameters of the crack are accurately extracted; wherein the morphological parameters include but are not limited to the length, average width, maximum width, direction, connectivity, area of the crack and its position coordinates in three-dimensional space.

[0114] Specifically, the length of the crack is extracted by applying a skeletonization algorithm to the binary segmentation mask to extract the crack center line, then calculating the pixel length, and converting it into actual physical length combined with the camera calibration parameters; the width of the crack is determined by calculating the average or maximum Euclidean distance from the crack skeleton line to the crack edge, and is also converted into physical units; the direction of the crack is determined by performing principal component analysis on the crack pixel set, or by calculating the tangent direction of each point on the crack skeleton line to determine its main direction; the connectivity of the crack is determined by analyzing its topological structure to distinguish single cracks, branched cracks or crack networks.

[0115] Wherein, the skeletonization algorithm is preferably Zhang-Suen thinning algorithm; the camera calibration parameters include focal length and pixel size.

[0116] Further, the feature extraction and segmentation unit can also integrate depth information to reconstruct the crack in three-dimensional geometry. Specifically, by matching the segmented crack pixels obtained from the binary segmentation mask with the corresponding depth map data, the point cloud representation of the crack in three-dimensional space can be obtained. Based on this point cloud data, the actual three-dimensional shape of the crack, the depth profile information including the average depth and maximum depth of the crack, and the surface roughness can be further calculated.

[0117] For example, by locally fitting the crack point cloud, the roughness of the surface around the crack can be quantified, or by calculating the average depth difference between the points inside the crack region and the points in the surrounding non-crack region, the depth information of the crack can be obtained. The accurate three-dimensional geometric feature data of the crack provides more accurate three-dimensional spatial positioning and geometric matching data for subsequent adaptive spraying parameter control, so that the spraying operation can truly realize accurate repair of crack defects.

[0118] In a specific embodiment, in the step of S4, the adaptive spraying parameter control unit receives the crack morphological parameters output by the multi-modal feature transformation unit and the current pose information of the spraying robot.

[0119] Wherein, the crack morphological parameters include but are not limited to length, width, direction, connectivity, three-dimensional position and depth information; the current pose information of the spraying robot is obtained by fusing IMU data and robot body encoder data.

[0120] Specifically, the adaptive spraying parameter control unit adopts a reinforcement learning paradigm to dynamically optimize the spraying parameters and motion trajectory of the spraying robot, and the adaptive spraying parameter control unit comprises a reinforcement learning agent, an environment model, and a reward function.

[0121] In this embodiment, the reinforcement learning agent is a policy network and a value network based on a deep neural network, and preferably adopts a Soft Actor-Critic (SAC) algorithm architecture; the policy network is a multi-layer perceptron (MLP) that receives the current state as input and outputs continuous spraying actions, i.e., spraying parameters and local trajectory adjustment instructions including spraying pressure, coating flow rate, spraying speed, spraying gun posture, and spraying gun-to-surface distance, and the output layer uses a tanh activation function to map the action value to a predefined action space range; the value network is two Q networks for double Q learning in the SAC algorithm, and like the policy network, is an MLP that receives the current state and action as input and evaluates the expected return (Q value) of executing the specific action in this state to guide the learning direction of the policy network.

[0122] In this embodiment, the input of the environment model includes the geometric features of the currently detected crack, the current pose of the spraying robot, the distance and angle between the spraying gun and the target surface, the current ambient light intensity, the target surface material properties, and the quality evaluation results of the previous spraying attempt; the environment model is a simulator based on physical simulation and empirical data, used to simulate the physical characteristics in the spraying process, including coating fluid mechanics, deposition efficiency, drying time, coating adhesion, and evaluation of the spraying effect by the visual feedback system; the environment model is used to update the environment state according to the action of the agent and calculate the reward.

[0123] wherein the geometric features include, for example, length 200 mm, average width 2 mm, maximum depth 1.5 mm, three-dimensional position coordinates, principal direction, curvature, etc.; the current pose of the spraying robot includes, for example, position [X, Y, Z] and orientation [Roll, Pitch, Yaw]; the material properties include, for example, concrete, steel, water absorption, surface roughness; the coating fluid mechanics include, for example, coating atomization effect, spraying mode, droplet size distribution; and the deposition efficiency includes, for example, coating thickness and uniformity.

[0124] In this embodiment, the reward function is used to quantify the quality and efficiency of the spraying operation and guide the learning direction of the reinforcement learning agent, and the reward function is designed as follows: when the spraying operation can achieve complete coverage of the crack area, uniform coating thickness, good adhesion, and minimal material waste, the reward value reaches a maximum.

[0125] Specifically, the reward value is calculated by multiple indicators, including the difference between actual spraying coverage and target coverage, material consumption, path execution deviation, overspraying area beyond the crack area, and coating adhesion score; each indicator is provided with a corresponding weight coefficient, and optionally, the coverage weight is 5.0, the material waste weight is 2.0, the path deviation weight is 1.0, the overspraying weight is 3.0, and the adhesion weight is 4.0, which are set according to expert experience and preliminary experiments to reasonably balance the importance between different optimization objectives.

[0126] Among them, the target coverage is generally set to 100%, and the actual coverage is obtained by comparing the images before and after spraying or analyzing the depth information through the visual feedback system. The material consumption is in milliliters, the path deviation refers to the average offset distance between the actual path and the ideal path, the overspraying area refers to the area that is sprayed to the non-target area, and the adhesion score is a comprehensive score between 0 and 1, which is calculated by indirect indicators such as coating thickness, uniformity and surface roughness.

[0127] Further, through continuous optimization of the reward function, the reinforcement learning agent can gradually master an efficient spraying strategy, and adaptively adjust the spraying parameters according to the real-time detected crack morphology, robot state and environmental conditions, including spraying pressure 0.1-0.6 MPa, flow 10-50 ml / min, moving speed 50-200 mm / s, spraying gun attitude and distance between spraying gun and surface 100-300 mm, to realize precise, efficient and high-quality crack repair. The adaptive spraying parameter control unit can dynamically adjust the spraying strategy according to the real-time detected crack characteristics, robot motion state and environmental changes, and visual feedback after spraying, so as to overcome the limitations of static preset of spraying parameters in traditional methods and realize adaptive spraying.

[0128] Among them, the spraying gun attitude includes pitch, yaw and roll angles within ±45°.

[0129] As a preferred embodiment, the reinforcement learning agent of the adaptive spraying parameter control unit can be initialized in combination with expert experience or traditional PID control strategy at the initial stage of training, so as to accelerate convergence and provide a stable initial spraying strategy.

[0130] Exemplarily, a set of lookup table rules based on crack width and depth is preset, combined with a PID controller to preliminarily determine the spraying flow, pressure and speed; the reinforcement learning agent can be trained offline in a highly simulated bridge surface and spraying environment; the simulation environment is built using Unity3D or Gazebo physical engine, accurately simulating the geometric characteristics of the bridge surface, the crack morphology, the rheological behavior of the paint, the working characteristics of the spraying equipment, and the imaging process of the visual sensor, to ensure the effectiveness of the offline training strategy; after obtaining the preliminary strategy model through offline training, a small amount of actual spraying test is performed for online fine-tuning and deployment, and the strategy is further optimized through the interaction data with the real environment to adapt to unpredictable factors in actual spraying operation.

[0131] wherein the crack morphology includes length, width, depth and curvature; the rheological behavior of the paint includes viscosity and surface tension; the working characteristics include spray cone angle and spray force; and the imaging process includes illumination, shadow and reflection.

[0132] Further, the optimized spraying parameters and local trajectory adjustment instructions output by the adaptive spraying parameter control unit are sent to the spraying robot actuator via the spraying robot motion control system, to drive the spraying gun to perform the repair spraying operation.

[0133] wherein the spraying robot actuator includes a multi-degree-of-freedom mechanical arm and a spraying gun integrated with a nozzle. The mechanical arm is a six-axis industrial robot, such as a KUKA KR CYBERTECH nano series mechanical arm, which has the characteristics of high precision and high repeatability, and can accurately execute the spraying trajectory. The spraying gun is an air atomizing spraying gun, whose spraying pressure can be accurately adjusted within the range of 0.1-0.6 MPa by a proportional valve, and the paint flow can be accurately adjusted within the range of 10-50 ml / min by a peristaltic pump or a metering pump, and the nozzle outlet diameter is 0.5-1.5 mm, which can be replaced or dynamically adjusted according to the width and depth of the crack, to achieve the best coating coverage effect and spraying precision; for example, for micro-cracks with a width less than 1 mm, a 0.5 mm diameter nozzle can be selected for low pressure and low flow fine spraying; for structural cracks with a width exceeding 3 mm, a 1.2 mm or 1.5 mm diameter nozzle is selected, combined with higher pressure and flow, to ensure sufficient filling and coverage.

[0134] In one specific embodiment, in the step of S5, during the spraying operation, the spraying robot utilizes the multi-modal perception unit for real-time visual feedback and quality verification. The visual feedback is configured to collect and analyze images of the sprayed area again after the spraying is completed or during the spraying process. By comparing the image data before and after spraying, or using depth information to evaluate the coating thickness and uniformity, the spraying quality can be evaluated in real time, such as coverage completeness, coating uniformity, presence of sagging, orange peel, or insufficient thickness, etc.

[0135] Wherein, the comparison of image data before and after spraying uses image difference or feature matching technology; the evaluation of coating thickness and uniformity is through comparing the depth value changes of sprayed area and non-sprayed area.

[0136] Specifically, if the coverage rate is less than 95% or the coating thickness uniformity standard deviation is greater than 0.1 mm, i.e., the preset standard is not met, the quality evaluation result will be fed back to the reinforcement learning agent of the adaptive spraying parameter control unit as new state information. The reinforcement learning agent will adjust the strategy based on this feedback, generate a re-spraying instruction or correct the current spraying parameters until the spraying quality meets the preset requirements. Through the closed-loop feedback mechanism, the reliability of the spraying operation is ensured, and the need for manual intervention and rework is minimized.

[0137] Experimental group: This experimental group is used to verify the performance of the spraying robot crack identification and adaptive spraying method based on machine vision in Example 1 in the actual bridge crack repair.

[0138] The application scenario is an experimental platform simulating a concrete bridge, and the spraying material is a polymer modified mortar with a maximum aggregate particle size ≤4.75 mm and an initial flow value ≥290 mm, purchased from Sichuan Jin Hongyuan New Materials; The experimental platform is pre-fabricated with various forms of cracks; The characteristics of a typical crack are shown in Table 1.

[0139] Table 1: Characteristics of a typical crack

[0140]

[0141] Experimental steps:

[0142] 1. The spraying robot carries the multi-modal perception unit along the preset path to scan the crack area.

[0143] 2. The multi-modal data stream is processed in real time and enters the multi-modal feature transformation unit to generate a fusion feature pyramid.

[0144] 3. The feature extraction and segmentation unit performs pixel-level segmentation on the fusion feature pyramid and extracts the morphological parameters and three-dimensional geometric information of the crack.

[0145] 4、Crack parameters and spraying robot pose are input to the adaptive spraying parameter control unit, and the reinforcement learning agent outputs the optimized spraying parameters (spraying pressure 0.35 MPa, coating flow 35 ml / min, spraying speed 120 mm / s, spraying gun pitch angle 5°, distance from surface 180 mm) and trajectory adjustment instructions based on the current state.

[0146] 5、The spraying robot actuator performs the spraying operation, and after the spraying is completed, the multi-modal perception unit collects the sprayed area image again, performs quality assessment, and the evaluation result is fed back to the reinforcement learning agent. If it does not meet the standard, it will be re-sprayed or the parameters will be fine-tuned.

[0147] Control group: This control group is used to show the performance of the traditional feature pyramid network (FPN) based spraying robot crack recognition and fixed parameter spraying method in the same experimental scene.

[0148] Method description: This control group uses a crack segmentation network based on FPN (PANet), which only uses RGB color images as input and does not contain time series feature transformation and cross-modal attention fusion mechanism. After crack segmentation, the spraying parameters are preset according to the empirical rule (based on crack average width and depth lookup table determination), which does not have dynamic adaptive ability and no reinforcement learning feedback closed loop.

[0149] Hardware configuration: The same as the hardware configuration in the experimental group, but the depth camera and IMU data are not used for feature fusion in the crack identification stage, only for auxiliary robot positioning. The spraying gun parameters are set to fixed values, spraying pressure 0.4 MPa, coating flow 40 ml / min, spraying speed 150 mm / s, spraying gun pitch angle 0°, distance from surface 200 mm.

[0150] The experimental results of the above experimental group and control group are shown in Table 2.

[0151] Table 2: Experimental results of the experimental group and the control group

[0152]

[0153] As shown in Table 2:

[0154] From the above comparison data, the method proposed in this paper has achieved significant improvement in crack detection F1-score, reaching 0.965, compared with 0.882 of the control group, an increase of 9.41%; This is mainly due to the rich information provided by the multi-modal perception unit, the robustness of the dynamic time series multi-modal feature transformation unit to feature drift, and the effective suppression of noise by the cross-modal spatial attention fusion module, especially in complex lighting and background interference, it can more accurately identify real cracks.

[0155] In terms of spraying performance, the spraying coverage of the proposed method is as high as 98.7%, which is much higher than 85.3% of the control group, and is increased by 15.71%; and then directly reflects the excellent ability of the reinforcement learning agent in the adaptive spraying parameter control unit to dynamically adjust the spraying parameters according to the crack morphology and the robot state, in terms of coating uniformity, the standard deviation of the coating thickness of the proposed method is only 0.08 mm, which is reduced by 61.90% compared with 0.21 mm of the control group, indicating that the proposed method can realize more fine and uniform coating deposition, at the same time, the material utilization rate is increased from 75.8% to 92.1%, reducing 21.50% of material waste, showing the economic benefit of the present application.

[0156] Although the spraying time of the proposed method is slightly longer than that of the control group, the time cost is increased by about 25%, mainly because the reinforcement learning agent needs to perform more iterations of optimization in the decision-making process, and may introduce slight trajectory adjustment and more fine parameter changes in the spraying process; however, considering the comprehensive requirements of crack repair tasks on quality and efficiency, the proposed method significantly improves key quality indicators such as crack recognition accuracy, spraying coverage, coating uniformity and material utilization rate, while slightly increasing the time cost, which is completely acceptable, and from the long-term maintenance and repair effect, the comprehensive benefits it brings far exceed the time investment.

[0157] Embodiment 2: Figure 4 An electronic device structure schematic diagram that can be used to implement the embodiment 1 of the present application is shown. The electronic device 10 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0158] As shown in Figure 4 The electronic device 10 includes at least one processor 11, and a memory, such as a Read-Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., which is in communication connection with the at least one processor 11, wherein the memory stores a computer program which can be executed by the at least one processor 11, and the computer program is executed by the at least one processor 11 to enable the at least one processor 11 to execute the method provided by the present application.

[0159] Further, the processor 11 can perform various appropriate actions and processes in accordance with a computer program stored in a read only memory (ROM) 12 or a computer program loaded from the storage unit 18 into a random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0160] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, and the like; an output unit 17, such as various types of displays, a speaker, and the like; a storage unit 18, such as a magnetic disk, an optical disk, and the like; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0161] Still further, the processor 11 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processing (DSP), and any appropriate processor, controller, microcontroller, and the like. The processor 11 performs various methods and processes described above, such as the method of resource management for a database.

[0162] In some specific embodiments, the method of resource management for a database can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method of resource management for a database described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the method of resource management for a database by any other appropriate means, such as by means of firmware.

[0163] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a System on Chip (SOC), a Complex Programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0164] Computer programs used to implement the present methods can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flow diagrams and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as part of a standalone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0165] In the context of the present application, the computer readable storage medium stores computer instructions for causing a processor to implement the method of resource management of a database provided by the present application when executed. The computer readable storage medium can be a tangible medium which can contain or store the computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium can be a machine readable signal medium. More specific examples of the machine readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0166] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a Cathode Ray Tube (CRT) or a Liquid Crystal Display (LCD) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0167] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0168] Optionally, the computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS) services.

[0169] The above is according to the ideal embodiment of the present application, through the above description, for those skilled in the art, it is obvious that the present application is not limited to the details of the above exemplary embodiments, and can be realized in other specific forms without departing from the spirit or basic characteristics of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, the scope of the present application is defined by the appended claims rather than the above description, therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be regarded as limiting the claims involved.

[0170] In addition, it should be understood that, although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be combined appropriately to form other embodiments that those skilled in the art can understand.

Claims

1. A machine vision-based method for crack recognition and adaptive spraying in a spraying robot, characterized in that, Includes the following steps: S1. Real-time acquisition of multimodal raw data streams of the surface of the bridge structure to be inspected, and generation of multimodal data after spatiotemporal alignment, noise suppression, and feature enhancement; the multimodal raw data streams include high-resolution color images, high-precision depth images, and posture and motion information of the spraying robot; S2. From multimodal data, a multimodal feature transformation unit is used to extract and transform a fusion feature pyramid that has multiple scales for dynamic changes over time; the multimodal feature transformation unit includes a parallel feature extraction network, an intramodal temporal feature transformer group, and a cross-modal spatial attention fusion module; S3. For the fused feature pyramid, feature extraction and segmentation units are used to segment it, identify and extract the morphological parameters of the crack region and its position coordinates in three-dimensional space; S4. Based on the shape parameters, position coordinates and the current posture information of the painting robot, output optimized painting parameters and local trajectory adjustment instructions to perform repair painting operations. In step S2, the parallel feature extraction network includes independent feature extraction paths for color images and depth images. Each path uses a lightweight convolutional neural network as the backbone network to extract an initial feature map set of multiple scales from a single frame image. The intramodal temporal feature transformer group includes at least two parallel temporal transformer modules, corresponding to the color image modality and the depth image modality, respectively. Each temporal transformer module receives a multi-frame, multi-scale feature map sequence from the parallel feature extraction network for its corresponding modality, and captures the dynamic change pattern of crack features in the temporal dimension through an encoder structure based on the Transformer architecture, including a multi-head self-attention mechanism and a feedforward neural network, to generate a multi-scale temporal feature map set that is stable in the temporal dimension and robust to feature drift. In the multimodal feature transformation unit, the posture and motion information of the painting robot are further integrated into the input of the intramodal temporal feature transformer group; Specifically, the posture and motion information of the painting robot is processed by a one-dimensional convolutional neural network to extract its temporal features. These temporal features are then used as conditional codes and concatenated with or fused with feature vectors from the color image modality and depth image modality through a gating mechanism. The combined inputs are then fed into the temporal converter module to provide additional contextual information. The cross-modal spatial attention fusion module is used to receive color image modal multi-scale feature maps and depth image modal multi-scale feature maps from the intra-modal temporal feature transformer group, and for each feature pyramid level, perform the following operations: Upsampling or downsampling operations are performed on the feature maps of the color image modality and the feature maps of the depth image modality to align their spatial resolutions. We construct query, key, and value matrices, and employ a cross-modal attention mechanism to assign attention weights to the value matrix by calculating the similarity between queries and keys. This enables feature fusion from different modalities and suppresses the influence of structured noise through deep information. The cross-modal spatial attention fusion module outputs a multi-scale feature pyramid that integrates color and depth information and undergoes precise alignment and noise suppression in the spatial dimension.

2. The method for crack recognition and adaptive spraying of a spraying robot based on machine vision according to claim 1, characterized in that: In step S1, the spatiotemporal alignment of the multimodal raw data stream includes: Acquire each set of color and depth images at the same time; Using pre-calibrated extrinsic parameters of each sensor, the depth image and color image are registered to the same coordinate system, generating pixel-aligned RGB-D image pairs; and The posture and motion information data of the painting robot and the RGB-D image pairs are timestamped to generate data frames with a unified time sequence.

3. The method for crack recognition and adaptive spraying of a spraying robot based on machine vision according to claim 1, characterized in that: In step S1, noise suppression of the multimodal raw data stream includes: An image denoising algorithm based on nonlocal mean filtering is applied to color images; Denoising algorithms based on bilateral filtering or median filtering are applied to depth images; and Kalman filtering or complementary filtering is applied to the posture and motion information data of the painting robot.

4. The method for crack recognition and adaptive spraying of a spraying robot based on machine vision according to claim 1, characterized in that: In step S1, the feature enhancement of the multimodal raw data stream includes: Adaptive histogram equalization is applied to color images to enhance image contrast and extract gradient information from the images; Apply depth gradient calculations to depth images to highlight surface height variations; and The instantaneous velocity and acceleration vectors of the painting robot are calculated based on its posture and motion information data, and used as context input for subsequent feature transformation.

5. The method for crack recognition and adaptive spraying of a spraying robot based on machine vision according to claim 1, characterized in that: In step S3, the feature extraction and segmentation unit adopts a segmentation network based on an encoder-decoder architecture. The encoder part of the segmentation network is composed of or connected to the fusion feature pyramid. The decoder part receives the multi-scale fusion features output by the encoder and gradually restores the spatial resolution through upsampling, skip connections and convolutional layers, and finally outputs a pixel-level crack probability map or a binary segmentation mask. The morphological parameters of the crack are accurately extracted through morphological processing and geometric analysis algorithms. These morphological parameters include the crack's length, average width, maximum width, direction, connectivity, area, and position coordinates in three-dimensional space. By matching the segmented crack pixels with the corresponding depth image data, the point cloud representation of the crack in three-dimensional space is obtained, and based on this point cloud data, the actual three-dimensional shape, depth profile information and surface roughness and other advanced geometric features of the crack are calculated.

6. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to at least one of the processors; The memory stores a computer program executed by at least one of the processors, which enables the at least one processor to perform the machine vision-based crack recognition and adaptive spraying method for spraying robots according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the machine vision-based crack recognition and adaptive spraying method for a painting robot as described in any one of claims 1-5.

Images