Multi-source data fusion visualization system for underwater intelligent maintenance robots

Abstract

The present application discloses a multi-source data fusion visualization system for an underwater intelligent maintenance robot, which relates to the field of visualization technology. The system performs multi-source image analysis on underwater visible light images and underwater sonar detection images by using deep learning-based image processing technology, extracts underwater visible light features and underwater sonar detection features respectively, performs feature gating enhancement processing on both of them respectively, and then uses a dual attention mechanism to perform fine-grained joint perception of underwater visible light features and sonar detection features to obtain multi-source data fusion features, and uses this to reconstruct a visualized image and generate an underwater detection multi-source fusion image, which can make full use of the advantages of multi-source data, effectively improve image quality, and thus improve the accuracy and efficiency of underwater maintenance.

Description

Technical Field

[0001] The present application relates to the field of visualization technology, and more specifically, to a multi-source data fusion visualization system for an underwater intelligent maintenance robot. Background Art

[0002] With the introduction of the modern concept of "digital intelligence technology" to assist ecological and environmental governance, the industries of digitalization, intelligence and sewage treatment equipment have begun to merge and upgrade, realizing cross-field integration. In the field of sewage treatment, the use of intelligent underwater robot technology has brought opportunities for improving the quality and efficiency of sewage treatment operations. Intelligent underwater robots can not only intelligently control the depth and heading of underwater navigation, but also carry various sensors, such as visible light cameras, magnetometers, sonar imaging, laser radar, etc., so that underwater intelligent robots can operate in sewage environments and perform data image transmission, high-precision visual dimensional variable measurement, sewage sludge shallow profiling measurement, etc., which greatly improves the intelligence level of sewage treatment.

[0003] For example, a Chinese patent with application number 201911076410.3 discloses a target detection method, system, and device based on the vision of an underwater working robot, including obtaining a binocular visual image; extracting image features through a densely connected network, and extracting target images in the left image and the right image as detection results through a feature pyramid network; matching the target images in the detection results through position matching to obtain one or more corresponding target image pairs; based on the refractive index of water in the water area and the refractive index of the transparent waterproof layer of the binocular camera, obtaining the three-dimensional coordinates of the center point of the corresponding target of each target image pair in the binocular camera coordinate system through an optical path tracing method, thereby improving the accuracy of target detection and positioning.

[0004] However, in sewage environments, due to low visibility and high impurity content of sewage, underwater light attenuation is fast and field of view is limited. Relying on a single visible light imaging camera or binocular camera and other sensors, the characteristics of the underwater image data obtained are relatively simple. In addition, existing technologies mostly use a single image processing algorithm or a simple overlay method to try to improve image quality, which cannot fully explore the potential connections between multi-source data, often resulting in poor fusion effects between multi-source data and failure to fully utilize the advantages of multi-source data.

[0005] Therefore, an optimized multi-source data fusion visualization system for underwater intelligent maintenance robots is needed to solve the above technical problems. Summary of the invention

[0006] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a multi-source data fusion visualization system for an underwater intelligent maintenance robot, which performs multi-source image analysis on underwater visible light images and underwater sonar detection images by using deep learning-based image processing technology, extracts underwater visible light features and underwater sonar detection features respectively, performs feature gating enhancement processing on both, and uses a dual attention mechanism to perform fine-grained joint perception of underwater visible light features and sonar detection features to obtain multi-source data fusion features, and uses this to reconstruct visual images and generate underwater detection multi-source fusion images. In this way, the advantages of multi-source data can be fully utilized, the image quality can be effectively improved, and thus the accuracy and efficiency of underwater maintenance can be improved.

[0007] According to one aspect of the present application, a multi-source data fusion visualization system for an underwater intelligent maintenance robot is provided, comprising:

[0008] An underwater image acquisition module is used to acquire underwater visible light images and underwater sonar detection images collected by a visible light camera and a sonar detector of an underwater robot;

[0009] An underwater image feature extraction module is used to extract image features from the underwater visible light image and the underwater sonar detection image to obtain an underwater visible light feature map and an underwater sonar detection feature map respectively;

[0010] A feature gating enhancement module is used to perform feature gating enhancement processing on the underwater visible light feature map and the underwater sonar detection feature map to obtain a grid-granularity underwater visible light enhancement feature map and a grid-granularity underwater sonar detection enhancement feature map;

[0011] A fine-grained joint perception module is used to perform fine-grained joint perception on the grid-grained underwater visible light enhancement feature map and the grid-grained underwater sonar detection enhancement feature map to obtain an underwater visible light-sonar multi-source detection fusion feature map;

[0012] The multi-source data fusion visualization module is used to generate an underwater detection multi-source fusion image based on the underwater visible light-sonar multi-source detection fusion feature map.

[0013] The present application has at least the following technical effects: Compared with the prior art, the multi-source data fusion visualization system of an underwater intelligent maintenance robot provided by the present application performs multi-source image analysis on underwater visible light images and underwater sonar detection images by using image processing technology based on deep learning, extracts underwater visible light features and underwater sonar detection features respectively, performs feature gating enhancement processing on both respectively, and uses a dual attention mechanism to perform fine-grained joint perception of underwater visible light features and sonar detection features to obtain multi-source data fusion features, and uses this to reconstruct visual images and generate underwater detection multi-source fusion images. In this way, the advantages of multi-source data can be fully utilized to effectively improve image quality, thereby improving the accuracy and efficiency of underwater maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0015] Figure 1 A block diagram of a multi-source data fusion visualization system for an underwater intelligent maintenance robot according to an embodiment of the present application;

[0016] Figure 2 This is a data flow diagram of a multi-source data fusion visualization system of an underwater intelligent maintenance robot according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0018] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0019] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0020] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0021] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0022] Specifically, underwater intelligent inspection robots are generally equipped with visible light imaging cameras and sonar imaging sensors to obtain underwater image data; underwater visible light images and sonar images have significant differences in imaging principles, information content, resolution, etc., and existing technologies often use a single image processing algorithm or a simple superposition method to try to improve image quality, which cannot fully explore the potential connections between multi-source data, often resulting in poor fusion effects between the two and failure to fully utilize the advantages of multi-source data. Therefore, an optimized multi-source data fusion visualization system for underwater intelligent maintenance robots is needed to solve the above technical problems.

[0023] In the technical solution of the present application, a multi-source data fusion visualization system for an underwater intelligent maintenance robot is proposed. Figure 1 It is a block diagram of a multi-source data fusion visualization system of an underwater intelligent maintenance robot according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of a multi-source data fusion visualization system of an underwater intelligent maintenance robot according to an embodiment of the present application. Figure 1 and Figure 2As shown, according to the multi-source data fusion visualization system 300 of the underwater intelligent maintenance robot of the embodiment of the present application, it includes: an underwater image acquisition module 310, which is used to acquire the underwater visible light image and the underwater sonar detection image collected by the visible light camera and the sonar detector of the underwater robot; an underwater image feature extraction module 320, which is used to extract image features from the underwater visible light image and the underwater sonar detection image respectively to obtain an underwater visible light feature map and an underwater sonar detection feature map; a feature gating enhancement module 330, which is used to extract the underwater visible light feature map and the underwater sonar detection feature map; The image and the underwater sonar detection feature map are subjected to feature gating enhancement processing to obtain a grid-granularity underwater visible light enhancement feature map and a grid-granularity underwater sonar detection enhancement feature map; a fine-grained joint perception module 340 is used to perform fine-grained joint perception on the grid-granularity underwater visible light enhancement feature map and the grid-granularity underwater sonar detection enhancement feature map to obtain an underwater visible light-sonar multi-source detection fusion feature map; a multi-source data fusion visualization module 350 is used to generate an underwater detection multi-source fusion image based on the underwater visible light-sonar multi-source detection fusion feature map.

[0024] In particular, the underwater image acquisition module 310 is used to acquire underwater visible light images and underwater sonar detection images collected by the visible light camera and sonar detector of the underwater robot. Specifically, the visible light camera can capture visual information such as color, texture and shape of the underwater environment, and is particularly suitable for areas with good lighting conditions, but is limited by factors such as turbidity of the water body, light scattering and absorption. On the contrary, the sonar detector detects the shape, size and position of underwater objects by emitting sound waves and receiving echoes. It is not sensitive to light and can penetrate turbid water bodies, but cannot provide intuitive visual information such as color and texture. Therefore, the two types of image data collected by the visible light camera and the sonar detector are naturally complementary. By combining the two types of data, more comprehensive, accurate and reliable underwater environment information can be provided for the underwater intelligent maintenance robot, which is helpful to realize the automation and intelligent maintenance and maintenance of underwater facilities or equipment, and improve work efficiency and safety.

[0025] In particular, the underwater image feature extraction module 320 is used to extract image features from the underwater visible light image and the underwater sonar detection image, respectively, to obtain an underwater visible light feature map and an underwater sonar detection feature map. In a specific example of the present application, the underwater visible light image and the underwater sonar detection image are respectively input into an image feature extractor based on a hole convolutional neural network model to obtain the underwater visible light feature map and the underwater sonar detection feature map. In order to fully extract image features, the present application uses a hole convolutional neural network model with excellent performance in image processing tasks as an image feature extractor to process the underwater visible light image and the underwater sonar detection image, respectively. As a variant of CNN, the dilated convolution increases the receptive field by inserting "holes" into the standard convolution kernel, and can capture a wider range of image context information while keeping the number of parameters unchanged, so that it can effectively capture the rich color distribution, texture changes and other visual information in the underwater visible light image, as well as the precise object shape, size and other information in the underwater sonar detection image, thereby generating underwater visible light feature maps and underwater sonar detection feature maps. It is worth mentioning that the dilated convolutional neural network is a convolutional neural network (CNN) that uses dilated convolution operations to expand the receptive field while maintaining spatial resolution.

[0026] In particular, the feature gating enhancement module 330 is used to perform feature gating enhancement processing on the underwater visible light feature map and the underwater sonar detection feature map to obtain a grid-sized underwater visible light enhancement feature map and a grid-sized underwater sonar detection enhancement feature map. In a specific example of the present application, the underwater visible light feature map and the underwater sonar detection feature map are input into a feature gating enhancement module based on grid energy significance guidance to obtain the grid-sized underwater visible light enhancement feature map and the grid-sized underwater sonar detection enhancement feature map. Considering that the underwater visible light image and the underwater sonar detection image must contain some background information that is irrelevant to the underwater maintenance task, in order to improve the quality of the feature map, the present application introduces a feature gating enhancement module based on grid energy significance guidance to process the underwater visible light feature map and the underwater sonar detection feature map, and dynamically weights the feature map by learning the significant feature areas in the feature map, thereby highlighting important features and suppressing irrelevant information, thereby enhancing the expressiveness of the features.

[0027] In an embodiment of the present application, the underwater visible light feature map and the underwater sonar detection feature map are input into a feature gating enhancement module based on grid energy significance guidance to obtain the grid-granularity underwater visible light enhancement feature map and the grid-granularity underwater sonar detection enhancement feature map, including: first, gridding the underwater visible light feature map to obtain a set of underwater visible light local feature maps; that is, using the grid to standardize the attention window, and decomposing the input feature map into multiple local feature maps by gridding; then, calculating the energy significance description factor of each underwater visible light local feature map in the set of underwater visible light local feature maps to obtain a set of underwater visible light local feature energy significance description factors; wherein, the energy significance description factor of each local feature map is calculated using the statistical characteristics of each local feature map to reveal the significance or importance of each local feature map in the global feature map, which is helpful to identify the key area in the feature map; then, the set of underwater visible light local feature energy significance description factors is input into a local feature adaptive selector based on a gating function to obtain an underwater visible light local feature map. A set of significant modulation weights; further, using each underwater visible light local feature significant modulation weight in the set of underwater visible light local feature significant modulation weights as a weight, weighting each underwater visible light local feature map in the set of underwater visible light local feature maps to obtain a set of enhanced underwater visible light local feature maps; adaptively selecting the energy significance description factor of each local feature map through a gating function, and converting each energy significance description factor into a corresponding weight through normalization and masking, so as to weight each local feature map, thereby strengthening the local features with high significance and weakening the features with low significance, so that the model can more effectively utilize the key information in the feature map and improve the discrimination and expression ability of the feature representation; further, the set of enhanced underwater visible light local feature maps is feature aggregated according to the grid division method to obtain a local significant guided enhanced underwater visible light feature map; then the local significant guided enhanced underwater visible light feature map is input into a feature uniform dispersion module based on a hole convolution layer to obtain the grid-granularity underwater visible light enhanced feature map. Here, considering that the attention granularity of this module is the grid granularity, there will be feature transition gaps at the grid boundary. To this end, the weighted enhanced local feature map is further re-aggregated in the grid division manner, and the aggregated feature map is subjected to a dilated convolution operation to disperse the features, so as to achieve smooth transition of features and fusion of global information, and finally generate a grid-scale underwater visible light enhancement feature map and a grid-scale underwater sonar detection enhancement feature map.

[0028] Among them, the process of calculating the energy significance description factor of each underwater visible light local feature map in the set of underwater visible light local feature maps to obtain a set of underwater visible light local feature energy significance description factors includes: calculating the difference between the maximum eigenvalue and the minimum eigenvalue of the underwater visible light local feature map, and dividing the difference by the weighted sum of the feature mean, feature variance and bias term of the underwater visible light local feature map to obtain the underwater visible light local feature energy significance description factor.

[0029] More specifically, the process of inputting the set of underwater visible light local feature energy significance description factors into a local feature adaptive selector based on a gating function to obtain a set of underwater visible light local feature significant modulation weights includes: inputting the set of underwater visible light local feature energy significance description factors into a Sigmoid function for normalization processing to obtain a set of normalized underwater visible light local feature energy significance description factors; based on a preset threshold, masking the set of normalized underwater visible light local feature energy significance description factors to obtain a set of underwater visible light local feature significant modulation weights.

[0030] In summary, in the above embodiment, the underwater visible light feature map and the underwater sonar detection feature map are input into a feature gating enhancement module based on grid energy significance guidance to obtain the grid-granularity underwater visible light enhancement feature map and the grid-granularity underwater sonar detection enhancement feature map, including: processing the underwater visible light feature map with the following feature gating enhancement formula to obtain the grid-granularity underwater visible light enhancement feature map, wherein the feature gating enhancement formula is:

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037] in, represents the underwater visible light characteristic graph, represents the meshing process, , , and respectively represent the first, second, and third in the set of underwater visible light local feature maps and A local feature map of underwater visible light, The value of is the number of feature maps in the set of underwater visible light local feature maps, , , and Respectively represent the The maximum eigenvalue, minimum eigenvalue, feature mean and feature variance of the underwater visible light local feature map, represents the bias term, Indicates the The energy significance description factor of the underwater visible light local feature map, Indicates mask processing, express The function's variables, is the preset threshold, Indicates The significant modulation weight of the underwater visible light local feature, represents feature aggregation processing, represents the local significant guided enhancement of underwater visible light feature map, represents the dilated convolution operation, The grid granularity underwater visible light enhancement feature map is represented.

[0038] In particular, the fine-grained joint perception module 340 is used to perform fine-grained joint perception on the grid-grained underwater visible light enhancement feature map and the grid-grained underwater sonar detection enhancement feature map to obtain an underwater visible light-sonar multi-source detection fusion feature map. In a specific example of the present application, the grid-grained underwater visible light enhancement feature map and the grid-grained underwater sonar detection enhancement feature map are input into a two-way global attention joint perception module based on fine-grained deconstruction to obtain the underwater visible light-sonar multi-source detection fusion feature map. In order to effectively fuse underwater visible light features and sonar detection features to fully utilize the advantages of both and make up for the shortcomings of a single information source, the present application introduces a two-way global attention joint perception module based on fine-grained deconstruction to fuse the grid-grained underwater visible light enhancement feature map and the grid-grained underwater sonar detection enhancement feature map, and improves the semantic joint perception ability of the model through feature fine-grained deconstruction and a two-way global attention mechanism to achieve fine-grained joint perception of multi-source data.

[0039] In an embodiment of the present application, the grid-granularity underwater visible light enhancement feature map and the grid-granularity underwater sonar detection enhancement feature map are input into a bidirectional global attention joint perception module based on fine-grained deconstruction to obtain the underwater visible light-sonar multi-source detection fusion feature map, including: first, performing feature fine-grained deconstruction on the grid-granularity underwater visible light enhancement feature map and the grid-granularity underwater sonar detection enhancement feature map to obtain a set of local feature matrices of the underwater visible light channel dimension and a set of local feature matrices of the underwater sonar detection channel dimension; by decomposing each feature map along the channel dimension into a set of local feature matrices of the channel dimension, so as to more carefully analyze the local information in the feature map. Next, each local feature matrix of the underwater visible light channel dimension in the set of local feature matrices of the underwater visible light channel dimension is used as a query feature matrix, and the set of local feature matrices of the underwater sonar detection channel dimension is used as a set of key feature matrices. The query feature matrix and the set of key feature matrices are input into a one-way global attention interaction module based on the first converter structure to obtain a set of local feature matrices of the underwater visible light channel dimension optimized by one-way global attention. Each local feature matrix of the underwater sonar detection channel dimension in the set of local feature matrices of the underwater sonar detection channel dimension is used as a query feature matrix, and the set of local feature matrices of the underwater visible light channel dimension is used as a set of key feature matrices. The set of the query feature matrix and the key feature matrix is ​​input into a one-way global attention interaction module based on the second transformer structure to obtain a set of one-way global attention optimized underwater sonar detection channel dimension local feature matrices; that is, after fine-grained deconstruction of the feature map, the multi-head attention mechanism in the Transformer structure is used to measure the correlation between the two channel dimension local feature matrix sets, strengthen the relationship between the features, and more comprehensively capture the global bidirectional semantic association information between the grid-granularity underwater visible light enhancement feature map and the grid-granularity underwater sonar detection enhancement feature map by performing one-way semantic interaction between each channel dimension local feature matrix and all channel dimension local feature matrices in the other set. Furthermore, the set of local feature matrices of the one-way global attention optimized underwater visible light channel dimension and the set of local feature matrices of the one-way global attention optimized underwater sonar detection channel dimension are respectively subjected to feature coupling along the channel dimension to obtain a one-way global interactive optimization underwater visible light enhancement feature map and a one-way global interactive optimization underwater sonar detection enhancement feature map; then, the position-weighted sum between the one-way global interactive optimization underwater visible light enhancement feature map and the one-way global interactive optimization underwater sonar detection enhancement feature map is calculated to obtain the underwater visible light-sonar multi-source detection fusion feature map.That is, the attention-optimized local feature matrix is ​​recombined along the channel dimension to form complete underwater visible light features and sonar detection features, and the two are fused by weighted summing by position to integrate the key information of the two, thus generating an underwater visible light-sonar multi-source detection fusion feature map containing rich interactive information.

[0040] Among them, each local feature matrix of the underwater visible light channel dimension in the set of local feature matrices of the underwater visible light channel dimension is used as a query feature matrix, and the set of local feature matrices of the underwater sonar detection channel dimension is used as a set of key feature matrices. The query feature matrix and the set of key feature matrices are input into a one-way global attention interaction module based on a first converter structure to obtain a set of local feature matrices of the underwater visible light channel dimension optimized by one-way global attention. The process includes: respectively calculating the matrix product of the query feature matrix and the transposed matrix of each key feature matrix in the set of key feature matrices, and then dividing the matrix product by the square root of the scale of the key feature matrix and passing it through a softmax function to obtain a set of attention weight matrices; respectively multiplying each attention weight matrix in the set of attention weight matrices with the query feature matrix, and then summing the multiplication results by position points to obtain the one-way global attention optimized local feature matrix of the underwater visible light channel dimension.

[0041] In summary, in the above embodiment, the grid-granularity underwater visible light enhancement feature map and the grid-granularity underwater sonar detection enhancement feature map are fine-grained jointly perceived to obtain an underwater visible light-sonar multi-source detection fusion feature map, including: processing the grid-granularity underwater visible light enhancement feature map and the grid-granularity underwater sonar detection enhancement feature map with the following attention joint perception formula to obtain the underwater visible light-sonar multi-source detection fusion feature map, wherein the attention joint perception formula is:

[0042]

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049] in, represents the underwater visible light enhancement feature map of the grid granularity, represents the grid-size underwater sonar detection enhancement feature map, Indicates deconstruction processing, , , and Respectively represent the first, second, and third local feature matrices in the underwater visible light channel dimension. and The local feature matrix of underwater visible light channel dimension, The value of is the number of feature matrices in the set of local feature matrices of the underwater visible light channel dimension, , , , Respectively represent the first, second, and third local feature matrices in the set of the underwater sonar detection channel dimension and The local feature matrix of underwater sonar detection channel dimension, represents the matrix multiplication operation, represents the scale of the local feature matrix of the underwater sonar detection channel dimension, and the scale of the local feature matrix of the underwater sonar detection channel dimension is the same as the scale of the local feature matrix of the underwater visible light channel dimension, represents the transpose of a matrix, is the normalized exponential function, , , and They respectively represent the first, second, and third local feature matrices in the set of the one-way global attention optimization underwater visible light channel dimension. and A one-way global attention optimizes the local feature matrix of the underwater visible light channel dimension, , , and They respectively represent the first, second, and third local feature matrices in the set of the one-way global attention optimization underwater sonar detection channel dimension. and A one-way global attention optimizes the local feature matrix of underwater sonar detection channel dimension, represents feature aggregation processing, represents the one-way global interactive optimization of underwater visible light enhancement feature map, It represents the one-way global interactive optimization of underwater sonar detection enhancement feature map. and For different weight coefficients, It represents the underwater visible light-sonar multi-source detection fusion feature map.

[0050] In particular, the multi-source data fusion visualization module 350 is used to generate an underwater detection multi-source fusion image based on the underwater visible light-sonar multi-source detection fusion feature map. In a specific example of the present application, the underwater visible light-sonar multi-source detection fusion feature map is input into a visualization module based on a diffusion model to obtain the underwater detection multi-source fusion image. In order to achieve a comprehensive visualization display of underwater visible light and sonar detection data, the present application further inputs the underwater visible light-sonar multi-source detection fusion feature map into a visualization module based on a diffusion model for processing to generate an underwater detection multi-source fusion image. Specifically, the diffusion model is an advanced image generation technology that generates high-quality images by simulating the physical diffusion process. In the technical solution of the present application, the module first adds noise to the underwater visible light-sonar multi-source detection fusion feature map, and then gradually reduces the noise through a series of diffusion steps, restores the image resolution, and enhances the image detail expression, and finally generates a clear underwater detection multi-source fusion image, thereby realizing the comprehensive visualization of multi-source data, so that underwater maintenance personnel can more intuitively identify and analyze the status of underwater facilities or equipment, and improve the accuracy and efficiency of maintenance operations.

[0051] In the preferred example, since the grid-granularity underwater visible light enhancement feature map and the grid-granularity underwater sonar detection enhancement feature map represent the image semantic grid-granularity enhancement features of the underwater visible light image and the underwater sonar detection image respectively, when they are input into the bidirectional global attention joint perception module based on fine-grained deconstruction, the differences in multi-source image semantic features will have different attention weights based on the fine-grained decoupling of the spatial distribution of image semantic features. As a result, the underwater visible light-sonar multi-source detection fusion will also have a diversified set expression distribution of multi-source image global aggregation features. Therefore, it is expected to improve the balance between the accuracy and completeness of the regression mapping when the underwater visible light-sonar multi-source detection fusion is input into the visual module based on the diffusion model for generative regression, thereby improving the image quality of the obtained underwater detection multi-source fusion image.

[0052] Preferably, in an example of the present application, inputting the underwater visible light-sonar multi-source detection fusion feature map into a visual module based on a diffusion model to obtain the underwater detection multi-source fusion image includes:

[0053] Calculating the sum of the absolute values ​​of each eigenvalue of the underwater visible light-sonar multi-source detection fusion feature map to obtain a first underwater visible light-sonar multi-source detection fusion spatial structure value, and calculating the square root of the sum of the squares of each eigenvalue of the underwater visible light-sonar multi-source detection fusion feature map to obtain a second underwater visible light-sonar multi-source detection fusion spatial structure value;

[0054] Multiplying each eigenvalue of the underwater visible light-sonar multi-source detection fusion feature map by the first underwater visible light-sonar multi-source detection fusion space structure value and the second underwater visible light-sonar multi-source detection fusion space structure value, respectively, to obtain a first underwater visible light-sonar multi-source detection fusion structure reference value and a second underwater visible light-sonar multi-source detection fusion structure reference value corresponding to each eigenvalue;

[0055] Multiplying each eigenvalue of the underwater visible light-sonar multi-source detection fusion feature map by the scale of the underwater visible light-sonar multi-source detection fusion feature map and the square root of the scale, respectively, to obtain a first underwater visible light-sonar multi-source detection fusion scale transformation value and a second underwater visible light-sonar multi-source detection fusion scale transformation value corresponding to each eigenvalue;

[0056] Dividing the first underwater visible light-sonar multi-source detection fusion structure reference value by the difference between the first underwater visible light-sonar multi-source detection fusion space structure value and the first underwater visible light-sonar multi-source detection fusion scale transformation value to obtain a first underwater visible light-sonar multi-source detection fusion transformation adjustment value;

[0057] Dividing the second underwater visible light-sonar multi-source detection fusion structure reference value by the difference between the second underwater visible light-sonar multi-source detection fusion space structure value and the second underwater visible light-sonar multi-source detection fusion scale transformation value to obtain a second underwater visible light-sonar multi-source detection fusion transformation adjustment value;

[0058] Calculating a weighted sum of the first underwater visible light-sonar multi-source detection fusion transformation adjustment value and the second underwater visible light-sonar multi-source detection fusion transformation adjustment value to obtain each eigenvalue of the optimized underwater visible light-sonar multi-source detection fusion feature map;

[0059] The optimized underwater visible light-sonar multi-source detection fusion feature map is input into a visual module based on a diffusion model to obtain the underwater detection multi-source fusion image.

[0060] Among them, the underwater visible light-sonar multi-source detection fusion feature map The optimization expression is:

[0061]

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] in, is the underwater visible light-sonar multi-source detection fusion feature map, represents the set of real numbers, , , Represent the width, length and height of the feature map respectively, The first The eigenvalues ​​at the positions, represents the scale of the feature map, It represents the first underwater visible light-sonar multi-source detection fusion spatial structure value, Represents the second underwater visible light-sonar multi-source detection fusion spatial structure value, It means point multiplication by position. It means adding by position. It represents the first underwater visible light-sonar multi-source detection fusion transformation adjustment feature map, It represents the first underwater visible light-sonar multi-source detection fusion transformation adjustment value, It represents the second underwater visible light-sonar multi-source detection fusion transformation adjustment feature map, Indicates the second underwater visible light-sonar multi-source detection fusion transformation adjustment value, represents the weighted hyperparameter, Represents the optimized underwater visible light-sonar multi-source detection fusion feature map.

[0071] Therefore, in the preferred example, for the spatial structure information of the feature set of the underwater visible light-sonar multi-source detection fusion feature map in the high-dimensional space, a scale-based frame transformation of each eigenvalue of the underwater visible light-sonar multi-source detection fusion feature map is performed by taking the class-norm spatial structured representation of the underwater visible light-sonar multi-source detection fusion feature map as a reference window, and a spatial structure-based frame attention weight adjustment of each eigenvalue of the underwater visible light-sonar multi-source detection fusion feature map is implemented to ensure that the underwater visible light-sonar multi-source detection fusion feature map is in the feature The invariance of spatial transformation (translation, scaling and rotation) under spatial interaction can improve the convergence and generalization effect of generative regression of the feature set of the underwater visible light-sonar multi-source detection fusion feature map under complex spatial structure representation. In this way, the balanced executableness between mapping accuracy and mapping completeness is achieved in the generative regression process based on the diversified feature distribution of the underwater visible light-sonar multi-source detection fusion feature map, and the image quality of the underwater detection multi-source fusion image obtained by inputting the underwater visible light-sonar multi-source detection fusion feature map into the visual module based on the diffusion model is improved.

[0072] As described above, the multi-source data fusion visualization system 300 of the underwater intelligent maintenance robot according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a multi-source data fusion visualization algorithm of the underwater intelligent maintenance robot. In one possible implementation, the multi-source data fusion visualization system 300 of the underwater intelligent maintenance robot according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the multi-source data fusion visualization system 300 of the underwater intelligent maintenance robot can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the multi-source data fusion visualization system 300 of the underwater intelligent maintenance robot can also be one of the many hardware modules of the wireless terminal.

[0073] Alternatively, in another example, the multi-source data fusion visualization system 300 of the underwater intelligent maintenance robot and the wireless terminal may also be separate devices, and the multi-source data fusion visualization system 300 of the underwater intelligent maintenance robot may be connected to the wireless terminal via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0074] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. The multi-source data fusion visualization system of underwater intelligent maintenance robot is characterized by: include: An underwater image acquisition module is used to acquire underwater visible light images and underwater sonar detection images collected by a visible light camera and a sonar detector of an underwater robot; An underwater image feature extraction module is used to extract image features from the underwater visible light image and the underwater sonar detection image to obtain an underwater visible light feature map and an underwater sonar detection feature map respectively; A feature gating enhancement module is used to perform feature gating enhancement processing on the underwater visible light feature map and the underwater sonar detection feature map to obtain a grid-granularity underwater visible light enhancement feature map and a grid-granularity underwater sonar detection enhancement feature map; A fine-grained joint perception module is used to perform fine-grained joint perception on the grid-grained underwater visible light enhancement feature map and the grid-grained underwater sonar detection enhancement feature map to obtain an underwater visible light-sonar multi-source detection fusion feature map; The multi-source data fusion visualization module is used to generate an underwater detection multi-source fusion image based on the underwater visible light-sonar multi-source detection fusion feature map.

2. The multi-source data fusion visualization system of the underwater intelligent maintenance robot according to claim 1 is characterized in that: The underwater image feature extraction module is used to: The underwater visible light image and the underwater sonar detection image are respectively input into an image feature extractor based on a hole convolutional neural network model to obtain the underwater visible light feature map and the underwater sonar detection feature map.

3. The multi-source data fusion visualization system of the underwater intelligent maintenance robot according to claim 2 is characterized in that: The feature gating enhancement module is used to: The underwater visible light feature map and the underwater sonar detection feature map are input into a feature gated enhancement module based on grid energy significance guidance to obtain the grid-granularity underwater visible light enhancement feature map and the grid-granularity underwater sonar detection enhancement feature map.

4. The multi-source data fusion visualization system of the underwater intelligent maintenance robot according to claim 3 is characterized in that: The feature gating enhancement module comprises: A grid division unit, used for performing grid division on the underwater visible light characteristic map to obtain a set of underwater visible light local characteristic maps; A feature significance calculation unit, used for calculating the energy significance description factor of each underwater visible light local feature map in the set of underwater visible light local feature maps to obtain a set of underwater visible light local feature energy significance description factors; An adaptive selection unit, used for inputting the set of underwater visible light local feature energy significance description factors into a local feature adaptive selector based on a gating function to obtain a set of underwater visible light local feature significance modulation weights; a weighting unit, configured to weight each underwater visible light local feature map in the set of underwater visible light local feature maps by using each underwater visible light local feature significant modulation weight in the set of underwater visible light local feature significant modulation weights as a weight, so as to obtain a set of enhanced underwater visible light local feature maps; A feature aggregation unit, used for performing feature aggregation on the set of enhanced underwater visible light local feature maps according to the grid division method to obtain a local significant guided enhanced underwater visible light feature map; A feature dispersion unit, used for inputting the local significant guided enhanced underwater visible light feature map into a feature dispersion module based on a hole convolution layer to obtain the grid-granularity underwater visible light enhancement feature map; The feature gating enhancement module is used to process the underwater visible light feature map using the following feature gating enhancement formula to obtain a grid-scale underwater visible light enhanced feature map, wherein the feature gating enhancement formula is: in, represents the underwater visible light characteristic graph, represents the meshing process, , , and respectively represent the first, second, and third in the set of underwater visible light local feature maps and A local feature map of underwater visible light, The value of is the number of feature maps in the set of underwater visible light local feature maps, , , and Respectively represent the The maximum eigenvalue, minimum eigenvalue, feature mean and feature variance of the underwater visible light local feature map, represents the bias term, Indicates the The energy significance description factor of the underwater visible light local feature map, Indicates mask processing, express The function's variables, is the preset threshold, Indicates The significant modulation weight of the underwater visible light local feature, represents feature aggregation processing, represents the local significant guided enhancement of underwater visible light feature map, represents the dilated convolution operation, The grid granularity underwater visible light enhancement feature map is represented.

5. The multi-source data fusion visualization system of the underwater intelligent maintenance robot according to claim 4 is characterized in that: The feature significance calculation unit is used to: The difference between the maximum eigenvalue and the minimum eigenvalue of the underwater visible light local feature map is calculated, and the difference is divided by the weighted sum of the feature mean, feature variance and bias term of the underwater visible light local feature map to obtain the underwater visible light local feature energy significance description factor.

6. The multi-source data fusion visualization system of the underwater intelligent maintenance robot according to claim 5 is characterized in that: The adaptive selection unit is used to: Inputting the set of underwater visible light local feature energy significance description factors into a Sigmoid function for normalization processing to obtain a set of normalized underwater visible light local feature energy significance description factors; Based on a preset threshold, a mask process is performed on the set of normalized underwater visible light local feature energy significance description factors to obtain a set of underwater visible light local feature significance modulation weights.

7. The multi-source data fusion visualization system of the underwater intelligent maintenance robot according to claim 6 is characterized in that: The fine-grained joint perception module is used to: The grid-granularity underwater visible light enhancement feature map and the grid-granularity underwater sonar detection enhancement feature map are input into a bidirectional global attention joint perception module based on fine-grained deconstruction to obtain the underwater visible light-sonar multi-source detection fusion feature map.

8. The multi-source data fusion visualization system of the underwater intelligent maintenance robot according to claim 7 is characterized in that: The fine-grained joint perception module includes: A fine-grained deconstruction unit, used for performing feature fine-grained deconstruction on the grid-grained underwater visible light enhancement feature map and the grid-grained underwater sonar detection enhancement feature map to obtain a set of local feature matrices of underwater visible light channel dimension and a set of local feature matrices of underwater sonar detection channel dimension; A first unidirectional global attention interaction unit is used to use each local feature matrix of the underwater visible light channel dimension in the set of local feature matrices of the underwater visible light channel dimension as a query feature matrix, and use the set of local feature matrices of the underwater sonar detection channel dimension as a set of key feature matrices, and input the query feature matrix and the set of key feature matrices into a unidirectional global attention interaction module based on a first converter structure to obtain a set of local feature matrices of the underwater visible light channel dimension optimized by unidirectional global attention; A second unidirectional global attention interaction unit is used to use each underwater sonar detection channel dimension local feature matrix in the set of underwater sonar detection channel dimension local feature matrices as a query feature matrix, and use the set of underwater visible light channel dimension local feature matrices as a set of key feature matrices, and input the query feature matrix and the set of key feature matrices into a unidirectional global attention interaction module based on a second converter structure to obtain a set of unidirectional global attention optimized underwater sonar detection channel dimension local feature matrices; A feature coupling unit is used to perform feature coupling along the channel dimension on the set of local feature matrices of the one-way global attention optimization underwater visible light channel dimension and the set of local feature matrices of the one-way global attention optimization underwater sonar detection channel dimension, respectively, to obtain a one-way global interactive optimization underwater visible light enhancement feature map and a one-way global interactive optimization underwater sonar detection enhancement feature map; The multi-source feature fusion unit is used to calculate the position-weighted sum between the one-way global interactive optimization underwater visible light enhancement feature map and the one-way global interactive optimization underwater sonar detection enhancement feature map to obtain the underwater visible light-sonar multi-source detection fusion feature map.

9. The multi-source data fusion visualization system of the underwater intelligent maintenance robot according to claim 8 is characterized in that: The first one-way global attention interaction unit is used to: Calculating the matrix product of the query feature matrix and the transposed matrix of each key feature matrix in the set of key feature matrices respectively, and then dividing the matrix product by the square root of the scale of the key feature matrix and passing it through a softmax function to obtain a set of attention weight matrices; Each attention weight matrix in the set of attention weight matrices is multiplied with the query feature matrix respectively, and then the multiplication results are summed by position point to obtain the one-way global attention optimized underwater visible light channel dimension local feature matrix.

10. The multi-source data fusion visualization system of the underwater intelligent maintenance robot according to claim 9 is characterized in that: The multi-source data fusion visualization module is used to: The underwater visible light-sonar multi-source detection fusion feature map is input into a visual module based on a diffusion model to obtain the underwater detection multi-source fusion image.

Images