A 3D visual information fusion optimization system for image analysis

Abstract

A three-dimensional visual information fusion optimization system for image analysis. The system first obtains a visible light image and an infrared image of an observation area, then extracts image features from the infrared image and the visible light image to obtain a visible light image semantic feature map and an infrared image semantic feature map, then fuses the visible light image semantic feature map and the infrared image semantic feature map through a global interactive attention fusion module to obtain a multi-source complementary semantic feature map, and finally generates an information fusion optimized image of the observation area based on the multi-source complementary semantic feature map.

Description

Technical Field

[0001] The present disclosure relates to the field of three-dimensional visual information fusion, and in particular to a three-dimensional visual information fusion optimization system for image analysis. Background Art

[0002] With the continuous development of technology and the widespread application of aircraft, images have become one of the important means of obtaining large-scale geographic information. As two common data sources, visible light images and infrared images have unique information characteristics and advantages. Visible light images can provide rich color information and detailed features, while infrared images can reflect the thermal distribution of the target and have certain nighttime monitoring capabilities.

[0003] However, a single image is often difficult to fully represent the information of the observed area, so it is of great significance to fuse and optimize multi-source data. In other words, we expect a three-dimensional visual information fusion optimization system for image analysis to provide more refined and comprehensive data support for geographic information applications, environmental monitoring and other fields. Summary of the invention

[0004] The present disclosure is made in view of the above problems. One object of the present disclosure is to provide a three-dimensional visual information fusion optimization system for image analysis.

[0005] The embodiment of the present disclosure provides a three-dimensional visual information fusion optimization system for image analysis, which includes:

[0006] A multi-source image acquisition module, used to acquire visible light images and infrared images of the observation area;

[0007] An image feature extraction module, used for performing image feature extraction on the infrared image and the visible light image to obtain a visible light image semantic feature map and an infrared image semantic feature map;

[0008] A multi-source information complementary fusion module, used for fusing the visible light image semantic feature map and the infrared image semantic feature map through a global interactive attention fusion module to obtain a multi-source complementary semantic feature map; and

[0009] An image optimization module is used to generate an information fusion optimized image of the observation area based on the multi-source complementary semantic feature map.

[0010] For example, according to the three-dimensional visual information fusion optimization system for image analysis according to an embodiment of the present disclosure, the multi-source image acquisition module includes:

[0011] a visible light unit, configured to obtain the visible light image of the observation area collected by a visible light sensor carried by the aircraft; and

[0012] The infrared unit is used to obtain the infrared image of the observation area collected by the infrared sensor carried by the aircraft.

[0013] For example, according to the three-dimensional visual information fusion optimization system for image analysis according to an embodiment of the present disclosure, the image feature extraction module includes:

[0014] a balancing correction unit, configured to perform balancing correction on the infrared image based on prior information to obtain a corrected infrared image; and

[0015] A feature encoding unit is used to perform feature encoding on the visible light image and the corrected infrared image using a deep learning network model to obtain a semantic feature map of the visible light image and a semantic feature map of the infrared image.

[0016] For example, according to the three-dimensional visual information fusion optimization system for image analysis according to an embodiment of the present disclosure, the equalization correction unit is used to:

[0017] The infrared image is processed using the following balanced correction formula to obtain the corrected infrared image; wherein the balanced correction formula is:

[0018] f θ =f a +u θ ⊙f b

[0019] Among them, f a is the gray level corresponding to the minimum value of each pixel in the infrared image, f b is the gray level corresponding to the maximum value of each pixel in the infrared image, u θ is the θth value randomly sampled from a uniform distribution, f θ is the θth pixel in the corrected infrared image, and ⊙ represents point multiplication by position.

[0020] For example, according to the three-dimensional visual information fusion optimization system for image analysis according to an embodiment of the present disclosure, the feature encoding unit is used to:

[0021] Passing the visible light image through an image feature extractor based on a convolutional neural network model to obtain a semantic feature map of the visible light image; and

[0022] The corrected infrared image is passed through the image feature extractor based on the convolutional neural network model to obtain the infrared image semantic feature map.

[0023] For example, according to the three-dimensional visual information fusion optimization system for image analysis according to an embodiment of the present disclosure, the image feature extractor based on the convolutional neural network model includes an input layer, a convolution layer, an activation layer, a pooling layer and an output layer.

[0024] For example, according to the three-dimensional visual information fusion optimization system for image analysis according to an embodiment of the present disclosure, the multi-source information complementary fusion module is used to:

[0025] The visible light image semantic feature map and the infrared image semantic feature map are processed by the following information complementary fusion formula to obtain an information complementary representation feature vector; wherein the information complementary fusion formula is:

[0026]

[0027] Among them, U1 and U2 are the semantic feature map of the visible light image and the semantic feature map of the infrared image respectively, and U is a complementary fusion feature map. represents cascade processing, S c represents the cth eigenvalue in the information complementary characterization eigenvector, F gp (U c ) represents the global mean pooling process for the c-th feature matrix in the complementary fusion feature map, H and W are the height and width of the c-th feature matrix in the complementary fusion feature map, respectively, and U c (i, j) is the eigenvalue at position (i, j) in the c-th feature matrix of the complementary fusion feature map;

[0028] Activating the information complementarity representation feature vector to obtain a condensed information complementarity representation feature vector;

[0029] The visible light image semantic feature map, the infrared image semantic feature map and the condensed information complementary representation feature vector are processed using the following attention weight application formula to obtain the multi-source complementary semantic feature map; wherein the attention weight application formula is:

[0030]

[0031] Among them, a c and b c represents the weighted probability vectors of the semantic feature map U1 of the visible light image and the semantic feature map U2 of the infrared image, Acz represents the vector element of the c channel corresponding to the semantic feature map of the visible light image in the condensed information complementary representation feature vector, Bcz represents the vector element of the c channel corresponding to the semantic feature map of the infrared image in the condensed information complementary representation feature vector, F c is the multi-source complementary semantic feature map, Indicates cascade processing.

[0032] For example, according to the three-dimensional visual information fusion optimization system for image analysis according to an embodiment of the present disclosure, the image optimization module includes:

[0033] A feature structure balancing and regularizing unit, used for performing dimension reduction and optimization processing on the multi-source complementary semantic feature map to obtain an optimized multi-source complementary semantic feature matrix; and

[0034] An image generating unit is used to obtain an information fusion optimized image of the observation area by passing the optimized multi-source complementary semantic feature matrix through an AIGC-based image generator.

[0035] For example, according to the three-dimensional visual information fusion optimization system for image analysis according to an embodiment of the present disclosure, the characteristic structure balance regularization unit includes:

[0036] A pooling dimension reduction subunit, used for performing global mean pooling processing on the multi-source complementary semantic feature map to obtain a multi-source complementary semantic feature matrix; and

[0037] The feature distribution optimization subunit is used to perform feature distribution clustering optimization on the multi-source complementary semantic feature matrix to obtain the optimized multi-source complementary semantic feature matrix.

[0038] According to the three-dimensional visual information fusion optimization system for image analysis of the embodiment of the present disclosure, it utilizes image processing technology and intelligent algorithms to perform image semantic understanding and complementary fusion of visible light images and infrared images of the observation area to intelligently generate an information fusion optimized image of the observation area. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings of the embodiments of the present disclosure. Obviously, the drawings described below only relate to some embodiments of the present disclosure, and are not intended to limit the present disclosure.

[0040] Figure 1 A schematic diagram of the application architecture of a three-dimensional visual information fusion optimization system for image analysis in an embodiment of the present disclosure is shown;

[0041] Figure 2 A schematic diagram of the structure of a three-dimensional visual information fusion optimization system for image analysis in an embodiment of the present disclosure is shown;

[0042] Figure 3 A flow chart of a three-dimensional visual information fusion optimization method for image analysis in an embodiment of the present disclosure is shown;

[0043] Figure 4An application scenario diagram of a three-dimensional visual information fusion optimization system for image analysis in an embodiment of the present disclosure is shown; and

[0044] Figure 5 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present disclosure, rather than all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present disclosure.

[0046] The terms used in this specification are those general terms currently widely used in the art in consideration of the functions of the present disclosure, but these terms may vary according to the intention of a person of ordinary skill in the art, precedents, or new technologies in the art. In addition, specific terms may be selected by the applicant, and in this case, their detailed meanings will be described in the detailed description of the present disclosure. Therefore, the terms used in the specification should not be understood as simple names, but rather as an overall description based on the meaning of the terms and the present disclosure.

[0047] Although the present disclosure makes various references to certain modules in the system according to embodiments of the present disclosure, any number of different modules can be used and run on a user terminal and / or server. The modules are illustrative only, and different aspects of the systems and methods can use different modules.

[0048] Flowcharts are used in this disclosure to illustrate the operations performed by the system according to the embodiments of the present disclosure. It should be understood that the preceding or following operations are not necessarily performed precisely in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0049] Figure 1 A schematic diagram of the application architecture of a three-dimensional visual information fusion optimization system for image analysis in an embodiment of the present disclosure is shown, including a server 100 and a terminal device 200.

[0050] The terminal device 200 and the server 100 can be connected via the Internet to achieve mutual communication. Optionally, the above-mentioned Internet uses standard communication technology and / or protocol. The Internet is usually the Internet, but it can also be any network, including but not limited to any combination of a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or a virtual private network. In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0051] The server 100 can provide various network services for the terminal device 200, wherein the server 100 can be a single server, a server cluster consisting of several servers, or a cloud computing center. Specifically, the server 100 may include a processor 110 (Center Processing Unit, CPU), a memory 120, an input device 130, and an output device 140, etc. The input device 130 may include a keyboard, a mouse, a touch screen, etc., and the output device 140 may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), etc.

[0052] The memory 120 may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor 110 with program instructions and data stored in the memory 120. In the embodiment of the present disclosure, the memory 120 may be used to store the corresponding method of the three-dimensional visual information fusion optimization system for image analysis in the embodiment of the present disclosure.

[0053] The processor 110 calls the program instructions stored in the memory 120, and the processor 110 is used to execute the steps of any one of the three-dimensional visual information fusion optimization methods for image analysis in the embodiments of the present disclosure according to the obtained program instructions.

[0054] In addition, the application architecture diagram in the embodiment of the present disclosure is intended to more clearly illustrate the technical solution in the embodiment of the present disclosure, and does not constitute a limitation on the technical solution provided in the embodiment of the present disclosure. Of course, for other application architectures and business applications, the technical solution provided in the embodiment of the present disclosure is also applicable to similar problems.

[0055] The following is a non-restrictive description of a three-dimensional visual information fusion optimization system for image analysis provided according to at least one embodiment of the present disclosure through several examples or embodiments. As described below, different features in these specific examples or embodiments can be combined with each other without conflicting with each other to obtain new examples or embodiments, and these new examples or embodiments also fall within the scope of protection of the present disclosure.

[0056] As two common data sources, visible light images and infrared images have unique information characteristics and advantages. Specifically, visible light images are easy to obtain, have high image resolution, strong contrast, and can reflect more detailed information such as color and texture; however, the imaging quality of visible light images is easily affected by conditions such as light intensity and weather, and does not have the ability to work 24 hours a day. For example, at night, in shadows, and under conditions such as cloud cover, the image quality decreases, and it is difficult to obtain effective target information. In this case, infrared images obtained by infrared sensors can be used as observation data sources. Among them, infrared sensors can detect infrared wavelengths that cannot be perceived by the human eye. The brightness value of the obtained infrared image is related to the temperature of the object. Generally, the higher the brightness, the higher the temperature of the area; the acquisition of infrared images is not easily affected by extreme weather. Under observation conditions such as night, shadows, and cloud cover, the image can still obtain effective target information and has the ability to work 24 hours a day. However, compared with visible light images, infrared images usually have lower resolution and contrast, less texture detail information, and under good observation conditions with sufficient light and no obstruction, visible light images can usually obtain clearer and richer target information.

[0057] Therefore, the image information obtained by a single sensor has its own limitations and one-sidedness, and cannot meet the detection requirements under actual complex scene conditions. If the multi-source image data can be fused, the key information of the multi-source image can be fully utilized, and the complementarity between the multi-source images can be used to make up for the lack of single-source image information. In this regard, the technical concept of this application is: using image processing technology and intelligent algorithms to perform image semantic understanding and complementary fusion of the visible light image and infrared image of the observation area, so as to intelligently generate an information fusion optimized image of the observation area.

[0058] Based on this, Figure 2 The structural schematic diagram of a three-dimensional visual information fusion optimization system 800 for image analysis in an embodiment of the present disclosure is shown. The three-dimensional visual information fusion optimization system 800 for image analysis includes: a multi-source image acquisition module 810, which is used to acquire visible light images and infrared images of the observation area; an image feature extraction module 820, which is used to extract image features from the infrared image and the visible light image to obtain a visible light image semantic feature map and an infrared image semantic feature map; a multi-source information complementary fusion module 830, which is used to obtain a multi-source complementary semantic feature map by passing the visible light image semantic feature map and the infrared image semantic feature map through a global interactive attention fusion module; and an image optimization module 840, which is used to generate an information fusion optimized image of the observation area based on the multi-source complementary semantic feature map. It should be understood that the design idea of ​​this system is to use two different types of images, visible light and infrared, to extract image features and perform information fusion to achieve a more comprehensive and accurate image analysis and optimization purpose. The application of the global interactive attention fusion module can help the system better integrate information from different sources and improve the effect and accuracy of the analysis.

[0059] Specifically, in the technical solution of the present application, first, a visible light image of the observation area collected by a visible light sensor carried by an aircraft is obtained; and an infrared image of the observation area collected by an infrared sensor carried by the aircraft is obtained. Among them, the visible light image can provide rich color information and detail features, which is helpful for accurate classification and identification of ground objects. Infrared images can reflect the thermal distribution of the target, which is very helpful for identifying targets with different thermal characteristics, such as monitoring and analysis at night or in low light conditions. Visible light images and infrared images are complementary. They can provide different types of information and features. By fusing the information expressed by the two types of images, the characteristics of the observation area can be understood more comprehensively and accurately.

[0060] Correspondingly, the multi-source image acquisition module 810 includes: a visible light unit, used to acquire the visible light image of the observation area collected by the visible light sensor carried by the aircraft; and an infrared unit, used to acquire the infrared image of the observation area collected by the infrared sensor carried by the aircraft.

[0061] It should be understood that infrared occurs in the infrared spectrum of the electromagnetic spectrum, and mainly senses infrared rays reflected by objects on the ground. Since infrared rays have a long wavelength and strong penetration in the atmosphere, infrared photography is not affected by smoke, and images of the ground can still be captured through a very thick atmosphere. However, there are some problems with infrared images, such as low contrast and blurred visual effects. In the technical solution of the present application, it is expected that the infrared image is subjected to equalization correction based on prior information to obtain a corrected infrared image. Here, the equalization correction processing based on prior information is to enhance the infrared image so that each pixel of the infrared image can be evenly distributed in the entire grayscale level range, thereby improving the contrast and brightness of the image.

[0062] That is to say, it is hoped that the image histogram distribution of the corrected infrared image is balanced, that is, ideally, the probability distribution of pixel values ​​in the image obeys uniform distribution. Therefore, in a specific example of the present application, the infrared image is enhanced by histogram equalization based on the prior condition of uniform distribution.

[0063] Then, the visible light image is passed through an image feature extractor based on a convolutional neural network model to obtain a semantic feature map of the visible light image. Among them, the convolutional neural network (CNN) model has a powerful feature extraction capability. Through the CNN model, high-level and abstract features can be learned from the visible light image, which helps to better understand the image content. Specifically, the CNN model usually includes multiple layers of convolution and pooling layers, which can extract the modal neighborhood structure and regional features in the visible light image step by step. At the same time, the corrected infrared image is passed through the image feature extractor based on the convolutional neural network model to obtain a semantic feature map of the infrared image. Similarly, the feature extraction characteristics of the convolutional neural network (CNN) are used to learn abstract features in the corrected infrared image, such as temperature distribution.

[0064] Accordingly, the image feature extraction module 820 includes: an equalization correction unit for performing equalization correction on the infrared image based on prior information to obtain a corrected infrared image; and a feature encoding unit for performing feature encoding on the visible light image and the corrected infrared image using a deep learning network model to obtain the semantic feature map of the visible light image and the semantic feature map of the infrared image. It should be understood that equalization correction based on prior information is a correction method that aims to improve the quality and accuracy of an image by utilizing prior knowledge or previously obtained information. In image processing, equalization correction is generally used to adjust the contrast and brightness of an image to better display the details and features in the image. In this system, the equalization correction unit in the image feature extraction module 820 processes the infrared image. Specifically, equalization correction based on prior information may utilize previously collected knowledge about the characteristics of the infrared image, such as sensor characteristics, environmental conditions, etc., to adjust the brightness and contrast of the infrared image to eliminate possible noise or unevenness, thereby obtaining a clearer and more accurate corrected infrared image. This prior information can be empirical knowledge, statistical data, physical models, etc., which is used to guide the correction process to ensure that the corrected infrared image can better adapt to the subsequent feature extraction and analysis process. In this way, the system can more effectively utilize previously acquired knowledge and improve the efficiency and accuracy of image processing.

[0065] The equalization correction unit is used to process the infrared image using the following equalization correction formula to obtain the corrected infrared image; wherein the equalization correction formula is:

[0066] f θ =f a +u θ ⊙f b

[0067] Among them, f a is the gray level corresponding to the minimum value of each pixel in the infrared image, f b is the gray level corresponding to the maximum value of each pixel in the infrared image, u θ is the θth value randomly sampled from a uniform distribution, f θ is the θth pixel in the corrected infrared image, and ⊙ represents point multiplication by position.

[0068] The feature encoding unit is used to: pass the visible light image through an image feature extractor based on a convolutional neural network model to obtain a semantic feature map of the visible light image; and pass the corrected infrared image through the image feature extractor based on a convolutional neural network model to obtain a semantic feature map of the infrared image. Specifically, the image feature extractor based on a convolutional neural network model includes an input layer, a convolution layer, an activation layer, a pooling layer, and an output layer.

[0069] It is worth mentioning that the Convolutional Neural Network (CNN) is a deep learning neural network that is specifically used to process data with a grid-like structure (such as images). The convolutional neural network contains the following main parts: 1. Input layer: used to receive raw input data, such as image data; 2. Convolution layer: extract features in the image by applying convolution operations. The convolution operation can capture local features at different locations in the image, thereby achieving translation invariance; 3. Activation layer: activation functions such as ReLU (Rectified Linear Unit) are usually added after the convolution layer to introduce nonlinearity and increase the expressive power of the network; 4. Pooling layer: pooling operations (such as maximum pooling or average pooling) are used to reduce the size of the feature map, reduce computational complexity, and retain important features; 5. Output layer: the final output network prediction result, which can be a classification label, regression value, etc., depending on the task of the network. In the system, the image feature extractor based on the convolutional neural network model uses the structure and characteristics of CNN to extract features from visible light images and rectified infrared images to obtain their semantic feature maps. These semantic feature maps can capture high-level semantic information in the image, which is helpful for subsequent image analysis and processing tasks. By using CNN, the system can automatically learn and extract features in the image without manually designing feature extractors, thereby improving the efficiency and accuracy of image processing.

[0070] Subsequently, the visible light image semantic feature map and the infrared image semantic feature map are fused through a global interactive attention fusion module to obtain a multi-source complementary semantic feature map. That is, the unique image semantic features contained in the visible light image and the infrared image are fused through the global interactive attention fusion module, and an attention mechanism is introduced to enhance the focus on important features, thereby comprehensively utilizing the complementary information between the two, effectively integrating the semantic features of the visible light image and the infrared image, and avoiding information loss.

[0071] Accordingly, the multi-source information complementary fusion module 830 is used to: process the visible light image semantic feature map and the infrared image semantic feature map according to the following information complementary fusion formula to obtain an information complementary representation feature vector; wherein the information complementary fusion formula is:

[0072]

[0073] Among them, U1 and U2 are the semantic feature map of the visible light image and the semantic feature map of the infrared image respectively, and U is a complementary fusion feature map. represents cascade processing, S c represents the cth eigenvalue in the information complementary characterization eigenvector, F gp (U c ) represents the global mean pooling process for the c-th feature matrix in the complementary fusion feature map, H and W are the height and width of the c-th feature matrix in the complementary fusion feature map, respectively, and U c (i, j) is the eigenvalue at the position (i, j) in the cth feature matrix in the complementary fusion feature map; the information complementary representation feature vector is activated to obtain a condensed information complementary representation feature vector; and the visible light image semantic feature map, the infrared image semantic feature map and the condensed information complementary representation feature vector are processed using the following attention weight application formula to obtain the multi-source complementary semantic feature map; wherein the attention weight application formula is:

[0074]

[0075] Among them, a c and b c represents the weighted probability vectors of the semantic feature map U1 of the visible light image and the semantic feature map U2 of the infrared image, Acz represents the vector element of the c channel corresponding to the semantic feature map of the visible light image in the condensed information complementary representation feature vector, Bcz represents the vector element of the c channel corresponding to the semantic feature map of the infrared image in the condensed information complementary representation feature vector, F c is the multi-source complementary semantic feature map, Indicates cascade processing.

[0076] Then, the multi-source complementary semantic feature map is subjected to global mean pooling processing to obtain a multi-source complementary semantic feature matrix; and the multi-source complementary semantic feature matrix is ​​passed through an AIGC-based image generator to obtain an information fusion optimized image of the observation area. Among them, the information in the multi-source complementary semantic feature map can be aggregated by global mean pooling to obtain representative overall features. This global mean pooling processing method can also reduce the dimension of the multi-source complementary semantic feature map, so that the multi-source complementary semantic feature matrix can be directly used as the input of the AIGC-based image generator to generate the corresponding information fusion optimized image.

[0077] Correspondingly, the image optimization module 840 includes: a feature structure balancing and regularizing unit, used to perform dimensionality reduction and optimization processing on the multi-source complementary semantic feature map to obtain an optimized multi-source complementary semantic feature matrix; and an image generation unit, used to pass the optimized multi-source complementary semantic feature matrix through an AIGC-based image generator to obtain an information fusion optimized image of the observation area.

[0078] It is worth mentioning that the main functions of AIGC include: Instance Normalization: Instance normalization is a normalization technique used to adjust the feature response of the intermediate layer of the neural network to make it more stable and easier to train. Instance normalization can improve the quality and stability of image generation and help generate more realistic and clear images. Group convolution: Group convolution is a structure in convolutional neural networks that divides input channels into multiple groups. The channels in each group share weights. Group convolution helps reduce the number of parameters and improve computational efficiency while maintaining the network's ability to extract features. By combining instance normalization and group convolution, AIGC can provide better performance and effects in image generation tasks, help optimize the multi-source complementary semantic feature matrix, and thus generate information fusion optimized images of the observed area. This technology can improve the accuracy, clarity, and realism of image generation, which is of great significance for image processing and analysis.

[0079] Among them, in one example, the feature structure balance and regularization unit includes: a pooling dimensionality reduction subunit, used to perform global mean pooling processing on the multi-source complementary semantic feature map to obtain a multi-source complementary semantic feature matrix; and a feature distribution optimization subunit, used to perform feature distribution clustering optimization on the multi-source complementary semantic feature matrix to obtain the optimized multi-source complementary semantic feature matrix.

[0080] In the above technical scheme, the visible light image semantic feature map and the infrared image semantic feature map respectively express the image semantic features of the visible light image and the corrected infrared image in the spatial distribution dimension of the feature matrix, and the channel distribution of the convolutional neural network model is followed between each feature matrix. Therefore, after the visible light image semantic feature map and the infrared image semantic feature map pass through the global interactive attention fusion module, global interactive attention fusion will be mainly performed based on the spatial distribution of image semantic features. However, considering the difference in the source image semantic spatial distribution of the visible light image and the corrected infrared image themselves, the inconsistent image semantic feature distribution of the visible light image semantic feature map and the infrared image semantic feature map in the spatial distribution dimension of the feature matrix will also make the fused image semantic feature spatial distribution of the multi-source complementary semantic feature map show a certain local distribution representation independence.

[0081] Therefore, when the multi-source complementary semantic feature map is subjected to global mean pooling processing to obtain the multi-source complementary semantic feature matrix, since the mean processing along the channel further strengthens the feature distribution in the spatial dimension of the feature matrix, the independence of the local feature distribution representation of the multi-source complementary semantic feature matrix will be more significant, so that when the multi-source complementary semantic feature matrix passes through the AIGC-based image generator as a whole, the mapping convergence of the semantic probability space domain of the generated image will be difficult due to the independence of the local feature distribution of the multi-source complementary semantic feature matrix, thereby affecting the training speed of the generator and the image quality of the final generated image.

[0082] Based on this, the applicant of the present application performs clustering optimization on the multi-source complementary semantic feature matrix, that is, first clustering the eigenvalues ​​of the multi-source complementary semantic feature matrix, for example, clustering based on the distance between eigenvalues, and then optimizing based on the in-class and out-of-class representations of the clustered features.

[0083] Accordingly, in a specific example, the feature distribution optimization subunit is further used to: perform feature distribution clustering optimization on the multi-source complementary semantic feature matrix according to the following optimization formula to obtain the optimized multi-source complementary semantic feature matrix; wherein the optimization formula is:

[0084]

[0085] Wherein, f is each eigenvalue of the multi-source complementary semantic feature matrix, n is the number of feature sets corresponding to the multi-source complementary semantic feature matrix, k is the number of cluster features, represents a cluster feature set, and f' is each eigenvalue of the optimized multi-source complementary semantic feature matrix.

[0086] Specifically, by taking the in-class features and out-of-class features of the multi-source complementary semantic feature matrix as different instance roles to perform class instance description based on clustering proportion distribution, and introducing the clustering response history based on the in-class and out-of-class dynamic context, a coordinated global perspective is maintained on the in-class distribution and out-of-class distribution of the overall features of the multi-source complementary semantic feature matrix, so that the optimized feature clustering operation of the multi-source complementary semantic feature matrix can maintain a coherent and consistent response of the in-class and out-of-class features, so that the regression convergence path based on feature clustering remains consistent in the process of generating probability mapping, and the mapping convergence effect of the multi-source complementary semantic feature matrix for the generated image semantic probability space domain is improved, thereby improving the training speed of the generator and the image quality of the information fusion optimized image of the observation area finally generated.

[0087] Based on the above embodiments, see Figure 3FIG. 1 is a flowchart of a three-dimensional visual information fusion optimization method for image analysis in an embodiment of the present disclosure. For example, the three-dimensional visual information fusion optimization method for image analysis can be executed by a server, which can be Figure 1 The server 100 shown in FIG. Figure 3 As shown, according to the embodiment of the present disclosure, the three-dimensional visual information fusion optimization method for image analysis includes the steps of: S510, acquiring a visible light image and an infrared image of an observation area; S520, performing image feature extraction on the infrared image and the visible light image to obtain a visible light image semantic feature map and an infrared image semantic feature map; S530, fusing the visible light image semantic feature map and the infrared image semantic feature map through a global interactive attention fusion module to obtain a multi-source complementary semantic feature map; and, S540, generating an information fusion optimized image of the observation area based on the multi-source complementary semantic feature map.

[0088] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned three-dimensional visual information fusion optimization method for image analysis have been referred to above. Figure 2 The three-dimensional visual information fusion optimization system 800 for image analysis has been introduced in detail, and therefore, its repeated description will be omitted.

[0089] Figure 4 FIG. 1 is an application scenario diagram of a three-dimensional visual information fusion optimization system for image analysis according to an embodiment of the present disclosure. Figure 4 As shown, in this application scenario, first, a visible light image of the observation area is obtained (for example, Figure 4 D1) and infrared images (e.g., Figure 4 Then, the infrared image and the visible light image are input to a server (for example, Figure 4 In S) shown in , the server is capable of using the three-dimensional visual information fusion optimization algorithm for image analysis to process the infrared image and the visible light image to obtain an information fusion optimized image of the observation area.

[0090] Based on the above embodiments, another exemplary embodiment of an electronic device is also provided in the embodiments of the present disclosure. In some possible implementations, the electronic device in the embodiments of the present disclosure may include a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the steps of the three-dimensional visual information fusion optimization method for image analysis in the above embodiments when executing the program.

[0091] For example, the electronic device is disclosed in the present invention Figure 1Taking the server 100 in the example as an example, the processor in the electronic device is the processor 110 in the server 100, and the memory in the electronic device is the memory 120 in the server 100.

[0092] An embodiment of the present disclosure also provides a computer-readable storage medium. Figure 5 FIG. 1 is a schematic diagram of a computer-readable storage medium 1000 according to an embodiment of the present disclosure. Figure 5 As shown, the computer-readable storage medium 1000 stores computer-executable instructions 1001. When the computer-executable instructions 1001 are executed by the processor, the three-dimensional visual information fusion optimization method for image analysis according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc.

[0093] The embodiments of the present disclosure also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the three-dimensional visual information fusion optimization method for image analysis according to the embodiments of the present disclosure.

[0094] Those skilled in the art will appreciate that the contents disclosed in this disclosure may be subject to various modifications and improvements. For example, the various devices or components described above may be implemented by hardware, or by software, firmware, or a combination of some or all of the three.

[0095] In addition, although the present disclosure makes various references to certain units in the system according to embodiments of the present disclosure, any number of different units may be used and run on the client and / or server. The units are illustrative only, and different aspects of the systems and methods may use different units.

[0096] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of a software functional module. The present disclosure is not limited to any particular form of combination of hardware and software.

[0097] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or highly formal sense, unless explicitly defined as such herein.

[0098] The above is an explanation of the present disclosure and should not be considered as a limitation thereof. Although several exemplary embodiments of the present disclosure have been described, it will be readily understood by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure as defined in the claims. It should be understood that the above is an explanation of the present disclosure and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present disclosure is defined by the claims and their equivalents.

Claims

1. A three-dimensional visual information fusion optimization system for image analysis, characterized in that: include: A multi-source image acquisition module, used to acquire visible light images and infrared images of the observation area; An image feature extraction module, used for performing image feature extraction on the infrared image and the visible light image to obtain a visible light image semantic feature map and an infrared image semantic feature map; A multi-source information complementary fusion module, used for fusing the visible light image semantic feature map and the infrared image semantic feature map through a global interactive attention fusion module to obtain a multi-source complementary semantic feature map; as well as An image optimization module, used for generating an information fusion optimized image of the observation area based on the multi-source complementary semantic feature map; The image optimization module comprises: A feature structure balancing and regularizing unit, used for performing dimension reduction and optimization processing on the multi-source complementary semantic feature map to obtain an optimized multi-source complementary semantic feature matrix; and An image generating unit, configured to pass the optimized multi-source complementary semantic feature matrix through an AIGC-based image generator to obtain an information fusion optimized image of the observation area; The characteristic structure balance regularization unit comprises: A pooling dimension reduction subunit, used for performing global mean pooling processing on the multi-source complementary semantic feature map to obtain a multi-source complementary semantic feature matrix; and A feature distribution optimization subunit, used for performing feature distribution clustering optimization on the multi-source complementary semantic feature matrix to obtain the optimized multi-source complementary semantic feature matrix; The feature distribution optimization subunit is further used to: perform feature distribution clustering optimization on the multi-source complementary semantic feature matrix according to the following optimization formula to obtain the optimized multi-source complementary semantic feature matrix; wherein the optimization formula is: Wherein, f is each eigenvalue of the multi-source complementary semantic feature matrix, n is the number of feature sets corresponding to the multi-source complementary semantic feature matrix, k is the number of cluster features, represents a cluster feature set, and f' is each eigenvalue of the optimized multi-source complementary semantic feature matrix.

2. The three-dimensional visual information fusion optimization system for image analysis according to claim 1, characterized in that: The multi-source image acquisition module comprises: a visible light unit, configured to obtain the visible light image of the observation area collected by a visible light sensor carried by the aircraft; and The infrared unit is used to obtain the infrared image of the observation area collected by the infrared sensor carried by the aircraft.

3. The three-dimensional visual information fusion optimization system for image analysis according to claim 2, characterized in that: The image feature extraction module comprises: a balancing correction unit, configured to perform balancing correction on the infrared image based on prior information to obtain a corrected infrared image; and A feature encoding unit is used to perform feature encoding on the visible light image and the corrected infrared image using a deep learning network model to obtain a semantic feature map of the visible light image and a semantic feature map of the infrared image.

4. The three-dimensional visual information fusion optimization system for image analysis according to claim 3 is characterized in that: The equalization correction unit is used for: The infrared image is processed using the following balanced correction formula to obtain the corrected infrared image; wherein the balanced correction formula is: f θ =f a +u θ ⊙f b Among them, f a is the gray level corresponding to the minimum value of each pixel in the infrared image, f b is the gray level corresponding to the maximum value of each pixel in the infrared image, u θ is the θth value randomly sampled from a uniform distribution, f θ is the θth pixel in the corrected infrared image, and ⊙ represents point multiplication by position.

5. The three-dimensional visual information fusion optimization system for image analysis according to claim 4, characterized in that: The feature encoding unit is used to: Passing the visible light image through an image feature extractor based on a convolutional neural network model to obtain a semantic feature map of the visible light image; and The corrected infrared image is passed through the image feature extractor based on the convolutional neural network model to obtain the infrared image semantic feature map.

6. The three-dimensional visual information fusion optimization system for image analysis according to claim 5, characterized in that: The image feature extractor based on the convolutional neural network model includes an input layer, a convolution layer, an activation layer, a pooling layer and an output layer.

7. The three-dimensional visual information fusion optimization system for image analysis according to claim 6, characterized in that: The multi-source information complementary fusion module is used to: The visible light image semantic feature map and the infrared image semantic feature map are processed by the following information complementary fusion formula to obtain an information complementary representation feature vector; wherein the information complementary fusion formula is: Among them, U1 and U2 are the semantic feature map of the visible light image and the semantic feature map of the infrared image respectively, and U is a complementary fusion feature map. represents cascade processing, S c represents the cth eigenvalue in the information complementary characterization eigenvector, F gp (U c ) represents the global mean pooling process for the c-th feature matrix in the complementary fusion feature map, H and W are the height and width of the c-th feature matrix in the complementary fusion feature map, respectively, and U c (i, j) is the eigenvalue at position (i, j) in the c-th feature matrix of the complementary fusion feature map; Activating the information complementarity representation feature vector to obtain a condensed information complementarity representation feature vector; The visible light image semantic feature map, the infrared image semantic feature map and the condensed information complementary representation feature vector are processed using the following attention weight application formula to obtain the multi-source complementary semantic feature map; wherein the attention weight application formula is: a c +b c =1 Among them, a c and b c represents the weighted probability vectors of the semantic feature map U1 of the visible light image and the semantic feature map U2 of the infrared image, Acz represents the vector element of the c channel corresponding to the semantic feature map of the visible light image in the condensed information complementary representation feature vector, Bcz represents the vector element of the c channel corresponding to the semantic feature map of the infrared image in the condensed information complementary representation feature vector, F c is the multi-source complementary semantic feature map, Indicates cascade processing.

Images