Image reconstruction algorithm, image processing method, device, vehicle and storage medium

Abstract

The application discloses an image reconstruction algorithm, an image processing method, an equipment, a vehicle and a storage medium. The image processing method comprises the following steps: acquiring a visible light image and an auxiliary modal image to be fused; and fusing and reconstructing the visible light image and the auxiliary modal image by using an image reconstruction algorithm to obtain a target image, wherein the image reconstruction algorithm comprises a step of using a high-frequency attention mechanism and a Fourier convolution mechanism in a high-frequency attention block to complete a feature extraction task of the visible light image. In this way, the image reconstruction algorithm can quickly and effectively extract high-frequency information in the visible light image of various image qualities, so that the target image obtained by fusing and reconstructing the visible light image and the auxiliary modal image based on the effectively extracted high-frequency information has clearer features in the visible light image and features in the auxiliary modal image. Therefore, the application can ensure that high-quality images are obtained when shooting in various shooting environments.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image reconstruction algorithm, an image processing method, a computing device, a vehicle, and a computer-readable storage medium. Background Technology

[0002] Most unmanned or low-human-intervention intelligent devices require precise environmental perception capabilities during operation. Currently, the environmental perception capabilities of various intelligent devices are mainly achieved by acquiring high-quality environmental images through camera systems.

[0003] However, current smart device camera systems mainly rely on visible light. Therefore, in various harsh shooting environments that affect visibility (such as low-light environments, rainy and foggy environments, dusty environments, etc.), the image quality of the environmental images acquired by the camera system is poor, resulting in poor environmental perception capabilities of smart devices. Therefore, how to acquire higher quality images when shooting in various shooting environments is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide an image reconstruction algorithm, image processing method, computing device, vehicle, and computer-readable storage medium that can ensure the acquisition of superior images when shooting in various shooting environments.

[0005] To achieve the above objectives:

[0006] In a first aspect, embodiments of this application provide an image reconstruction algorithm, comprising: in a first branch model for processing visible light images, performing feature extraction on an initial visible light image acquired by a first encoder at the current scale stage through a high-frequency attention block and / or downsampling through a convolutional layer to obtain a visible light feature map at a specific scale corresponding to the current scale stage; the high-frequency attention block employs a high-frequency attention mechanism and a Fourier convolution mechanism to complete the feature extraction task; the initial visible light image is an initial feature map obtained after shallow feature extraction of the visible light image, or a feature map to be processed acquired by the first encoder at the previous scale stage; fusing the visible light feature map at the specific scale with an auxiliary feature map at the corresponding scale to obtain a first fused feature map at the specific scale; the auxiliary feature map at the corresponding scale is output by a second branch model for processing auxiliary modal images; obtaining a feature map to be processed at the specific scale based on the visible light feature map at the specific scale and / or the first fused feature map; entering the next scale stage of the first encoder based on the feature map to be processed at the specific scale, and / or using a first decoder in the first branch model for feature processing to obtain a final processing result, and reconstructing the image based on the final processing result to obtain a target image.

[0007] In one embodiment, the auxiliary modal image is one of an infrared image, a radar image, or a lidar point cloud image.

[0008] In one embodiment, the high-frequency attention block includes a frequency domain branch and a spatial branch.

[0009] In one embodiment, feature extraction is performed on the initial visible light image acquired by the first encoder at the current scale stage using high-frequency attention blocks, including:

[0010] In the frequency domain branch, high-frequency information of the initial visible light image is mined and transformed to the frequency domain using a Fast Fourier Transform (FFT) to globally update the frequency domain data corresponding to the initial visible light image. The updated frequency domain data is then transformed to the spatial domain using an Inverse Fast Fourier Transform (IFFT) to obtain the first spatial domain data. In the spatial branch, local information of the initial visible light image is extracted, and feature processing is performed on the local information in the spatial domain to learn the local structural features on the initial visible light image using a multi-scale learning approach. Channel adaptive processing is then performed to obtain the second spatial domain data. The first and second spatial domain data are then fused and the number of channels is adjusted to output the feature extraction information.

[0011] In one embodiment, the step of mining high-frequency information of the initial visible light image in the frequency domain branch includes: dividing the feature layer of the initial visible light image into a first feature layer and a second feature layer according to the number of channels of the initial visible light image, and inputting them into a first processing sub-branch and a second processing branch respectively; in the first processing sub-branch, performing local feature extraction on the first feature layer through a 3×3 convolutional layer, and performing feature processing using a 1×1 convolutional layer and the GELU activation function to obtain a first processing result; in the second processing branch, performing high-frequency feature extraction on the second feature layer through a max pooling layer, and performing feature processing using a 1×1 convolutional layer and the GELU activation function to obtain a second processing result; and aggregating the first processing result and the second processing result in the channel dimension to obtain high-frequency information.

[0012] In one embodiment, the steps of extracting local information from the initial visible light image in the spatial branch, performing feature processing on the local information in the spatial domain to learn local structural features on the initial visible light image in a multi-scale learning manner, and performing channel adaptive processing to obtain second spatial domain data include: extracting local information through a 3×3 convolutional layer, establishing long-distance dependencies using a 5×5 depthwise convolution and a 5×5 depthwise dilated convolution, adjusting the relationship between each feature layer through a 1×1 convolutional layer, and adding an SE module to adaptively capture key feature layer information to obtain second spatial domain data.

[0013] In one embodiment, the step of fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels to output feature extraction information includes: fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels through a 1×1 convolutional layer to obtain and output feature extraction information.

[0014] In one embodiment, the mathematical expression corresponding to the step of mining high-frequency information from the initial visible light image in the frequency domain branch is:

[0015]

[0016]

[0017]

[0018]

[0019] in, Indicates a convolutional layer. This represents the activation function. This indicates a max pooling operation. This indicates an operation that splices along the channel dimension.

[0020] In one embodiment, the mathematical expression for the steps of converting high-frequency information to the frequency domain using a fast Fourier transform to globally update the frequency domain data corresponding to the initial visible light image, and converting the updated frequency domain data to the spatial domain using an inverse fast Fourier transform to obtain the first spatial domain data, is as follows:

[0021]

[0022] in, Indicates a convolutional layer. Represents the Fast Fourier Transform. This represents the inverse Fourier transform.

[0023] In one embodiment, the steps of extracting local information using 3×3 convolutional layers, establishing long-range dependencies using 5×5 depthwise convolutions and 5×5 depthwise dilated convolutions, adjusting the relationships between feature layers using 1×1 convolutional layers, and adding an SE module to adaptively capture key feature layer information to obtain the second spatial domain data are expressed by the following mathematical expression:

[0024]

[0025] in, Indicates a convolutional layer. Represents depthwise convolution. This represents depthwise dilated convolution. This indicates the SE module.

[0026] In one embodiment, the mathematical expression corresponding to the step of fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels through a 1×1 convolutional layer to obtain and output feature extraction information is:

[0027]

[0028] in, This indicates a convolutional layer.

[0029] In one embodiment, feature processing is performed using a first decoder based on a feature map to be processed at a specific scale to obtain a final processing result. This includes: using the first decoder to concatenate the feature map to be processed at a specific scale and a visible light feature map of the corresponding scale output by the first encoder to obtain a concatenated feature map at a specific scale; performing feature enhancement and / or upsampling on the concatenated feature map at a specific scale through a fusion learning block and / or through a deconvolution layer to obtain a reconstructed feature map at a specific scale; fusing the reconstructed feature map at a specific scale with an auxiliary reconstructed feature map at a corresponding scale to obtain a second fused feature map at a specific scale, wherein the auxiliary reconstructed feature map at the corresponding scale is output by a second branch model; obtaining a reconstructed feature map to be processed at a specific scale based on the reconstructed feature map at a specific scale and / or the second fused feature map; and processing the reconstructed feature map to be processed at a specific scale in the first branch model to obtain the final processing result.

[0030] In one embodiment, the workflow of the second branch model acquiring auxiliary feature maps includes: in the second branch model, performing feature extraction on the initial auxiliary modality image acquired by the second encoder through residual groups and / or downsampling through convolutional layers to obtain an auxiliary feature map at a specific scale. The initial auxiliary modality image is an initial auxiliary feature map obtained after shallow feature extraction on the auxiliary modality image, or an auxiliary feature map acquired by the second encoder in the previous scale stage.

[0031] In one embodiment, the workflow of the second branch model to obtain the auxiliary reconstruction feature map includes: in the second branch model, obtaining the auxiliary feature map output by the second encoder from the second decoder, performing feature enhancement through residual groups and / or upsampling through deconvolution layers to obtain an auxiliary reconstruction feature map at a specific scale.

[0032] In one embodiment, the step of reconstructing an image based on the final processing result to obtain a target image includes: performing a linear projection on the final processing result to obtain a reconstructed image; and, based on the reconstructed image, using minimization... LAfter optimizing the network parameters using a 1-pixel loss, the target image is obtained.

[0033] Secondly, embodiments of this application provide an image processing method, including: acquiring a visible light image to be fused and an auxiliary modal image; fusing and reconstructing the visible light image and the auxiliary modal image using an image reconstruction algorithm to obtain a target image, wherein the image reconstruction algorithm includes the step of using a high-frequency attention mechanism and a Fourier convolution mechanism in a high-frequency attention block to complete the feature extraction task of the associated visible light image.

[0034] Thirdly, embodiments of this application provide a computing device, including: a processor and a memory storing a computer program, wherein when the processor runs the computer program, the steps of the image processing method described above are implemented.

[0035] Fourthly, embodiments of this application provide a vehicle equipped with the aforementioned computing device.

[0036] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the image processing method described above.

[0037] The image reconstruction algorithm provided in this application includes: in a first branch model for processing visible light images, the initial visible light image acquired by the first encoder at the current scale stage is subjected to feature extraction through a high-frequency attention block and / or downsampling through a convolutional layer to obtain a visible light feature map at a specific scale corresponding to the current scale stage. The high-frequency attention block uses a high-frequency attention mechanism and a Fourier convolution mechanism to complete the feature extraction task. The initial visible light image is an initial feature map obtained after shallow feature extraction of the visible light image, or a feature map to be processed acquired by the first encoder at the previous scale stage. The visible light feature map at the specific scale is fused with an auxiliary feature map at the corresponding scale to obtain a first fused feature map at the specific scale. The auxiliary feature map at the corresponding scale is output by a second branch model for processing auxiliary modal images. Based on the visible light feature map at the specific scale and / or the first fused feature map, a feature map to be processed at the specific scale is obtained. Based on the feature map to be processed at the specific scale, the algorithm enters the next scale stage of the first encoder and / or uses the first decoder in the first branch model to perform feature processing to obtain the final processing result. Based on the final processing result, image reconstruction is performed to obtain the target image. Thus, the technical solution of this application completes the feature extraction task through the high-frequency attention mechanism and Fourier convolution mechanism in the high-frequency attention block. It can quickly and effectively extract high-frequency information from visible light images of various image qualities. After determining the visible light feature image corresponding to the visible light image based on the effectively extracted high-frequency information, it can perform multi-scale deep feature fusion and image reconstruction based on the auxiliary feature map corresponding to the visible light feature image and the additionally acquired auxiliary modal image to obtain the target image. Moreover, the target image has clearer features from the visible light image and the auxiliary modal image. Therefore, the technical solution of this application can ensure the acquisition of high-quality images when shooting in various shooting environments.

[0038] The image processing method provided in this application includes: acquiring a visible light image and an auxiliary modal image to be fused; and fusing and reconstructing the visible light image and the auxiliary modal image using an image reconstruction algorithm to obtain a target image. The image reconstruction algorithm includes the step of using a high-frequency attention mechanism and a Fourier convolution mechanism in a high-frequency attention block to complete the feature extraction task of the associated visible light image. Thus, because the technical solution of this application utilizes the high-frequency attention mechanism and Fourier convolution mechanism in a high-frequency attention block to complete the feature extraction task of the associated visible light image, the image reconstruction algorithm can quickly and effectively extract high-frequency information from visible light images of various image qualities. Therefore, the target image obtained by fusing and reconstructing the visible light image and the auxiliary modal image based on the effectively extracted high-frequency information has clearer features from both the visible light image and the auxiliary modal image. Therefore, the technical solution of this application can ensure the acquisition of high-quality images when shooting in various shooting environments. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0040] Figure 1 This is a schematic diagram of the first process of the image reconstruction algorithm provided in the first embodiment of this application.

[0041] Figure 2 This is a schematic diagram of a high-frequency attention block as exemplified in the first embodiment of this application.

[0042] Figure 3 This is a schematic diagram of the second process of the image reconstruction algorithm provided in the first embodiment of this application.

[0043] Figure 4 This is a schematic diagram of the principle framework of the image reconstruction algorithm exemplified in this application.

[0044] Figure 5 This is a schematic diagram of the multi-source data fusion module in this application example.

[0045] Figure 6 This is a schematic diagram illustrating the framework of a camera system implementing various functions using an applied image reconstruction algorithm, as exemplified in this application.

[0046] Figure 7 This is a schematic flowchart of the image processing method provided in the first embodiment of this application.

[0047] Figure 8 This is a schematic diagram of the computing device provided in this application.

[0048] The realization of the objectives, functional features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0050] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0051] It should be understood that although the terms first, second, third, etc., may be used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if," as used herein, can be interpreted as "when," "when," or "in response to determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms "comprising," "including," indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" as used herein are to be interpreted as inclusive, or mean any one or any combination thereof. Therefore, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A, B, and C". Exceptions to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0052] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0053] It should be noted that step designations such as S11 and S12 are used in this document for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S12 first and then S11, etc., but these should all be within the protection scope of this application.

[0054] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0055] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0056] First Embodiment

[0057] See Figure 1 This application provides an image reconstruction algorithm according to its first embodiment. This method can be executed by a computing device provided in this embodiment, which can be implemented in software and / or hardware. The computing device may be, for example, a controller in a camera system.

[0058] This embodiment provides an image reconstruction algorithm, including the following steps (e.g., steps S11 to S14):

[0059] Step S11: In the first branch model for processing visible light images, the initial visible light image acquired by the first encoder at the current scale stage is used for feature extraction through a high-frequency attention block and / or downsampling through a convolutional layer to obtain a visible light feature map at a specific scale corresponding to the current scale stage. The high-frequency attention block uses a high-frequency attention mechanism and a Fourier convolution mechanism to complete the feature extraction task.

[0060] In one embodiment, the initial visible light image can be an initial feature map obtained after shallow feature extraction of the visible light image, or a feature map to be processed obtained by the first encoder in the previous scale stage.

[0061] In one embodiment, when the initial visible light image is an initial feature map obtained after shallow feature extraction of the visible light image, the process before step S11 may include: performing shallow feature extraction and image channel expansion on the visible light image through linear mapping to obtain the initial feature map. The technical solution of this embodiment, by performing shallow feature extraction and expanding the image channel number on the visible light image through linear mapping, helps to increase the dimensionality and richness of features while maintaining the same image size.

[0062] In one embodiment, the feature map can characterize various features extracted from the original input image, such as edges, textures, shapes, etc.

[0063] In one embodiment, before step S11, the process may include: constructing a first branch model for processing visible light images; and constructing a second branch model for processing auxiliary modal images.

[0064] In one embodiment, the first branch model for processing visible light images can be a first multi-scale image reconstruction model. The first multi-scale image reconstruction model includes a first encoder and a first decoder.

[0065] In one embodiment, in the first multi-scale image reconstruction model, the first encoder mainly extracts and outputs visible light feature maps of multiple scales by performing feature extraction through high-frequency attention blocks and / or downsampling through convolutional layers on the initial feature map corresponding to the visible light image.

[0066] In one embodiment, the first encoder's operation includes multiple scale stages. Each scale stage can extract and output a visible light feature map of a certain scale by using high-frequency attention blocks for feature extraction and / or by using convolutional layers for downsampling. Optionally, the visible light feature maps output by the first encoder at different scale stages have different scales.

[0067] In one embodiment, the final scaling stage of the first encoder can extract and output a visible light feature map of one scale using only high-frequency attention blocks for feature extraction.

[0068] In one embodiment, the first encoder can synchronously output the visible light feature map obtained at each scale stage to the multi-source data fusion module, so that the multi-source data fusion module can perform layer-by-layer processing to achieve the fusion of the visible light feature map and the auxiliary feature map of the same scale, which is used to feed back the fused feature map (i.e. the first fused feature map below) to the first encoder.

[0069] In one embodiment, in the first multi-scale image reconstruction model, the first decoder mainly concatenates the feature map of one scale output by the final scale stage of the first encoder with other feature maps of the corresponding scale to obtain the initial scale concatenated feature map. Then, the initial scale concatenated feature map is used to extract and output reconstructed feature maps of multiple scales by performing feature enhancement through fusion learning blocks and / or upsampling through deconvolution layers.

[0070] In one embodiment, the first decoder's operation also includes multiple scale stages. Each scale stage can extract and output a reconstructed feature map of a certain scale by using feature enhancement through fusion learning blocks and / or upsampling through deconvolutional layers. Optionally, the scales of the reconstructed feature maps output by the first decoder at different scale stages are different.

[0071] In one embodiment, the final scaling stage of the first decoder can extract and output a reconstructed feature map with the same scale as the initial feature map corresponding to the visible light image by simply performing feature enhancement through the fusion of learning blocks.

[0072] In one embodiment, the first decoder can output the reconstructed feature map obtained at each scale stage to the multi-source data fusion module, so that the multi-source data fusion module can perform layer-by-layer processing to achieve the fusion of the reconstructed feature map and the auxiliary reconstructed feature map of the same scale, which is then fed back to the first decoder as a fused feature map (i.e., the second fused feature map below).

[0073] In one embodiment, the number of scale stages of the first decoder is the same as the number of scale stages of the first encoder. This ensures that the scale of the reconstructed feature map output by the final scale stage of the first decoder is consistent with the scale of the initial feature map corresponding to the visible light image. Thus, the reconstructed feature map output by the final scale stage can contain the effective global features of the visible light image, thereby ensuring that the target image obtained based on the reconstructed feature map can retain the effective global features of the visible light image.

[0074] In one embodiment, the second branch model for processing the auxiliary modality image can be a second multi-scale image reconstruction model. The second multi-scale image reconstruction model includes a second encoder and a second decoder.

[0075] In one embodiment, the first multi-scale image reconstruction model and the second multi-scale image reconstruction model may be the same or different.

[0076] In one embodiment, in the second multi-scale image reconstruction model, the second encoder mainly extracts and outputs auxiliary feature maps of multiple scales by performing feature extraction through residual groups and / or downsampling through convolutional layers on the initial auxiliary feature maps corresponding to the auxiliary modal images.

[0077] In one embodiment, the method for obtaining the initial auxiliary feature map corresponding to the auxiliary modality image may include: performing shallow feature extraction and image channel expansion on the auxiliary modality image through linear mapping to obtain the initial auxiliary feature map. The technical solution of this embodiment, by performing shallow feature extraction and expanding the image channel number on the initial auxiliary feature map through linear mapping, helps to increase the dimensionality and richness of features while maintaining the same image size.

[0078] In one embodiment, the second encoder's operation includes multiple scale stages. Each scale stage can extract and output an auxiliary feature map of a certain scale by using residual groups for feature extraction and / or by using convolutional layers for downsampling. Optionally, the auxiliary feature maps output by the second encoder at different scale stages have different scales.

[0079] In one embodiment, each scale of the auxiliary feature map in this embodiment can correspond to a visible light feature map of a certain scale output by the first encoder.

[0080] In one embodiment, the final scaling stage of the second encoder can extract and output an auxiliary feature map of one scale by using only residual groups for feature extraction.

[0081] In one embodiment, the second encoder can synchronously output the auxiliary feature map obtained at each scale stage to the multi-source data fusion module, so that the multi-source data fusion module can perform layer-by-layer processing to achieve the fusion of the visible light feature map and the auxiliary feature map of the same scale, which is used to feed back the fused feature map (i.e. the first fused feature map below) to the first encoder.

[0082] In one embodiment, in the second multi-scale image reconstruction model, the second decoder mainly extracts and outputs auxiliary reconstructed feature maps of multiple scales from the one-scale auxiliary feature map output by the second encoder in the final scale stage by performing feature enhancement through residual groups and / or upsampling through deconvolution layers.

[0083] In one embodiment, the second decoder also includes multiple scale stages. Each scale stage can extract and output an auxiliary reconstructed feature map of a certain scale by performing feature enhancement through residual groups and / or upsampling through deconvolutional layers. Optionally, the scales of the reconstructed feature maps output by the second decoder at different scale stages are different.

[0084] In one embodiment, the second decoder can output the auxiliary reconstruction feature map obtained at each scale stage to the multi-source data fusion module, so that the multi-source data fusion module can perform layer-by-layer processing to achieve the fusion of the reconstruction feature map and the auxiliary reconstruction feature map at the same scale, which is then fed back to the first decoder as a fused feature map (i.e., the second fused feature map below).

[0085] In one embodiment, the number of scale stages in the second decoder is at least one less than the number of scale stages in the second encoder. Thus, the auxiliary reconstruction feature map output by the final scale stage of the second decoder can characterize the key features of the auxiliary modality image (i.e., the global features of the non-auxiliary modality image). Therefore, the technical solution of this embodiment can achieve image reconstruction using only the key features of the auxiliary modality image, so that the target image obtained after image reconstruction can represent the effective global features of the visible light image, as well as all or part of the key features of the auxiliary modality image.

[0086] In one embodiment, a multi-scale image reconstruction model mainly refers to a model that can simultaneously process and utilize features at different scales during the image reconstruction process.

[0087] In one implementation, the high-frequency attention mechanism (HAB) primarily adaptively focuses on high-frequency information in the image. Optionally, the high-frequency information typically includes details and edge information of the image, which is crucial for reconstructing high-resolution images.

[0088] In one implementation, the Fourier convolution mechanism includes Fast Fourier Transform (FFT) and Inverse Fourier Transform (IFFT). Specifically, the combination of FFT and IFFT in the Fourier convolution mechanism with a High-Frequency Attention (HAB) mechanism is primarily used to enhance the extraction of high-frequency information. This combined strategy helps the model capture and utilize key information in the feature maps, improving model performance and accuracy while simplifying model design and optimization.

[0089] In one embodiment, the high-frequency attention block can effectively extract high-frequency information from visible light images of various image qualities, such as effectively extracting hidden high-frequency information from blurred visible light images or their corresponding feature maps. Therefore, when the technical solution of this embodiment reconstructs a visible light image based on the effectively extracted high-frequency information, it can obtain a clearer visible light image.

[0090] In one embodiment, see Figure 2 High-frequency attention blocks include frequency domain branches and spatial branches.

[0091] In one embodiment, feature extraction is performed on the initial visible light image acquired by the first encoder at the current scale stage using high-frequency attention blocks, including:

[0092] In the frequency domain branch, high-frequency information of the initial visible light image is mined and transformed to the frequency domain using a Fast Fourier Transform (FFT) to globally update the frequency domain data corresponding to the initial visible light image. The updated frequency domain data is then transformed to the spatial domain using an Inverse Fast Fourier Transform (IFFT) to obtain the first spatial domain data. In the spatial branch, local information of the initial visible light image is extracted, and feature processing is performed on the local information in the spatial domain to learn the local structural features on the initial visible light image using a multi-scale learning approach. Channel adaptive processing is then performed to obtain the second spatial domain data. The first and second spatial domain data are then fused and the number of channels is adjusted to output the feature extraction information.

[0093] In one embodiment, see Figure 2 In the frequency domain branch, the step of mining high-frequency information from the initial visible light image may include: dividing the feature layer of the initial visible light image into a first feature layer and a second feature layer according to the number of channels, and inputting them into a first processing sub-branch and a second processing branch, respectively; in the first processing sub-branch, performing local feature extraction on the first feature layer using a 3×3 convolutional layer (Conv 3×3), and performing feature processing using a 1×1 convolutional layer (Conv 1×1) and the GELU activation function to obtain a first processing result; in the second processing branch, performing high-frequency feature extraction on the second feature layer using a max pooling layer, and performing feature processing using a 1×1 convolutional layer (Conv 1×1) and the GELU activation function to obtain a second processing result; and aggregating the first processing result and the second processing result along the channel dimension to obtain high-frequency information.

[0094] In one embodiment, the mathematical expression corresponding to the step of mining high-frequency information from the initial visible light image in the frequency domain branch is:

[0095]

[0096]

[0097]

[0098]

[0099] in, Indicates a convolutional layer. This represents the activation function. This indicates a max pooling operation. This indicates an operation that splices along the channel dimension.

[0100] in, The feature layer representing the initial visible light image. This can represent the first feature layer. It can represent the second feature layer. This indicates the first processing result. This indicates the second processing result. It represents high-frequency information.

[0101] In one embodiment, the mathematical expression for the steps of converting high-frequency information to the frequency domain using a fast Fourier transform to globally update the frequency domain data corresponding to the initial visible light image, and converting the updated frequency domain data to the spatial domain using an inverse fast Fourier transform to obtain the first spatial domain data, is as follows:

[0102]

[0103] in, Indicates a convolutional layer. Represents the Fast Fourier Transform. This represents the inverse Fourier transform.

[0104] in, Represents high-frequency information. This represents the data in the first spatial domain.

[0105] In one embodiment, see Figure 2 In the spatial branch, local information of the initial visible light image is extracted, and feature processing is performed on the local information in the spatial domain to learn the local structural features on the initial visible light image in a multi-scale learning manner, and channel adaptive processing is performed to obtain the second spatial domain data. The steps may include: extracting local information through a 3×3 convolutional layer (Conv 3×3), establishing long-distance dependencies using a 5×5 depthwise convolution (depthwise convolution Conv 5×5) and a 5×5 depthwise dilated convolution (depthwise dilated convolution Conv 5×5), adjusting the relationship between each feature layer through a 1×1 convolutional layer (Conv 1×1), and adding an SE module to adaptively capture key feature layer information (channel adaptive) to obtain the second spatial domain data.

[0106] In one embodiment, the steps of extracting local information using 3×3 convolutional layers, establishing long-range dependencies using 5×5 depthwise convolutions and 5×5 depthwise dilated convolutions, adjusting the relationships between feature layers using 1×1 convolutional layers, and adding an SE module to adaptively capture key feature layer information to obtain the second spatial domain data are expressed by the following mathematical expression:

[0107]

[0108] in, Indicates a convolutional layer. Represents depthwise convolution. This represents depthwise dilated convolution. This indicates the SE module.

[0109] in, The feature layer representing the initial visible light image. This represents data from the second spatial domain.

[0110] In one embodiment, see Figure 2 The steps of fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels to output feature extraction information include: fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels through a 1×1 convolutional layer (Conv 1×1) to obtain and output feature extraction information.

[0111] In one embodiment, the feature extraction information can be a visible light feature map at a specific scale.

[0112] In one embodiment, the mathematical expression corresponding to the step of fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels through a 1×1 convolutional layer to obtain and output feature extraction information is:

[0113]

[0114] in, Indicates a convolutional layer. Represents data in the second spatial domain. Represents the data in the first spatial domain. This represents the information extracted from features.

[0115] In one embodiment, the effect of downsampling through convolutional layers includes, but is not limited to, at least one of the following:

[0116] (1) Reduce computational cost and prevent overfitting: Downsampling reduces the computational cost of the model by decreasing the size of the feature maps, which helps improve the model's running efficiency. At the same time, downsampling can reduce the number of model parameters, thereby reducing the risk of overfitting;

[0117] (2) Increase receptive field: Downsampling can increase the receptive field of subsequent layers, enabling the model to capture more global information, which helps the model better understand the global structure and content of the image;

[0118] (3) Preserving spatial information: When using convolutional layers for downsampling, the convolution operation can preserve the spatial structure and location information of the input features, which helps to maintain the spatial consistency of the image in the subsequent image reconstruction process.

[0119] Step S12: Perform feature fusion between the visible light feature map at a specific scale and the auxiliary feature map at the corresponding scale to obtain the first fused feature map at the specific scale. The auxiliary feature map at the corresponding scale is output by the second branch model that processes the auxiliary modality image.

[0120] In one embodiment, the auxiliary modal image can be one of the following images other than visible light images: infrared images, radar images, lidar point cloud images, etc. Optionally, the auxiliary modal image in this embodiment can be an infrared image.

[0121] In one embodiment, the workflow of the second branch model acquiring auxiliary feature maps may include: in the second branch model, performing feature extraction on the initial image of the auxiliary modality acquired by the second encoder through residual groups and / or downsampling through convolutional layers to obtain auxiliary feature maps of a specific scale.

[0122] In one embodiment, the initial auxiliary modality image can be an initial auxiliary feature map obtained after shallow feature extraction of the auxiliary modality image, or an auxiliary feature map obtained by the second encoder in the previous scale stage.

[0123] In one embodiment, when the initial auxiliary modality image is an initial auxiliary feature map obtained after shallow feature extraction of the auxiliary modality image, in the second branch model, before the step of extracting features from the initial auxiliary modality image acquired by the second encoder through residual groups and / or downsampling through convolutional layers to obtain an auxiliary feature map of a specific scale, the model may include: performing shallow feature extraction and image channel expansion on the auxiliary modality image through linear mapping to obtain the initial auxiliary feature map. The technical solution of this embodiment, by performing shallow feature extraction and expanding the image channel number on the auxiliary modality image through linear mapping, helps to increase the dimensionality and richness of features while maintaining the image size.

[0124] In one embodiment, the second branch model for processing auxiliary modal images can be a multi-scale image reconstruction model.

[0125] Step S13: Obtain the feature map to be processed at a specific scale based on the visible light feature map at a specific scale and / or the first fused feature map.

[0126] In one embodiment, step S13: Based on the visible light feature map at a specific scale and / or the first fused feature map, obtain a feature map to be processed at a specific scale, including but not limited to one of the following:

[0127] The visible light feature map at a specific scale and the first fused feature map are fused again to obtain the feature map to be processed at a specific scale.

[0128] The first fused feature map at a specific scale is used as the feature map to be processed at that specific scale.

[0129] Step S14: Based on the feature map to be processed at a specific scale, proceed to the next scale stage of the first encoder, and / or use the first decoder in the first branch model to perform feature processing to obtain the final processing result, and perform image reconstruction based on the final processing result to obtain the target image.

[0130] In one embodiment, the first encoder may include multiple scale stages, each scale stage acquiring or outputting a feature map to be processed at a different scale.

[0131] In one embodiment, feature processing using a first decoder to obtain a final processing result based on a feature map to be processed at a specific scale may include: using the first decoder to concatenate the feature map to be processed at a specific scale and a visible light feature map of the corresponding scale output by the first encoder to obtain a concatenated feature map at a specific scale; performing feature enhancement and / or upsampling on the concatenated feature map at a specific scale through a fusion learning block and / or through a deconvolution layer to obtain a reconstructed feature map at a specific scale; fusing the reconstructed feature map at a specific scale with an auxiliary reconstructed feature map at a corresponding scale to obtain a second fused feature map at a specific scale, wherein the auxiliary reconstructed feature map at the corresponding scale is output by the second branch model; obtaining a reconstructed feature map to be processed at a specific scale based on the reconstructed feature map at a specific scale and / or the second fused feature map; and processing the reconstructed feature map to be processed at a specific scale in the first branch model to obtain the final processing result.

[0132] In one embodiment, the workflow of the second branch model to obtain the auxiliary reconstruction feature map may include: in the second branch model, obtaining the auxiliary feature map output by the second encoder from the second decoder, performing feature enhancement through residual groups and / or upsampling through deconvolution layers to obtain an auxiliary reconstruction feature map at a specific scale.

[0133] In one embodiment, the step of reconstructing the image based on the final processing result to obtain the target image may include: performing a linear projection on the final processing result to obtain a reconstructed image; and, based on the reconstructed image, using minimization... L After optimizing the network parameters using a 1-pixel loss, the target image is obtained. Thus, the technical solution of this embodiment can achieve high-quality image reconstruction, making the target image possess clearer effective global features from the visible light image and key features from the auxiliary modal image.

[0134] The image reconstruction algorithm provided in this embodiment can be applied to the camera system of intelligent devices to enhance the environmental perception capabilities of these devices, enabling them to operate more efficiently with minimal or no human intervention. Furthermore, when applied to the camera system of an intelligent device, the image reconstruction algorithm allows the camera system to output high-quality images for use by downstream task modules of the intelligent device. For example, a path planning module can plan a path based on the image, and a target recognition module can identify characteristic scenes or target objects based on the image.

[0135] In one embodiment, the intelligent device can be a mining vehicle, and the image reconstruction algorithm can be applied to the camera system of the mining vehicle. The high-quality images output by the camera system can support mine environment monitoring and the functions of the vehicle's task modules. The functions of the vehicle task modules include, for example, the driving path planning and obstacle avoidance functions of the autonomous driving module; the scene or target object recognition function of the target recognition module; and the depth estimation function of the depth estimation module.

[0136] In one embodiment, the camera system of the mining vehicle can support the simultaneous capture of visible light and infrared images. Thus, the image reconstruction algorithm applied to the mining vehicle's camera system can utilize not only visible light images but also infrared images as auxiliary modal images for image reconstruction. This allows the mining vehicle's camera system to output a target image that possesses not only the effective global features of the visible light image but also the key features of the infrared image. Therefore, the image reconstruction algorithm applied to the mining vehicle's camera system can leverage the strong thermal information capture capability, strong penetration capability, and lack of light-related effects of infrared images to supplement feature information that is difficult to perceive in visible light images under various environments (such as mining vehicles at night or in foggy conditions), thereby reconstructing high-quality images. Furthermore, the multi-source data fusion module in the image reconstruction algorithm can interactively fuse features extracted from visible light and infrared images, alleviating the problem of low image quality and inability to provide important information caused by the extreme dependence of visible light images on light sources and their weak penetration capability. This ensures that the image output by the mining vehicle's camera system accurately and effectively reflects features such as roads, vehicles, personnel, and depth.

[0137] This embodiment provides an image reconstruction algorithm, including: Step S11: In the first branch model for processing visible light images, the initial visible light image acquired by the first encoder at the current scale stage is subjected to feature extraction through a high-frequency attention block and / or downsampling through a convolutional layer to obtain a visible light feature map at a specific scale corresponding to the current scale stage. The high-frequency attention block uses a high-frequency attention mechanism and a Fourier convolution mechanism to complete the feature extraction task; Step S12: The visible light feature map at the specific scale is fused with the auxiliary feature map at the corresponding scale to obtain a first fused feature map at the specific scale. The auxiliary feature map at the corresponding scale is output by the second branch model for processing auxiliary modal images; Step S13: Based on the visible light feature map at the specific scale and / or the first fused feature map, a feature map to be processed at the specific scale is obtained; Step S14: Based on the feature map to be processed at the specific scale, the algorithm enters the next scale stage of the first encoder and / or uses the first decoder in the first branch model to perform feature processing to obtain the final processing result, and performs image reconstruction based on the final processing result to obtain the target image.

[0138] Based on the same inventive concept as the foregoing embodiments, the following example illustrates a specific application scenario of an image reconstruction algorithm applied to a camera system of a mining vehicle, for reference:

[0139] In open-pit mining, a semi-continuous mining process, such as single-bucket excavator-mining vehicle-crushing station-belt conveyor, is currently widely used. The transportation efficiency of mining vehicles has a significant impact on the progress of mining operations. Due to the complex working conditions in mining areas, dust is frequently generated on roads, visibility is poor at night, and mining and blasting operations lead to dust storms. In rainy or foggy weather, roads become muddy and difficult to see. In these harsh environments, the perception of road conditions by manned mining vehicle drivers is significantly reduced. Therefore, mining vehicles can be equipped with camera systems and other additional sensing equipment to monitor the environment and improve drivers' perception of road conditions. Alternatively, in these harsh environments, unmanned or minimally manned mining vehicles can achieve environmental perception through camera systems and other additional sensing equipment, enabling intelligent operation and improving the economic efficiency of mine output.

[0140] See Figure 3 In this example, the camera system in the mining vehicle can simultaneously capture visible light and infrared images, and the image reconstruction algorithm in the camera system of the mining vehicle includes the following steps:

[0141] S100: Keeping the size of the captured visible light image unchanged, shallow feature extraction and image channel expansion are performed on the captured visible light image through linear mapping to obtain an initial feature map.

[0142] S101: Keeping the size of the captured infrared image unchanged, shallow feature extraction and image channel expansion are performed on the captured visible light image through linear mapping to obtain an initial auxiliary feature map.

[0143] S102: In the visible light image branch, the encoder uses high-frequency attention blocks to extract features from the initial feature map, and then performs downsampling through convolutional layers. The output results of each scale stage are synchronously input into the multi-source data fusion module for layer-by-layer processing.

[0144] Optionally, the encoder outputs results at different scale stages with varying scales.

[0145] Optionally, the visible light image branch can be a first multi-scale image reconstruction model that includes an encoder and a decoder.

[0146] S103: In the visible light image branch, the feature maps input from different scale stages of the decoder are concatenated with the feature maps output from the encoder at the corresponding scale. Then, feature enhancement is performed through a fusion learning block, followed by upsampling through deconvolution. The output results of each scale stage are synchronously input into the multi-source data fusion module for layer-by-layer processing.

[0147] Optionally, the number of scale stages in the decoder in the visible light image branch is consistent with the number of scale stages in the encoder.

[0148] S104: In the infrared image auxiliary branch, the residual group (RG) in the encoder is used to extract features from the initial auxiliary feature map, and then downsampled through the convolutional layer. The output results of each scale stage are synchronously input into the multi-source data fusion module for layer-by-layer processing.

[0149] Optionally, the encoder outputs results at different scale stages with varying scales.

[0150] Optionally, the infrared image-assisted branch can be a second multi-scale image reconstruction model that includes an encoder and a decoder.

[0151] S105: In the infrared image auxiliary branch, the decoder performs feature enhancement through residual groups and then uses deconvolution for upsampling. The output results of each scale stage are synchronously input into the multi-source data fusion module for layer-by-layer processing.

[0152] Optionally, in the infrared image-assisted branch, the number of decoder scale stages is 1 less than the number of encoder scale stages.

[0153] S106: For the multi-source data fusion module, receive the output results of the corresponding stages in the visible light image branch and the infrared image auxiliary branch, and perform interactive fusion. The processed features are then fused with the output features of the visible light image branch again, and then sent to the next scale stage of the encoder of the visible light image branch or the next scale stage of the decoder of the visible light image branch.

[0154] S107: Perform linear projection on the feature map of the final scale stage of the decoder in the visible light image branch, reconstruct the decoder output, and form a reconstructed image.

[0155] S108: Based on the reconstructed image and minimization L After optimizing the network parameters using a 1-pixel loss, the target image is obtained.

[0156] Optionally, the network parameters can be optimized by minimizing the L1 pixel loss, and the loss can be calculated by comparing the reconstructed image with the real image.

[0157] Optionally, see Figure 4 This embodiment illustrates the image reconstruction algorithm by using a case study where both the decoder and encoder have 3 scale stages in the visible light image branch, and the encoder has 3 scale stages and the decoder has 2 scale stages in the infrared image auxiliary branch.

[0158] Optionally, the specific process of the encoder in the visible light image branch includes:

[0159]

[0160]

[0161]

[0162]

[0163]

[0164] in, Represents the input visible light image The initial feature map obtained after shallow feature extraction. This indicates a High Frequency Attention Block (HAB). or Indicates a downsampling block (e.g., output block) The corresponding downsampling block and output block at the scale stage (corresponding downsampling block in the scale stage). This represents the output blocks of the encoder at each scale stage (see...). Figure 4 In to ), This represents the output blocks of the multi-source data fusion module at each scale stage of the encoder and decoder (see...). Figure 4 In to ), The input block for the next stage in the decoder and encoder (see...) Figure 4 In to ), This represents the direct output block of the encoder for the corresponding scale stage of the decoder (see...). Figure 4 In to ).

[0165] Optionally, the decoder in the visible light image branch consists of a fusion learning block (CLB) and an upsampling module. The fusion learning block comprises two key parts: the CLB and a convolutional layer. To enable the model to directly utilize feature information from the original input feature maps, the fusion learning block at each scale stage receives two inputs: the encoder directly outputs the feature map at the corresponding scale stage and the feature map from the previous scale stage after deconvolution. A compensation mechanism reduces information loss during computation, and this fusion strategy improves the model's fitting ability. After receiving multiple inputs, they are stacked along the channel dimension, then feature processing is performed using the CLB, followed by adjusting the number of channels through convolutional layers to promote information fusion, and finally, deconvolution is used to restore the feature map size. The specific process of the decoder in the visible light image branch includes:

[0166]

[0167]

[0168]

[0169]

[0170] in, Indicates CLB, This represents a deconvolution module (e.g., the output block in a decoder). The corresponding deconvolution module at the scaling stage and the output block in the decoder (corresponding deconvolution module at the scale stage). This represents the output block in the decoder.

[0171] Optionally, the specific process of the encoder in the infrared image-assisted branch includes:

[0172]

[0173]

[0174] in, From the input infrared image The initial auxiliary feature map obtained after shallow feature extraction. Represents the residual group (RG). Indicates the downsampling block (output block) The corresponding downsampling block and output block at the scale stage (corresponding downsampling block in the scale stage). This represents the output blocks of the encoder at each scale stage (see...). Figure 4 In to ), This represents the output block of the multi-source data fusion module connected to each scale stage of the encoder and decoder (see...). Figure 4 In to ).

[0175] Optionally, the residual group (RG) consists of multiple stacked residual channel attention blocks (RCAB) and a convolutional layer, as expressed by the formula:

[0176]

[0177]

[0178] in, Indicates a convolutional layer. Indicates the first N Each residual channel attention block This represents the channel attention module, and ReLU represents the activation function.

[0179] Optionally, the decoder in the infrared image auxiliary branch consists of a residual group (RG) and an upsampling module. First, feature processing is performed using the residual group (RG), and then deconvolution is used to restore the image size. The specific process of the decoder in the infrared image auxiliary branch includes:

[0180]

[0181] in, Represents the residual group (RG). This represents a deconvolution module (e.g., the output block in a decoder). The corresponding deconvolution module at the scaling stage and the output block in the decoder (corresponding deconvolution module at the scale stage). This represents the output block in the decoder.

[0182] Optionally, see Figure 5 The specific process of the multi-source data fusion module includes:

[0183]

[0184]

[0185]

[0186] in, This represents a convolutional layer, ReLU represents the activation function, Concat[] represents the feature concatenation operation, and Norm represents normalization. In the visible light branch, the encoder and decoder are connected to the output block of the multi-source data fusion module. This represents the output block of the multi-source data fusion module connected to the encoder and decoder in the infrared image auxiliary branch. Then, the mean and standard deviation of each channel in the target contrast feature map are calculated:

[0187]

[0188]

[0189] in, and This indicates calculation in the channel domain. P The mean and standard deviation, H and W express P The height and width, then element-wise... Q Add to the mean, W Adding it to the standard deviation yields the feature map. L and J :

[0190]

[0191]

[0192] Will J Compared with the instantiated auxiliary contrast feature map Multiply to get Z , feature map L Add element by element Z In the process, the final matching feature map is obtained. M :

[0193]

[0194]

[0195] Subsequently, the mapped and matched feature map M and the target contrast feature map P are aggregated through continuous upsampling and downsampling residual compensation:

[0196]

[0197]

[0198]

[0199]

[0200]

[0201] in, This represents a convolutional layer, ReLU represents the activation function, and Concat[] represents the feature concatenation operation. This indicates a transposed convolutional layer.

[0202] Optionally, L The formula for 1-pixel loss is expressed as:

[0203]

[0204] The technical solution in this example can quickly and accurately extract hidden high-frequency information from blurred visible light images when mining vehicles face numerous challenges in the mining area road environment, such as the extreme harsh environment of high dust, mud, and nighttime operation, which causes visible light images to become blurred. This is achieved by using high-frequency attention blocks in the image reconstruction algorithm to extract hidden high-frequency information. The enhanced image obtained by the subsequent image reconstruction process based on the extracted high-frequency information facilitates accurate perception of the mining area environment, thereby ensuring the accuracy and effectiveness of subsequent analysis and decision-making using the enhanced image.

[0205] In this example, the image reconstruction algorithm uses infrared images as auxiliary modal information in addition to visible light images. Leveraging the strong thermal information capture, penetrating power, and lack of light-related limitations of infrared images, it supplements the image with feature information that is difficult to perceive in visible light images, thus enhancing the image for mining vehicles in harsh environments such as nighttime and foggy conditions. Furthermore, the multi-source data fusion module in the image reconstruction algorithm interactively fuses features extracted from visible light and infrared images, alleviating the problem of low image quality and inability to provide important information caused by the extreme dependence of visible light images on light sources and their weak penetrating power. This ensures that the enhanced image obtained after the image reconstruction process accurately and effectively reflects features such as roads, vehicles, people, and depth.

[0206] This embodiment illustrates a framework for implementing various functions in a mining vehicle based on a camera system using applied image reconstruction algorithms. For example, see [link to example]. Figure 6 Visible light and infrared images are captured by the visible light camera and infrared camera in the camera system, respectively. The visible light and infrared images are fused and reconstructed by the image reconstruction algorithm to obtain an enhanced image. The enhanced image is fed back to the monitoring equipment of the mining vehicle to realize the real-time display of the mine road conditions, and the enhanced image is input into various functional algorithm models to realize functions such as obstacle avoidance, recognition, and depth estimation.

[0207] The technical solution in this example can perform an image reconstruction strategy that fuses and deblurs multi-source images of a mine taken in harsh environments. This strategy utilizes the characteristics of infrared images, such as thermal information capture, strong penetration ability, and immunity to light effects, to fuse them with visible light images as an auxiliary mode. By combining the high resolution of visible light images, a complete and clear target image (enhanced image) can be effectively reconstructed. Based on this target image, mining vehicles can have more reliable environmental perception capabilities in harsh environments such as night and fog, and it can also provide strong support for subsequent downstream tasks such as image recognition and depth estimation.

[0208] The high-frequency attention block in the image reconstruction algorithm in this example includes a frequency domain branch and a spatial branch. In the frequency domain branch, high-frequency details are mined and the global values ​​of the image are updated using Fourier convolution, which improves the model's utilization of global information. In the spatial branch, information perception is achieved by using convolutional blocks with different receptive fields, which promotes the complementary fusion of multi-scale information, thereby improving the model's fitting ability and learning a more reasonable mapping relationship.

[0209] In the image reconstruction algorithm of this example, the fusion module for infrared and visible light images (i.e., the multi-source data fusion module) can extract information such as the mean and variance of the infrared image and perform interactive calculations with the visible light image, ultimately achieving complementary fusion between different modal information.

[0210] Second Embodiment

[0211] See Figure 7 This is a second embodiment of an image processing method provided in this application. The method can be executed by a computing device provided in this application, which can be implemented in software and / or hardware.

[0212] This embodiment provides an image processing method, including the following steps (e.g., steps S21 to S22):

[0213] Step S21: Obtain the visible light image and auxiliary modal image to be fused.

[0214] Step S22: Use an image reconstruction algorithm to fuse and reconstruct the visible light image and the auxiliary modality image to obtain the target image. The image reconstruction algorithm includes the steps of using the high-frequency attention mechanism and Fourier convolution mechanism in the high-frequency attention block to complete the feature extraction task of the associated visible light image.

[0215] In one embodiment, the specific implementation of the image reconstruction algorithm or high-frequency attention block in this embodiment can refer to the image reconstruction algorithm provided in the first embodiment, and will not be repeated here.

[0216] In this embodiment, the high-frequency attention mechanism and Fourier convolution mechanism in the high-frequency attention block are used to complete the feature extraction task of the associated visible light image. Therefore, the image reconstruction algorithm can quickly and effectively extract high-frequency information from visible light images of various image qualities. The target image obtained by fusing and reconstructing the visible light image and the auxiliary modal image based on the effectively extracted high-frequency information has clearer features from both the visible light image and the auxiliary modal image. Therefore, the technical solution of this embodiment can ensure the acquisition of high-quality images when shooting under various conditions.

[0217] Based on the same inventive concept as the foregoing embodiments, this application provides a computing device, such as... Figure 8 As shown, the device includes: a processor 310 and a memory 311 storing a computer program; wherein, Figure 8 The processor 310 shown in the diagram does not indicate that there is only one processor 310, but only indicates the positional relationship of the processor 310 relative to other devices. In practical applications, there can be one or more processors 310; similarly, Figure 8 The memory 311 shown in the diagram has the same meaning, that is, it is only used to indicate the positional relationship of memory 311 relative to other devices. In practical applications, there can be one or more memories 311. When the processor 310 runs the computer program, it implements the image reconstruction algorithm or image processing method applied to the above-mentioned device.

[0218] The device may also include at least one network interface 312. The various components of the device are coupled together via a bus system 313. It is understood that the bus system 313 is used to implement communication between these components. In addition to a data bus, the bus system 313 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 8 The general designated all buses as Bus System 313.

[0219] The memory 311 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 311 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0220] The memory 311 in this embodiment is used to store various types of data to support the operation of the device. Examples of this data include any computer programs used to operate on the device, such as operating systems and applications. The operating system includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic services and handle hardware-based tasks. Applications can include various applications, such as media players and browsers, used to implement various application services. Here, the program implementing the image reconstruction algorithm or image processing method provided in this application can be included in the application.

[0221] This application provides engineering machinery equipped with the aforementioned computing device. The engineering machinery can be a vehicle, etc. Optionally, the vehicle can be an engineering vehicle that frequently faces harsh environments, such as a mining vehicle.

[0222] The construction machinery in this embodiment includes traditional construction machinery vehicles, as well as new energy vehicles used in the construction machinery field, such as new energy mixer trucks, new energy pump trucks, and new energy excavators. In addition, the construction machinery vehicles in this embodiment are also intelligent connected vehicles. Construction machinery vehicles include sensing / perception systems, communication systems, etc. The in-vehicle sensing / perception system collects vehicle operation data and information about the vehicle's surrounding environment, and the communication system enables network connections with other vehicles and the cloud. The collected vehicle operation data and information about the vehicle's surrounding environment are shared with the cloud and other authorized vehicles to achieve data sharing, remote analysis, intelligent driving, and other operations.

[0223] Based on the same inventive concept as the foregoing embodiments, this embodiment also provides a computer-readable storage medium storing a computer program. The computer-readable storage medium can be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc. When the computer program stored in the computer-readable storage medium is executed by a processor, it implements the above-mentioned image reconstruction algorithm or image processing method. For the specific steps implemented when the computer program is executed by the processor, please refer to [reference needed]. Figure 1 or Figure 7 The description of the illustrated embodiments will not be repeated here.

[0224] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0225] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0226] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image reconstruction algorithm, characterized in that, include: In the first branch model for processing visible light images, the initial visible light image acquired by the first encoder at the current scale stage is subjected to feature extraction through a high-frequency attention block to obtain a visible light feature map at a specific scale corresponding to the current scale stage. The high-frequency attention block uses a high-frequency attention mechanism and a Fourier convolution mechanism to complete the feature extraction task. The initial visible light image is the initial feature map obtained after shallow feature extraction of the visible light image, or the feature map to be processed acquired by the first encoder at the previous scale stage. The visible light feature map at the specific scale is fused with the auxiliary feature map at the corresponding scale to obtain the first fused feature map at the specific scale. The auxiliary feature map at the corresponding scale is output by the second branch model for processing the auxiliary modality image. Based on the visible light feature map at the specific scale and / or the first fused feature map, obtain the feature map to be processed at the specific scale; Based on the feature map to be processed at the specific scale, the process proceeds to the next scale stage of the first encoder, and / or uses the first decoder in the first branch model to perform feature processing to obtain the final processing result, and performs image reconstruction based on the final processing result to obtain the target image. Specifically, for the initial visible light image acquired by the first encoder at the current scale stage, feature extraction is performed using high-frequency attention blocks, including: In the frequency domain branch, high-frequency information of the initial visible light image is mined, and the high-frequency information is converted to the frequency domain through fast Fourier transform to globally update the frequency domain data corresponding to the initial visible light image. The updated frequency domain data is then converted to the spatial domain through inverse fast Fourier transform to obtain the first spatial domain data. In the spatial branch, local information of the initial visible light image is extracted, and feature processing is performed on the local information in the spatial domain to learn the local structural features on the initial visible light image in a multi-scale learning manner, and channel adaptive processing is performed to obtain the second spatial domain data. After fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels, feature extraction information is output.

2. The image reconstruction algorithm of claim 1, wherein, The auxiliary modal image is one of infrared image, radar image, or lidar point cloud image.

3. The image reconstruction algorithm according to claim 1, characterized in that, The step of mining high-frequency information from the initial visible light image in the frequency domain branch includes: Based on the number of channels in the initial visible light image, the feature layer of the initial visible light image is divided into a first feature layer and a second feature layer, and then input into the first processing sub-branch and the second processing branch, respectively. In the first processing sub-branch, the first feature layer is subjected to local feature extraction through a 3×3 convolutional layer, and feature processing is performed using a 1×1 convolutional layer and the GELU activation function to obtain the first processing result; In the second processing branch, the second feature layer is subjected to high-frequency feature extraction through a max pooling layer, and feature processing is performed using a 1×1 convolutional layer and the GELU activation function to obtain the second processing result; The first processing result and the second processing result are aggregated along the channel dimension to obtain the high-frequency information; and / or, The steps of extracting local information from the initial visible light image in the spatial branch, performing feature processing on the local information in the spatial domain to learn local structural features on the initial visible light image based on multi-scale learning, and performing channel adaptive processing to obtain the second spatial domain data include: The local information is extracted using 3×3 convolutional layers, long-range dependencies are established using 5×5 depthwise convolutions and 5×5 depthwise dilated convolutions, the relationships between feature layers are adjusted using 1×1 convolutional layers, and a SE module is added to adaptively capture key feature layer information to obtain the data in the second spatial domain; and / or, The step of fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels to output feature extraction information includes: The first spatial domain data and the second spatial domain data are fused and the number of channels is adjusted through a 1×1 convolutional layer to obtain and output the feature extraction information.

4. The image reconstruction algorithm of claim 3, wherein, In the frequency domain branch, the mathematical expression corresponding to the step of mining the high-frequency information of the initial visible light image is: in, Indicates a convolutional layer. This represents the activation function. This indicates a max pooling operation. This indicates an operation that splices along the channel dimension. This represents the feature layer of the initial visible light image. Indicates the first feature layer. This represents the second feature layer. This indicates the first processing result. This indicates the second processing result. This represents the high-frequency information; and / or, The mathematical expression for the steps of converting the high-frequency information to the frequency domain using a Fast Fourier Transform (FFT) to globally update the frequency domain data corresponding to the initial visible light image, and converting the updated frequency domain data to the spatial domain using an Inverse Fast Fourier Transform (IFFT) to obtain the first spatial domain data, is as follows: in, Indicates a convolutional layer. Represents the Fast Fourier Transform. This represents the inverse Fourier transform. This represents the data in the first spatial domain; and / or, The mathematical expression for the steps of extracting local information using 3×3 convolutional layers, establishing long-range dependencies using 5×5 depthwise convolutions and 5×5 depthwise dilated convolutions, adjusting the relationships between feature layers using 1×1 convolutional layers, and adding an SE module to adaptively capture key feature layer information to obtain the data in the second spatial domain is as follows: in, Indicates a convolutional layer. Represents depthwise convolution. This represents depthwise dilated convolution. Indicates the SE module, This represents the second spatial domain data; and / or, The mathematical expression corresponding to the step of fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels through a 1×1 convolutional layer to obtain and output the feature extraction information is: in, Indicates a convolutional layer. This represents the feature extraction information.

5. The image reconstruction algorithm according to claim 1, characterized in that, Based on the feature map to be processed at the specific scale, feature processing is performed using a first decoder to obtain the final processing result, including: Using the first decoder, the feature map to be processed at the specific scale and the visible light feature map at the corresponding scale output by the first encoder are stitched together to obtain the stitched feature map at the specific scale. For the stitched feature map at the specific scale, feature enhancement is performed by fusing learning blocks and / or upsampling is performed by deconvolution layers to obtain the reconstructed feature map at the specific scale; The reconstructed feature map at the specific scale is fused with the auxiliary reconstructed feature map at the corresponding scale to obtain the second fused feature map at the specific scale, wherein the auxiliary reconstructed feature map at the corresponding scale is output by the second branch model; Based on the reconstructed feature map and / or the second fused feature map at the specific scale, obtain the reconstructed feature map to be processed at the specific scale; The specific scale of the reconstructed feature map to be processed is processed in the first branch model to obtain the final processing result.

6. The image reconstruction algorithm according to claim 5, characterized in that, The workflow of the second branch model in obtaining auxiliary feature maps includes: In the second branch model, the initial auxiliary modality image acquired by the second encoder is subjected to feature extraction through residual groups and / or downsampling through convolutional layers to obtain an auxiliary feature map at the specific scale. The initial auxiliary modality image is either an initial auxiliary feature map obtained after shallow feature extraction of the auxiliary modality image, or an auxiliary feature map acquired by the second encoder in the previous scale stage; and / or, The workflow of the second branch model in obtaining the auxiliary reconstructed feature map includes: In the second branch model, the auxiliary feature map of the second encoder output obtained by the second decoder is enhanced by residual groups and / or upsampled by deconvolution layers to obtain the auxiliary reconstructed feature map at the specific scale.

7. The image reconstruction algorithm according to claim 5, characterized in that, The step of reconstructing the image based on the final processing result to obtain the target image includes: The final processing result is linearly projected to obtain a reconstructed image; Based on the reconstructed image, using minimization L After optimizing the network parameters using a 1-pixel loss, the target image is obtained.

8. An image processing method, characterized in that, include: Acquire the visible light image and auxiliary modal image to be fused; The visible light image and the auxiliary modal image are fused and reconstructed using the image reconstruction algorithm according to any one of claims 1 to 7 to obtain the target image.

9. A computing device, characterized in that, include: The processor and the memory storing the computer program implement the steps of the image processing method of claim 8 when the processor runs the computer program.

10. A vehicle, characterized in that, It is equipped with the computing device as described in claim 9.

11. A computer-readable storage medium, characterized in that, The system contains a computer program that, when executed by a processor, implements the steps of the image processing method of claim 8.

Images