Image reconstruction algorithm, image processing method and device, vehicle and storage medium

Abstract

The invention discloses an image reconstruction algorithm, an image processing method and device, a vehicle and a storage medium. The image processing method comprises the steps of obtaining a visible light image and an auxiliary modal image to be fused; and performing fusion reconstruction on the visible light image and the auxiliary modal image by using an image reconstruction algorithm to obtain a target image, the image reconstruction algorithm comprising a step of completing a feature extraction task of the associated visible light image by using a high-frequency attention mechanism and a Fourier convolution mechanism in the high-frequency attention block. Therefore, the high-frequency information in the visible light image with various image qualities can be rapidly and effectively extracted through the image reconstruction algorithm, so that the target image is obtained by fusing and reconstructing the visible light image and the auxiliary modal image based on the effectively extracted high-frequency information; and features in a visible light image and features in an auxiliary modal image are clearer. Therefore, it can be ensured that images with high quality can be obtained when shooting is carried out in various shooting environments.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image reconstruction algorithm, an image processing method, a computing device, a vehicle, and a computer-readable storage medium. Background Art

[0002] Most of the unmanned or low-involvement smart devices need to have accurate environmental perception capabilities when they are running. Currently, the environmental perception capabilities of various smart devices are mainly achieved by obtaining high-quality environmental images through camera systems.

[0003] However, the camera systems of current smart devices mainly rely on visible light. Therefore, in various harsh shooting environments that affect vision (such as low lighting environments, rainy and foggy environments, dusty environments, etc.), the image quality of the environmental images obtained by the camera system is poor, which makes the environmental perception ability of the smart device poor. Therefore, how to obtain images with better quality when shooting in various shooting environments is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] The purpose of the present application is to provide an image reconstruction algorithm, an image processing method, a computing device, a vehicle and a computer-readable storage medium, which can ensure that better images are obtained when shooting in various shooting environments.

[0005] To achieve the above objectives: In a first aspect, an embodiment of the present application provides an image reconstruction algorithm, including: in a first branch model for processing a visible light image, a visible light initial image obtained by a first encoder at a current scale stage is subjected to feature extraction through a high-frequency attention block and / or down-sampled through a convolution layer to obtain a visible light feature map of a specific scale corresponding to the current scale stage, the high-frequency attention block uses a high-frequency attention mechanism and a Fourier convolution mechanism to complete the feature extraction task, the visible light initial image is an initial feature map obtained after shallow feature extraction of the visible light image, or a feature map to be processed obtained by the first encoder at a previous scale stage; the visible light feature map of a specific scale is subjected to feature fusion with an auxiliary feature map of a corresponding scale to obtain a first fused feature map of a specific scale, and the auxiliary feature map of the corresponding scale is output by a second branch model for processing an auxiliary modality image; according to the visible light feature map of a specific scale and / or the first fused feature map, a feature map to be processed of a specific scale is obtained; according to the feature map to be processed of a specific scale, the next scale stage of the first encoder is entered, and / or the first decoder in the first branch model is used to perform feature processing to obtain a final processing result, and image reconstruction is performed based on the final processing result to obtain a target image.

[0006] In one embodiment, the auxiliary modality image is one of an infrared image, a radar image, and a lidar point cloud image.

[0007] In one embodiment, the high-frequency attention block includes a frequency domain branch and a spatial branch.

[0008] In one embodiment, the visible light initial image acquired by the first encoder at the current scale stage is subjected to feature extraction through a high-frequency attention block, including: In the frequency domain branch, the high-frequency information of the visible light initial image is mined, and the high-frequency information is converted to the frequency domain through fast Fourier transform, so as to globally update the frequency domain data corresponding to the visible light initial image, and the updated frequency domain data is converted to the spatial domain through inverse fast Fourier transform to obtain first spatial domain data; in the spatial branch, the local information of the visible light initial image is extracted, and feature processing is performed on the local information in the spatial domain, so as to learn the local structural features on the visible light initial image in a multi-scale learning-based manner, and channel adaptive processing is performed to obtain second spatial domain data; after the first spatial domain data and the second spatial domain data are fused and the number of channels is adjusted, the feature extraction information is output.

[0009] In one embodiment, in the frequency domain branch, the step of mining the high-frequency information of the visible light initial image includes: dividing the feature layer of the visible light initial image into a first feature layer and a second feature layer according to the number of channels of the visible light initial image, and inputting them into a first processing sub-branch and a second processing branch respectively; in the first processing sub-branch, performing local feature extraction on the first feature layer through a 3×3 convolution layer, and performing feature processing using a 1×1 convolution layer and a GELU activation function to obtain a first processing result; in the second processing branch, performing high-frequency feature extraction on the second feature layer through a maximum pooling layer, and performing feature processing using a 1×1 convolution layer and a GELU activation function to obtain a second processing result; and aggregating the first processing result and the second processing result in the channel dimension to obtain high-frequency information.

[0010] In one embodiment, in the spatial branch, local information of the visible light initial image is extracted, and feature processing is performed on the local information in the spatial domain, so as to learn the local structural features on the visible light initial image in a multi-scale learning-based manner, and perform channel adaptive processing to obtain the second spatial domain data. The steps include: extracting local information through a 3×3 convolution layer, establishing long-distance dependency using a 5×5 depth convolution and a 5×5 depth dilated convolution, adjusting the relationship between feature layers through a 1×1 convolution layer, and adding an SE module to adaptively capture key feature layer information to obtain the second spatial domain data.

[0011] In one embodiment, after the first spatial domain data and the second spatial domain data are fused and the number of channels is adjusted, the step of outputting feature extraction information includes: fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels through a 1×1 convolution layer to obtain and output feature extraction information.

[0012] In one embodiment, in the frequency domain branch, the step of mining the high-frequency information of the visible light initial image corresponds to the mathematical expression:

[0013]

[0014]

[0015]

[0016] in, represents the convolutional layer, represents the activation function, represents the maximum pooling operation, Represents a concatenation operation along the channel dimension.

[0017] In one embodiment, the high-frequency information is converted to the frequency domain by fast Fourier transform to globally update the frequency domain data corresponding to the visible light initial image, and the updated frequency domain data is converted to the spatial domain by inverse fast Fourier transform to obtain the first spatial domain data. The corresponding mathematical expression is:

[0018] in, represents the convolutional layer, represents the fast Fourier transform, Represents the inverse Fourier transform.

[0019] In one embodiment, the steps of extracting local information through a 3×3 convolution layer, establishing long-distance dependencies using a 5×5 depth convolution and a 5×5 depth dilated convolution, adjusting the relationship between feature layers through a 1×1 convolution layer, and adding an SE module to adaptively capture key feature layer information to obtain the second spatial domain data, correspond to the following mathematical expressions:

[0020] in, represents the convolutional layer, represents the depthwise convolution, represents the depth-dilated convolution, Indicates the SE module.

[0021] In one embodiment, the first spatial domain data and the second spatial domain data are fused and the number of channels is adjusted through a 1×1 convolution layer to obtain and output feature extraction information. The corresponding mathematical expression is:

[0022] in, Represents a convolutional layer.

[0023] In one embodiment, according to a feature map to be processed at a specific scale, a first decoder is used to perform feature processing to obtain a final processing result, including: using the first decoder, splicing the feature map to be processed at a specific scale and the visible light feature map of the corresponding scale output by the first encoder to obtain a spliced ​​feature map of the specific scale; performing feature enhancement on the spliced ​​feature map of the specific scale through a fusion learning block and / or upsampling through a deconvolution layer to obtain a reconstructed feature map of the specific scale; performing feature fusion on the reconstructed feature map of the specific scale with the auxiliary reconstructed feature map of the corresponding scale to obtain a second fused feature map of the specific scale, wherein the auxiliary reconstructed feature map of the corresponding scale is output by the second branch model; obtaining a reconstructed feature map to be processed at a specific scale according to the reconstructed feature map of the specific scale and / or the second fused feature map; processing the reconstructed feature map to be processed at a specific scale in the first branch model to obtain the final processing result.

[0024] In one embodiment, the workflow of the second branch model to obtain the auxiliary feature map includes: in the second branch model, the auxiliary modality initial image obtained by the second encoder is subjected to feature extraction through a residual group and / or down-sampled through a convolutional layer to obtain an auxiliary feature map of a specific scale, wherein the auxiliary modality initial image is an initial auxiliary feature map obtained after performing shallow feature extraction on the auxiliary modality image, or an auxiliary feature map obtained by the second encoder at the previous scale stage.

[0025] In one embodiment, the workflow of the second branch model obtaining the auxiliary reconstruction feature map includes: in the second branch model, the second decoder obtains the auxiliary feature map output by the second encoder, performs feature enhancement through the residual group and / or upsamples through the deconvolution layer to obtain an auxiliary reconstruction feature map of a specific scale.

[0026] In one embodiment, the step of reconstructing the image according to the final processing result to obtain the target image includes: linearly projecting the final processing result to obtain a reconstructed image; and minimizing the reconstructed image. L After optimizing the network parameters with 1 pixel loss, the target image is obtained.

[0027] In a second aspect, an embodiment of the present application provides an image processing method, comprising: obtaining a visible light image and an auxiliary modality image to be fused; fusing and reconstructing the visible light image and the auxiliary modality image using an image reconstruction algorithm to obtain a target image, wherein the image reconstruction algorithm includes the step of using a high-frequency attention mechanism and a Fourier convolution mechanism in a high-frequency attention block to complete the feature extraction task of the associated visible light image.

[0028] In a third aspect, an embodiment of the present application provides a computing device, comprising: a processor and a memory storing a computer program, and when the processor runs the computer program, the steps of the image processing method described above are implemented.

[0029] In a fourth aspect, an embodiment of the present application provides a vehicle equipped with the above-mentioned computing device.

[0030] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the image processing method described above.

[0031] The image reconstruction algorithm provided by the embodiment of the present application includes: in a first branch model for processing visible light images, a high-frequency attention block is used to extract features of an initial visible light image obtained by a first encoder at a current scale stage and / or down-sampled through a convolution layer to obtain a visible light feature map of a specific scale corresponding to the current scale stage, the high-frequency attention block uses a high-frequency attention mechanism and a Fourier convolution mechanism to complete the feature extraction task, the initial visible light image is an initial feature map obtained after shallow feature extraction of the visible light image, or a feature map to be processed obtained by the first encoder at a previous scale stage; the visible light feature map of a specific scale is feature-fused with an auxiliary feature map of a corresponding scale to obtain a first fused feature map of a specific scale, and the auxiliary feature map of the corresponding scale is output by a second branch model for processing auxiliary modality images; according to the visible light feature map of a specific scale and / or the first fused feature map, a feature map to be processed of a specific scale is obtained; according to the feature map to be processed of a specific scale, the next scale stage of the first encoder is entered, and / or the first decoder in the first branch model is used to perform feature processing to obtain a final processing result, and image reconstruction is performed according to the final processing result to obtain a target image. In this way, the technical solution of the present application completes the feature extraction task through the high-frequency attention mechanism and Fourier convolution mechanism in the high-frequency attention block, and can realize fast and effective extraction of high-frequency information in visible light images of various image qualities, so as to determine the visible light feature image corresponding to the visible light image based on the effectively extracted high-frequency information, and then perform multi-scale deep feature fusion and image reconstruction based on the auxiliary feature map corresponding to the visible light feature image and the additionally collected auxiliary modality image to obtain the target image, and the target image has clearer features in the visible light image and the features in the auxiliary modality image. Therefore, the technical solution of the present application can ensure that images of better quality are obtained when shooting in various shooting environments.

[0032] The image processing method provided by the embodiment of the present application includes: obtaining a visible light image and an auxiliary modality image to be fused; fusing and reconstructing the visible light image and the auxiliary modality image using an image reconstruction algorithm to obtain a target image, wherein the image reconstruction algorithm includes the step of using the high-frequency attention mechanism and the Fourier convolution mechanism in the high-frequency attention block to complete the feature extraction task of the associated visible light image. In this way, in the technical solution of the present application, since the high-frequency attention mechanism and the Fourier convolution mechanism in the high-frequency attention block are used to complete the feature extraction task of the associated visible light image, the image reconstruction algorithm can realize the rapid and effective extraction of high-frequency information in visible light images of various image qualities, so that the target image obtained by fusing and reconstructing the visible light image and the auxiliary modality image based on the effectively extracted high-frequency information has clearer features in the visible light image and features in the auxiliary modality image. Therefore, the technical solution of the present application can ensure that images with better quality are obtained when shooting in various shooting environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0034] Figure 1 It is a first flow chart of the image reconstruction algorithm provided in the first embodiment of the present application.

[0035] Figure 2 Schematic diagram of a high-frequency attention block according to the first embodiment of the present application.

[0036] Figure 3 It is a second flow chart of the image reconstruction algorithm provided in the first embodiment of the present application.

[0037] Figure 4 It is a principle framework diagram of the image reconstruction algorithm of the present application example.

[0038] Figure 5 It is a schematic diagram of a multi-source data fusion module of an example of this application.

[0039] Figure 6 It is a schematic diagram of the framework of a camera system applying an image reconstruction algorithm to realize various functions in the example of this application.

[0040] Figure 7 It is a flowchart of the image processing method provided in the first embodiment of the present application.

[0041] Figure 8 It is a structural diagram of the computing device provided by this application.

[0042] The realization of the purpose, functional features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. The above-mentioned drawings have shown clear embodiments of this application, which will be described in more detail later. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0044] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0045] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in this article, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this article, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "at the time of..." or "when..." or "in response to determination". Furthermore, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising", "including" indicate that there are described features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Thus, “A, B, or C” or “A, B and / or C” means “any of the following: A; B; C; A and B; A and C; B and C; A, B, and C.” An exception to this definition will occur only when a combination of elements, functions, steps, or operations are inherently mutually exclusive in some manner.

[0046] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0047] It should be noted that, in this article, step codes such as S11, S12, etc. are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence. When implementing it specifically, those skilled in the art may execute S12 first and then S11, etc., but these should all be within the scope of protection of this application.

[0048] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0050] First embodiment See also Figure 1 , is an image reconstruction algorithm provided in the first embodiment of the present application, and the method can be executed by a computing device provided in the embodiment of the present application, and the computing device can be implemented in software and / or hardware. The computing device is, for example, a controller in a camera system.

[0051] This embodiment provides an image reconstruction algorithm, including the following steps (for example, steps S11 to S14): Step S11: In the first branch model for processing visible light images, the visible light initial image obtained by the first encoder at the current scale stage is feature extracted through the high-frequency attention block and / or down-sampled through the convolution layer to obtain a visible light feature map of a specific scale corresponding to the current scale stage. The high-frequency attention block uses a high-frequency attention mechanism and a Fourier convolution mechanism to complete the feature extraction task.

[0052] In one implementation, the visible light initial image may be an initial feature map obtained after shallow feature extraction is performed on the visible light image, or a feature map to be processed obtained by the first encoder in the previous scaling stage.

[0053] In one embodiment, when the initial visible light image is an initial feature map obtained after shallow feature extraction is performed on the visible light image, before step S11, the following may be included: shallow feature extraction and image channel number expansion are performed on the visible light image through linear mapping to obtain the initial feature map. The technical solution of this embodiment performs shallow feature extraction on the visible light image and expands the number of image channels through linear mapping, which helps to increase the dimension and richness of the features while keeping the image size unchanged.

[0054] In one embodiment, the feature map may represent various features extracted from the original input image, such as edges, textures, shapes, etc.

[0055] In one embodiment, before step S11, the method may include: constructing a first branch model for processing visible light images; and constructing a second branch model for processing auxiliary modality images.

[0056] In one embodiment, the first branch model for processing the visible light image may be a first multi-scale image reconstruction model. The first multi-scale image reconstruction model includes a first encoder and a first decoder.

[0057] In one embodiment, in the first multi-scale image reconstruction model, the first encoder mainly extracts and outputs visible light feature maps of multiple scales by performing feature extraction on the initial feature map corresponding to the visible light image through a high-frequency attention block and / or downsampling through a convolutional layer.

[0058] In one embodiment, the working steps of the first encoder include multiple scale stages, and each scale stage can extract and output a visible light feature map of a scale by performing feature extraction through a high-frequency attention block and / or downsampling through a convolutional layer. Optionally, the scales of the visible light feature maps output by the first encoder at different scale stages are different.

[0059] In one embodiment, the final scaling stage of the first encoder may extract and output a visible light feature map of a scale by only performing feature extraction through the high-frequency attention block.

[0060] In one embodiment, the first encoder can synchronously output the visible light feature map obtained at each scale stage to the multi-source data fusion module, so that the multi-source data fusion module performs layer-by-layer processing to fuse the visible light feature map and the auxiliary feature map of the same scale to obtain a fused feature map (i.e., the first fused feature map described below) for feedback to the first encoder.

[0061] In one embodiment, in the first multi-scale image reconstruction model, the first decoder mainly splices a feature map of one scale output by the last scaling stage of the first encoder with other feature maps of corresponding scales to obtain a spliced ​​feature map of an initial scale, and then, for the spliced ​​feature map of the initial scale, performs feature enhancement by a fusion learning block and / or performs upsampling by a deconvolution layer to extract and output reconstructed feature maps of multiple scales.

[0062] In one embodiment, the working steps of the first decoder also include multiple scale stages, and each scale stage can extract and output a reconstructed feature map of a scale by performing feature enhancement through a fusion learning block and / or upsampling through a deconvolution layer. Optionally, the scales of the reconstructed feature maps output by the first decoder at different scale stages are different.

[0063] In one embodiment, the final scaling stage of the first decoder may extract and output a reconstructed feature map having the same scale as the initial feature map corresponding to the visible light image by only fusing the learning block for feature enhancement.

[0064] In one embodiment, the first decoder can output the reconstructed feature map obtained at each scale stage to the multi-source data fusion module, so that the multi-source data fusion module performs layer-by-layer processing to fuse the reconstructed feature map and the auxiliary reconstructed feature map of the same scale to obtain a fused feature map (i.e., the second fused feature map described below) for feedback to the first decoder.

[0065] In one embodiment, the number of scaling stages of the first decoder is the same as the number of scaling stages of the first encoder, so that the scale of the reconstructed feature map output by the last scaling stage of the first decoder can be consistent with the scale of the initial feature map corresponding to the visible light image. In this way, the reconstructed feature map output by the last scaling stage can include the effective global features of the visible light image, thereby ensuring that the target image obtained based on the reconstructed feature map can retain the effective global features of the visible light image.

[0066] In one embodiment, the second branch model for processing the auxiliary modality image may be a second multi-scale image reconstruction model. The second multi-scale image reconstruction model includes a second encoder and a second decoder.

[0067] In one embodiment, the first multi-scale image reconstruction model and the second multi-scale image reconstruction model may be the same or different.

[0068] In one embodiment, in the second multi-scale image reconstruction model, the second encoder mainly extracts and outputs auxiliary feature maps of multiple scales for the initial auxiliary feature map corresponding to the auxiliary modality image by performing feature extraction through a residual group and / or downsampling through a convolutional layer.

[0069] In one embodiment, the method for obtaining the initial auxiliary feature map corresponding to the auxiliary modality image may include: performing shallow feature extraction and image channel expansion on the auxiliary modality image through linear mapping to obtain the initial auxiliary feature map. The technical solution of this embodiment performs shallow feature extraction and image channel expansion on the initial auxiliary feature map through linear mapping, which helps to increase the dimension and richness of the features while keeping the image size unchanged.

[0070] In one embodiment, the working steps of the second encoder include multiple scale stages, and each scale stage can extract and output an auxiliary feature map of a scale by performing feature extraction through a residual group and / or downsampling through a convolutional layer. Optionally, the scales of the auxiliary feature maps output by the second encoder at different scale stages are different.

[0071] In one implementation, each scale of the auxiliary feature map in this embodiment may correspond to a scale of the visible light feature map output by the first encoder.

[0072] In one embodiment, the final scaling stage of the second encoder may extract and output an auxiliary feature map of one scale by performing feature extraction only through the residual group.

[0073] In one embodiment, the second encoder can synchronously output the auxiliary feature map obtained at each scale stage to the multi-source data fusion module, so that the multi-source data fusion module performs layer-by-layer processing to fuse the visible light feature map and the auxiliary feature map of the same scale to obtain a fused feature map for feedback to the first encoder (i.e., the first fused feature map described below).

[0074] In one embodiment, in the second multi-scale image reconstruction model, the second decoder mainly extracts and outputs auxiliary reconstruction feature maps of multiple scales by performing feature enhancement through a residual group and / or upsampling through a deconvolution layer on an auxiliary feature map of one scale outputted by the last scaling stage of the second encoder.

[0075] In one embodiment, the working steps of the second decoder also include multiple scale stages, and each scale stage can extract and output an auxiliary reconstruction feature map of a scale by performing feature enhancement through a residual group and / or upsampling through a deconvolution layer. Optionally, the scales of the reconstructed feature maps output by the second decoder at different scale stages are different.

[0076] In one embodiment, the second decoder can output the auxiliary reconstruction feature map obtained at each scale stage to the multi-source data fusion module, so that the multi-source data fusion module performs layer-by-layer processing to fuse the reconstructed feature map and the auxiliary reconstruction feature map of the same scale to obtain a fused feature map (i.e., the second fused feature map described below) for feedback to the first decoder.

[0077] In one embodiment, the number of scaling stages of the second decoder is at least one less than the number of scaling stages of the second encoder, so that the auxiliary reconstruction feature map output by the last scaling stage of the second decoder can characterize the key features of the auxiliary modality image (i.e., the global features of the non-auxiliary modality image). Therefore, the technical solution of this embodiment can achieve image reconstruction using only the key features of the auxiliary modality image, so that the target image obtained after image reconstruction can represent the effective global features of the visible light image and all or part of the key features of the auxiliary modality image.

[0078] In one embodiment, the multi-scale image reconstruction model mainly refers to a model that can simultaneously process and utilize features of different scales during the image reconstruction process.

[0079] In one embodiment, the high-frequency attention mechanism (HAB) is mainly to adaptively focus on high-frequency information in the image. Optionally, high-frequency information usually contains details and edge information of the image, which is crucial for reconstructing high-resolution images.

[0080] In one embodiment, the Fourier convolution mechanism includes a fast Fourier transform (FFT) and an inverse Fourier transform (IFFT). Specifically, the combination of the fast Fourier transform (FFT) and the inverse Fourier transform (IFFT) in the Fourier convolution mechanism with the high-frequency attention mechanism (HAB) is mainly used to enhance the extraction process of high-frequency information. This combination strategy helps the model capture and utilize key information in the feature map, improves the performance and accuracy of the model, and simplifies the design and optimization process of the model.

[0081] In one implementation, the high-frequency attention block can effectively extract high-frequency information from visible light images of various image qualities, for example, effectively extract hidden high-frequency information from a blurred visible light image or its corresponding feature map. Therefore, when the technical solution of this embodiment reconstructs the visible light image based on the effectively extracted high-frequency information, a clearer visible light image can be obtained.

[0082] In one embodiment, see Figure 2 ,The high-frequency attention block includes a frequency domain branch and a spatial branch.

[0083] In one embodiment, the visible light initial image acquired by the first encoder at the current scale stage is subjected to feature extraction through a high-frequency attention block, including: In the frequency domain branch, the high-frequency information of the visible light initial image is mined, and the high-frequency information is converted to the frequency domain through fast Fourier transform, so as to globally update the frequency domain data corresponding to the visible light initial image, and the updated frequency domain data is converted to the spatial domain through inverse fast Fourier transform to obtain first spatial domain data; in the spatial branch, the local information of the visible light initial image is extracted, and feature processing is performed on the local information in the spatial domain, so as to learn the local structural features on the visible light initial image in a multi-scale learning-based manner, and channel adaptive processing is performed to obtain second spatial domain data; after the first spatial domain data and the second spatial domain data are fused and the number of channels is adjusted, the feature extraction information is output.

[0084] In one embodiment, see Figure 2 In the frequency domain branch, the step of mining high-frequency information of the visible light initial image may include: dividing the feature layer of the visible light initial image into a first feature layer and a second feature layer according to the number of channels of the visible light initial image, and inputting them into the first processing sub-branch and the second processing branch respectively; in the first processing sub-branch, performing local feature extraction on the first feature layer through a 3×3 convolution layer (Conv 3×3), and performing feature processing using a 1×1 convolution layer (Conv 1×1) and a GELU activation function to obtain a first processing result; in the second processing branch, performing high-frequency feature extraction on the second feature layer through a maximum pooling layer, and performing feature processing using a 1×1 convolution layer (Conv 1×1) and a GELU activation function to obtain a second processing result; and aggregating the first processing result and the second processing result in the channel dimension to obtain high-frequency information.

[0085] In one embodiment, in the frequency domain branch, the step of mining the high-frequency information of the visible light initial image corresponds to the mathematical expression:

[0086]

[0087]

[0088]

[0089] in, represents the convolutional layer, represents the activation function, represents the maximum pooling operation, Represents a concatenation operation along the channel dimension.

[0090] in, The feature layer representing the initial image of visible light, can represent the first feature layer, can represent the second feature layer, Indicates the first processing result, represents the second processing result, Represents high-frequency information.

[0091] In one embodiment, the high-frequency information is converted to the frequency domain by fast Fourier transform to globally update the frequency domain data corresponding to the visible light initial image, and the updated frequency domain data is converted to the spatial domain by inverse fast Fourier transform to obtain the first spatial domain data. The corresponding mathematical expression is:

[0092] in, represents the convolutional layer, represents the fast Fourier transform, Represents the inverse Fourier transform.

[0093] in, Represents high-frequency information, Represents the first spatial domain data.

[0094] In one embodiment, see Figure 2 , in the spatial branch, extracting local information of the visible light initial image, and performing feature processing in the spatial domain for the local information, learning local structural features on the visible light initial image in a multi-scale learning-based manner, and performing channel adaptive processing to obtain the second spatial domain data step, can include: extracting local information through a 3×3 convolution layer (Conv 3×3), using a 5×5 depth convolution (depth convolution Conv 5×5) and a 5×5 depth expansion convolution (depth expansion convolution Conv 5×5) to establish a long-distance dependency relationship, adjusting the relationship between each feature layer through a 1×1 convolution layer (Conv 1×1), and adding an SE module to adaptively capture key feature layer information (channel adaptation) to obtain the second spatial domain data.

[0095] In one embodiment, the steps of extracting local information through a 3×3 convolution layer, establishing long-distance dependencies using a 5×5 depth convolution and a 5×5 depth dilated convolution, adjusting the relationship between feature layers through a 1×1 convolution layer, and adding an SE module to adaptively capture key feature layer information to obtain the second spatial domain data, correspond to the following mathematical expressions:

[0096] in, represents the convolutional layer, represents the depthwise convolution, represents the depth-dilated convolution, Indicates the SE module.

[0097] in, The feature layer representing the initial image of visible light, Represents the second spatial domain data.

[0098] In one embodiment, see Figure 2 After fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels, the step of outputting feature extraction information includes: fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels through a 1×1 convolution layer (Conv 1×1) to obtain and output feature extraction information.

[0099] In one embodiment, the feature extraction information may be a visible light feature map of a specific scale.

[0100] In one embodiment, the first spatial domain data and the second spatial domain data are fused and the number of channels is adjusted through a 1×1 convolution layer to obtain and output feature extraction information. The corresponding mathematical expression is:

[0101] in, represents the convolutional layer, represents the second spatial domain data, represents the first spatial domain data, Represents feature extraction information.

[0102] In one embodiment, the effects of downsampling by the convolutional layer include but are not limited to at least one of the following: (1) Reduce the amount of computation and prevent overfitting: Downsampling reduces the amount of computation of the model by reducing the size of the feature map, which helps improve the operating efficiency of the model. At the same time, downsampling can reduce the number of model parameters, thereby reducing the risk of overfitting; (2) Increasing the receptive field: Downsampling can increase the receptive field of subsequent layers, allowing the model to capture more global information, helping the model to better understand the global structure and content of the image; (3) Preserving spatial information: When using convolutional layers for downsampling, the convolution operation can preserve the spatial structure and position information of the input features, which helps maintain the spatial consistency of the image during subsequent image reconstruction.

[0103] Step S12: feature fusion of the visible light feature map of a specific scale and the auxiliary feature map of a corresponding scale to obtain a first fused feature map of the specific scale, where the auxiliary feature map of the corresponding scale is output by the second branch model for processing the auxiliary modality image.

[0104] In one embodiment, the auxiliary modality image may be an infrared image, a radar image, a laser radar point cloud image, or any other image other than a visible light image. Optionally, the auxiliary modality image in this embodiment may be an infrared image.

[0105] In one embodiment, the workflow of the second branch model to obtain the auxiliary feature map may include: in the second branch model, the auxiliary modality initial image obtained by the second encoder is subjected to feature extraction through a residual group and / or down-sampled through a convolutional layer to obtain an auxiliary feature map of a specific scale.

[0106] In one embodiment, the auxiliary modality initial image may be an initial auxiliary feature map obtained after shallow feature extraction is performed on the auxiliary modality image, or an auxiliary feature map obtained by the second encoder in the previous scaling stage.

[0107] In one embodiment, when the auxiliary modality initial image is an initial auxiliary feature map obtained after shallow feature extraction of the auxiliary modality image, in the second branch model, before the step of extracting features through a residual group and / or downsampling through a convolutional layer on the auxiliary modality initial image obtained by the second encoder to obtain an auxiliary feature map of a specific scale, it may include: performing shallow feature extraction and image channel expansion on the auxiliary modality image through linear mapping to obtain the initial auxiliary feature map. The technical solution of this embodiment performs shallow feature extraction on the auxiliary modality image and expands the number of image channels through linear mapping, which helps to increase the dimension and richness of features while keeping the image size unchanged.

[0108] In one embodiment, the second branch model for processing auxiliary modality images may be a multi-scale image reconstruction model.

[0109] Step S13: obtaining a feature map to be processed at a specific scale according to the visible light feature map at a specific scale and / or the first fused feature map.

[0110] In one embodiment, step S13: obtaining a feature map to be processed at a specific scale according to the visible light feature map at a specific scale and / or the first fused feature map, includes but is not limited to one of the following situations: The visible light feature map of a specific scale is fused again with the first fused feature map to obtain a feature map to be processed of a specific scale; The first fused feature map of a specific scale is used as a feature map to be processed of a specific scale.

[0111] Step S14: According to the feature map to be processed at a specific scale, enter the next scale stage of the first encoder, and / or use the first decoder in the first branch model to perform feature processing to obtain a final processing result, and perform image reconstruction according to the final processing result to obtain a target image.

[0112] In one embodiment, the first encoder may include multiple scale stages, and the scale of the feature map to be processed acquired or output by each scale stage is different.

[0113] In one embodiment, according to the feature map to be processed at a specific scale, feature processing is performed using the first decoder to obtain a final processing result, which may include: using the first decoder to splice the feature map to be processed at a specific scale and the visible light feature map of the corresponding scale output by the first encoder to obtain a spliced ​​feature map of the specific scale; performing feature enhancement on the spliced ​​feature map of the specific scale through a fusion learning block and / or upsampling through a deconvolution layer to obtain a reconstructed feature map of the specific scale; performing feature fusion on the reconstructed feature map of the specific scale with the auxiliary reconstructed feature map of the corresponding scale to obtain a second fused feature map of the specific scale, wherein the auxiliary reconstructed feature map of the corresponding scale is output by the second branch model; obtaining the reconstructed feature map to be processed at a specific scale according to the reconstructed feature map of the specific scale and / or the second fused feature map; processing the reconstructed feature map to be processed at a specific scale in the first branch model to obtain the final processing result.

[0114] In one embodiment, the workflow of the second branch model obtaining the auxiliary reconstruction feature map may include: in the second branch model, obtaining the auxiliary feature map output by the second encoder for the second decoder, performing feature enhancement through the residual group and / or upsampling through the deconvolution layer to obtain an auxiliary reconstruction feature map of a specific scale.

[0115] In one embodiment, the step of reconstructing the image according to the final processing result to obtain the target image may include: linearly projecting the final processing result to obtain a reconstructed image; and minimizing the reconstructed image. L After optimizing the network parameters with a loss of 1 pixel, the target image is obtained. In this way, the technical solution of this embodiment can achieve high-quality image reconstruction, so that the target image has clearer effective global features in the visible light image and key features in the auxiliary modality image.

[0116] The image reconstruction algorithm provided in this embodiment can be applied to the camera system of the smart device to improve the environmental perception capability of the smart device, so that the smart device can better realize unmanned or less-manned intelligent operation. In addition, when the image reconstruction algorithm is applied to the camera system of the smart device, the camera system can output high-quality images for use by the downstream task modules of the smart device, such as the path planning module planning the path according to the image, the target recognition module identifying the feature scene or target object according to the image, etc.

[0117] In one embodiment, the smart device may be a mining vehicle, and the image reconstruction algorithm may be applied to the camera system of the mining vehicle, and the high-quality images output by the camera system may support the monitoring of the mining environment and the functions of the vehicle mission module. The functions of the vehicle mission module may include, for example, the driving path planning function and the obstacle avoidance function of the automatic driving module; for example, the scene or target object recognition function of the target recognition module; for example, the depth estimation function of the depth estimation module.

[0118] In one embodiment, the camera system of the mining vehicle can support the simultaneous shooting of visible light images and infrared images. In this way, the image reconstruction algorithm applied to the camera system of the mining vehicle can use the infrared image as an auxiliary modal image in addition to the visible light image, and reconstruct the image so that the camera system of the mining vehicle outputs a target image, which has not only the effective global features of the visible light image, but also the key features of the infrared image. Therefore, the image reconstruction algorithm applied to the camera system of the mining vehicle can use the infrared image's strong thermal information capture ability, strong penetration ability, and unaffected by light to supplement the feature information that is difficult to perceive in the visible light image in various environments (such as mining vehicles at night, foggy and other harsh environments) to reconstruct high-quality images. In addition, the multi-source data fusion module in the image reconstruction algorithm can interactively fuse the features extracted from the visible light image and the infrared image, alleviate the problem of low image quality and inability to provide important information due to the problems of the visible light image's extreme dependence on the light source and weak penetration ability, so that the image output by the camera system of the mining vehicle can accurately and effectively reflect the characteristics of roads, vehicles, personnel, depth, etc.

[0119] The present embodiment provides an image reconstruction algorithm, including: step S11: in a first branch model for processing a visible light image, a visible light initial image obtained by a first encoder at a current scale stage is subjected to feature extraction through a high-frequency attention block and / or down-sampled through a convolution layer to obtain a visible light feature map of a specific scale corresponding to the current scale stage, and the high-frequency attention block uses a high-frequency attention mechanism and a Fourier convolution mechanism to complete the feature extraction task; step S12: the visible light feature map of a specific scale is subjected to feature fusion with an auxiliary feature map of a corresponding scale to obtain a first fused feature map of a specific scale, and the auxiliary feature map of the corresponding scale is output by a second branch model for processing an auxiliary modality image; step S13: according to the visible light feature map of a specific scale and / or the first fused feature map, a feature map to be processed of a specific scale is obtained; step S14: according to the feature map to be processed of a specific scale, the next scale stage of the first encoder is entered, and / or the first decoder in the first branch model is used to perform feature processing to obtain a final processing result, and image reconstruction is performed according to the final processing result to obtain a target image.

[0120] Based on the same inventive concept as the above embodiments, the following example illustrates a specific application scenario of an image reconstruction algorithm applied to a camera system of a mining vehicle for reference: In the process of open-pit mining, semi-continuous mining processes such as single-bucket excavator-mining vehicle-crushing station-belt conveyor are currently used. Among them, the transportation efficiency of mining vehicles has a great impact on the progress of mining operations. Due to the complex working conditions in the mining area, there is often dust on the road, the visibility is poor during night operations, and mining and blasting operations will cause dust to spread. In rainy and foggy weather, the road is muddy and difficult to identify. In these harsh environments, the driver's ability to perceive the road conditions in the mining area of ​​manned mining vehicles is significantly reduced. Therefore, mining vehicles can be equipped with camera systems and other additional sensing equipment to achieve environmental monitoring and improve the driver's ability to perceive the road conditions in the mining area; or in these harsh environments, unmanned and less-manned mining vehicles can achieve environmental perception through the arranged camera systems and other additional sensing equipment to achieve intelligent operation and improve the economic benefits of mine output.

[0121] See also Figure 3 In this example, the camera system in the mining vehicle can synchronously capture visible light images and infrared images, and the image reconstruction algorithm in the camera system in the mining vehicle includes the following steps: S100: Keeping the size of the captured visible light image unchanged, performing shallow feature extraction and image channel number expansion on the captured visible light image through linear mapping to obtain an initial feature map.

[0122] S101: Keeping the size of the captured infrared image unchanged, performing shallow feature extraction and image channel number expansion on the captured visible light image through linear mapping to obtain an initial auxiliary feature map. S102: In the visible light image branch, the encoder uses the high-frequency attention block to extract features from the initial feature map, and then downsamples it through the convolution layer. The output results of each scale stage are synchronously input into the multi-source data fusion module for layer-by-layer processing.

[0123] Optionally, the encoder outputs results of different scales at different scaling stages.

[0124] Optionally, the visible light image branch may be a first multi-scale image reconstruction model including an encoder and a decoder.

[0125] S103: In the visible light image branch, the feature maps of the decoder input at different scale stages are spliced ​​with the feature maps of the corresponding scales output by the encoder, and then feature enhancement is performed through the fusion learning block, and then up-sampled through deconvolution. The output results of each scale stage are synchronously input into the multi-source data fusion module for layer-by-layer processing.

[0126] Optionally, the number of scale stages of the decoder in the visible light image branch is consistent with the number of scale stages of the encoder.

[0127] S104: In the infrared image auxiliary branch, the residual group (RG) in the encoder is used to extract features from the initial auxiliary feature map, and then downsampled through the convolution layer. The output results of each scale stage are synchronously input into the multi-source data fusion module for layer-by-layer processing.

[0128] Optionally, the encoder outputs results of different scales at different scaling stages.

[0129] Optionally, the infrared image auxiliary branch may be a second multi-scale image reconstruction model including an encoder and a decoder.

[0130] S105: In the infrared image auxiliary branch, the decoder performs feature enhancement through the residual group and then uses deconvolution for upsampling. The output results of each scale stage are synchronously input into the multi-source data fusion module for layer-by-layer processing.

[0131] Optionally, in the infrared image auxiliary branch, the number of decoder scale stages is 1 less than the number of encoder scale stages.

[0132] S106: The multi-source data fusion module receives the output results of the corresponding stages in the visible light image branch and the infrared image auxiliary branch, and performs interactive fusion. The processed features are fused again with the output features of the visible light image branch, and then transmitted to the next scale stage of the encoder of the visible light image branch or the next scale stage of the decoder of the visible light image branch.

[0133] S107: Linearly project the feature map of the final scaling stage of the decoder in the visible light image branch, and reconstruct the decoder output result to form a reconstructed image.

[0134] S108: Based on the reconstructed image and minimization L After optimizing the network parameters with 1 pixel loss, the target image is obtained.

[0135] Optionally, the network parameters are optimized by minimizing the L1 pixel loss, and the loss is calculated by comparing the reconstructed image with the real image.

[0136] Alternatively, see Figure 4 In this embodiment, the number of scale stages of the decoder and the number of scale stages of the encoder in the visible light image branch are both 3, and the number of scale stages of the encoder and the number of scale stages of the decoder in the infrared image auxiliary branch are 3 and 2, and the image reconstruction algorithm is illustrated as follows: Optionally, the specific process of the encoder in the visible light image branch includes:

[0137]

[0138]

[0139]

[0140]

[0141] in, Represents the input visible light image After shallow feature extraction, the initial feature map is obtained. represents the High Frequency Attention Block (HAB), or represents a downsampling block (e.g. output block The corresponding downsampling block and output block of the scale stage The corresponding downsampling block in the scaling stage), Represents the output blocks of each scale stage of the encoder (see Figure 4 In to ), represents the output blocks of the multi-source data fusion module at each scale stage of the encoder and decoder (see Figure 4 In to ), The input block for the next stage in the decoder and encoder (see Figure 4 In to ), Represents the direct output blocks of each scale stage of the encoder to the corresponding scale stage of the decoder (see Figure 4 In to ).

[0142] Optionally, the decoder in the visible light image branch is composed of a fusion learning block (CLB) and an upsampling module. The fusion learning block consists of two key parts: CLB and convolution layer. In order to enable the model to directly utilize the feature information in the feature map of the original input, the fusion learning block of each scale stage receives two inputs: the feature map directly output by the encoder at the corresponding scale stage and the feature map of the previous scale stage after deconvolution. The compensation mechanism can reduce the information loss caused by the calculation process. This fusion strategy is conducive to improving the fitting ability of the model. After receiving multiple inputs, they are stacked in the channel dimension, and then feature processing is performed using CLB. Then, the number of channels is adjusted through the convolution layer to promote information fusion, and finally deconvolution is used to restore the size of the feature map. The specific process of the decoder in the visible light image branch includes:

[0143]

[0144]

[0145]

[0146] in, Indicates CLB, represents a deconvolution module (e.g., the output block in the decoder Corresponding deconvolution module at the scaling stage, output block in the decoder The corresponding deconvolution module of the scale stage), Represents an output block in the decoder.

[0147] Optionally, the specific process of the encoder in the infrared image auxiliary branch includes:

[0148]

[0149] in, The input infrared image The initial auxiliary feature map obtained by shallow feature extraction, represents the residual group (RG), represents the downsampling block (output block The corresponding downsampling block and output block of the scale stage The corresponding downsampling block in the scaling stage), Represents the output blocks of each scale stage of the encoder (see Figure 4 In to ), Represents the output block of the encoder and decoder at each scale stage connected to the multi-source data fusion module (see Figure 4 In to ). Optionally, the residual group (RG) consists of multiple stacked residual channel attention blocks (RCABs) and a convolutional layer, and the formula is expressed as:

[0150]

[0151] in, represents the convolutional layer, Indicates N residual channel attention blocks, represents the channel attention module, and ReLU represents the activation function.

[0152] Optionally, the decoder in the infrared image auxiliary branch is composed of a residual group (RG) and an upsampling module. The residual group (RG) is first used for feature processing, and then deconvolution is used to restore the image size. The specific process of the decoder in the infrared image auxiliary branch includes:

[0153] in, represents the residual group (RG), represents a deconvolution module (e.g., the output block in the decoder Corresponding deconvolution module at the scaling stage, output block in the decoder The corresponding deconvolution module of the scale stage), Represents an output block in the decoder.

[0154] Alternatively, see Figure 5 ,The specific process of the multi-source data fusion module includes:

[0155]

[0156]

[0157] in, ReLU represents the convolutional layer, ReLU represents the activation function, Concat[] represents the feature concatenation operation, and Norm represents normalization. The encoder and decoder in the visible light branch are connected to the output block of the multi-source data fusion module. It represents the output block of the encoder and decoder connected to the multi-source data fusion module in the infrared image auxiliary branch. Then the mean and standard deviation of each channel in the target contrast feature map are calculated:

[0158]

[0159] in, and represents the value calculated in the channel domain P The mean and standard deviation of H and W express P , and then at the element level Q Add to the mean, W Add to the standard deviation to get the feature map L and J :

[0160]

[0161] Will J With the instantiated auxiliary contrast feature map Multiply to get Z , the feature map L Add element by element Z The final matching feature map is obtained M :

[0162]

[0163] The mapped and matched feature map M and the target contrast feature map P are then aggregated through continuous up- and down-sampling residual compensation:

[0164]

[0165]

[0166]

[0167]

[0168] in, represents the convolutional layer, ReLU represents the activation function, and Concat[] represents the feature concatenation operation. Represents a transposed convolutional layer.

[0169] Optionally, L The 1 pixel loss formula is expressed as:

[0170] The technical solution of this example can quickly and accurately extract hidden high-frequency information from the blurred visible light image through the high-frequency attention block in the image reconstruction algorithm when mining vehicles face many challenges in the mining road environment, such as blurred visible light images caused by extremely harsh environments such as high dust, mud and night working environments. The enhanced image obtained by the subsequent image reconstruction process based on the extracted high-frequency information can facilitate subsequent accurate perception of the mining environment based on the enhanced image, so as to ensure the accuracy and effectiveness of subsequent analysis and decision-making using the enhanced image.

[0171] In the technical solution of this example, in addition to using visible light images in the image reconstruction process, the image reconstruction algorithm also adds infrared images as auxiliary modal information, using the characteristics of infrared images that have strong thermal information capture capabilities, strong penetration capabilities, and are not affected by light, to supplement the feature information that is difficult to perceive in visible light images for mining vehicles in harsh environments such as night and fog, so as to obtain enhanced images. In addition, through the multi-source data fusion module in the image reconstruction algorithm, the features extracted from visible light images and infrared images can be interactively fused, alleviating the problem of low image quality and inability to provide important information due to the extreme dependence of visible light images on light sources and weak penetration capabilities, so that the enhanced image obtained after completing the image reconstruction process can accurately and effectively reflect the characteristics of roads, vehicles, personnel, depth, etc.

[0172] The mining vehicle of this embodiment is based on the camera system using the image reconstruction algorithm to realize the framework of various functions, for example, see Figure 6 , the visible light camera and infrared camera in the camera system are used to capture visible light images and infrared images respectively, and the visible light image and infrared image are fused and reconstructed through the image reconstruction algorithm to obtain an enhanced image, which is fed back to the monitoring equipment of the mining vehicle to realize the real-time display of the mine road conditions, and the enhanced image is input into the algorithm models of various functions to realize the functions of obstacle avoidance, recognition, depth estimation, etc.

[0173] The technical solution of this example can execute a fusion deblurring and defogging image reconstruction strategy based on multi-source images of mines taken under harsh environments. This strategy utilizes the characteristics of infrared images such as thermal information capture, strong penetration ability, and being unaffected by light, and fuses them with visible light images as an auxiliary modality. Combined with the high-resolution characteristics of visible light images, a complete and clear target image (enhanced image) can be effectively reconstructed. Based on this target image, mining vehicles can have more reliable environmental perception capabilities in harsh environments such as night and fog, and can provide strong support for subsequent downstream tasks such as image recognition and depth estimation.

[0174] The high-frequency attention block in the image reconstruction algorithm of this example includes a frequency domain branch and a spatial branch. In the frequency domain branch, the model's utilization of global information is improved by mining high-frequency details and using Fourier convolution to update the global value of the image. In the spatial branch, information perception is performed using convolution blocks with different receptive fields, which promotes the complementary fusion of multi-scale information, thereby improving the model's fitting ability and learning a more reasonable mapping relationship.

[0175] The fusion module for infrared images and visible light images (i.e., multi-source data fusion module) in the image reconstruction algorithm of this example can extract information such as the mean and variance of the infrared image, and perform interactive calculations with the visible light image, ultimately achieving complementary fusion between different modal information.

[0176] Second embodiment See also Figure 7 , is an image processing method provided in the second embodiment of the present application. The method can be executed by a computing device provided in the embodiment of the present application. The computing device can be implemented in software and / or hardware.

[0177] This embodiment provides an image processing method, including the following steps (for example, step S21 to step S22): Step S21: Acquire the visible light image and the auxiliary modality image to be fused.

[0178] Step S22: Use an image reconstruction algorithm to fuse and reconstruct the visible light image and the auxiliary modality image to obtain a target image, wherein the image reconstruction algorithm includes a step of using the high-frequency attention mechanism and the Fourier convolution mechanism in the high-frequency attention block to complete the feature extraction task of the associated visible light image.

[0179] In one implementation, the specific implementation of the image reconstruction algorithm or the high-frequency attention block in this embodiment can refer to the image reconstruction algorithm provided in the first embodiment, and will not be described in detail here.

[0180] In the technical solution of this embodiment, the high-frequency attention mechanism and Fourier convolution mechanism in the high-frequency attention block are used to complete the feature extraction task of the associated visible light image. Therefore, the image reconstruction algorithm can realize the rapid and effective extraction of high-frequency information in visible light images of various image qualities, so that the target image obtained by fusing and reconstructing the visible light image and the auxiliary modality image based on the effectively extracted high-frequency information has clearer features in the visible light image and the auxiliary modality image. Therefore, the technical solution of this embodiment can ensure that images with better quality are obtained when shooting in various shooting environments.

[0181] Based on the same inventive concept as the above embodiments, the present application embodiment provides a computing device, such as Figure 8 As shown, the device includes: a processor 310 and a memory 311 storing a computer program; wherein, Figure 8 The processor 310 shown in the figure is not used to indicate that the number of the processor 310 is one, but is only used to indicate the positional relationship of the processor 310 relative to other devices. In actual applications, the number of the processor 310 may be one or more; similarly, Figure 8The memory 311 shown in the figure has the same meaning, that is, it is only used to refer to the position relationship of the memory 311 relative to other devices. In practical applications, the number of memories 311 can be one or more. When the processor 310 runs the computer program, an image reconstruction algorithm or image processing method applied to the above device is implemented.

[0182] The device may also include: at least one network interface 312. The various components in the device are coupled together via a bus system 313. It is understood that the bus system 313 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 313 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 8 Various buses are labeled as bus system 313.

[0183] The memory 311 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 311 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memories.

[0184] The memory 311 in the embodiment of the present application is used to store various types of data to support the operation of the device. Examples of these data include: any computer program used to operate on the device, such as an operating system and an application program. Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program may include various applications, such as a media player (MediaPlayer), a browser (Browser), etc., which are used to implement various application services. Here, a program that implements the image reconstruction algorithm or image processing method provided by the present application may be included in the application program.

[0185] The embodiment of the present application provides an engineering machine equipped with the above-mentioned computing device. The engineering machine may be a vehicle, etc. Optionally, the vehicle may be an engineering vehicle that often faces harsh environments, such as a mining vehicle.

[0186] The engineering machinery of this embodiment includes traditional engineering machinery vehicles and also includes new energy vehicles used in the field of engineering machinery, such as new energy mixer trucks, new energy pump trucks, new energy excavators, etc.; in addition, the engineering machinery vehicles of this embodiment also belong to intelligent networked vehicles. The engineering machinery vehicles include sensor / perception systems, communication systems, etc., and collect vehicle operation data and vehicle surrounding environment information through the sensor / perception system in the vehicle. The network connection with other vehicles and the cloud is realized through the communication system, and the collected vehicle operation data, vehicle surrounding environment information, etc. are shared to the cloud and other authorized vehicles to realize data sharing, remote analysis, intelligent driving and other operations.

[0187] Based on the same inventive concept as the above-mentioned embodiment, this embodiment further provides a computer-readable storage medium, in which a computer program is stored. The computer-readable storage medium may be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it may also be various devices including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc. When the computer program stored in the computer-readable storage medium is executed by the processor, the above-mentioned image reconstruction algorithm or image processing method is implemented. For the specific steps implemented when the computer program is executed by the processor, please refer to Figure 1 or Figure 7 The description of the illustrated embodiment will not be repeated here.

[0188] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0189] In this document, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion of elements other than those listed and may also include additional elements not expressly listed.

[0190] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image reconstruction algorithm, characterized in that: include: In the first branch model for processing the visible light image, the visible light initial image obtained by the first encoder at the current scale stage is subjected to feature extraction through a high-frequency attention block and / or down-sampled through a convolution layer to obtain a visible light feature map of a specific scale corresponding to the current scale stage, the high-frequency attention block uses a high-frequency attention mechanism and a Fourier convolution mechanism to complete the feature extraction task, the visible light initial image is an initial feature map obtained after shallow feature extraction of the visible light image, or a feature map to be processed obtained by the first encoder at the previous scale stage; Performing feature fusion on the visible light feature map of the specific scale and the auxiliary feature map of the corresponding scale to obtain a first fused feature map of the specific scale, wherein the auxiliary feature map of the corresponding scale is output by the second branch model for processing the auxiliary modality image; Acquire the to-be-processed feature map of the specific scale according to the visible light feature map of the specific scale and / or the first fused feature map; According to the feature map to be processed at the specific scale, enter the next scale stage of the first encoder, and / or use the first decoder in the first branch model to perform feature processing to obtain a final processing result, and perform image reconstruction according to the final processing result to obtain a target image.

2. The image reconstruction algorithm according to claim 1, characterized in that: The auxiliary modality image is one of an infrared image, a radar image, and a laser radar point cloud image; and / or, the high-frequency attention block includes a frequency domain branch and a spatial branch; For the visible light initial image obtained by the first encoder at the current scale stage, feature extraction is performed through the high-frequency attention block, including: In the frequency domain branch, high-frequency information of the visible light initial image is mined, and the high-frequency information is converted to the frequency domain by fast Fourier transform, so as to globally update the frequency domain data corresponding to the visible light initial image, and the updated frequency domain data is converted to the spatial domain by inverse fast Fourier transform, so as to obtain first spatial domain data; In the spatial branch, local information of the visible light initial image is extracted, and feature processing is performed on the local information in the spatial domain, local structural features on the visible light initial image are learned in a multi-scale learning manner, and channel adaptive processing is performed to obtain second spatial domain data; After fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels, feature extraction information is output.

3. The image reconstruction algorithm according to claim 2, characterized in that: In the frequency domain branch, the step of mining high-frequency information of the visible light initial image comprises: According to the number of channels of the visible light initial image, the feature layer of the visible light initial image is divided into a first feature layer and a second feature layer, and the layers are input into a first processing sub-branch and a second processing sub-branch respectively; In the first processing sub-branch, local feature extraction is performed on the first feature layer through a 3×3 convolution layer, and feature processing is performed using a 1×1 convolution layer and a GELU activation function to obtain a first processing result; In the second processing branch, high-frequency features are extracted from the second feature layer through a maximum pooling layer, and feature processing is performed using a 1×1 convolution layer and a GELU activation function to obtain a second processing result; Aggregating the first processing result and the second processing result in a channel dimension to obtain the high-frequency information; and / or, In the spatial branch, extracting local information of the visible light initial image, performing feature processing on the local information in the spatial domain, learning local structural features on the visible light initial image in a multi-scale learning-based manner, and performing channel adaptive processing to obtain a second spatial domain data step, including: Extracting the local information through a 3×3 convolution layer, establishing long-distance dependencies using a 5×5 depth convolution and a 5×5 depth dilated convolution, adjusting the relationship between feature layers through a 1×1 convolution layer, and adding an SE module to adaptively capture key feature layer information to obtain the second spatial domain data; and / or, The step of fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels and then outputting feature extraction information comprises: The first spatial domain data and the second spatial domain data are fused and the number of channels is adjusted through a 1×1 convolution layer to obtain and output the feature extraction information.

4. The image reconstruction algorithm according to claim 3, characterized in that: In the frequency domain branch, the step of mining the high-frequency information of the visible light initial image corresponds to the mathematical expression: in, represents the convolutional layer, represents the activation function, represents the maximum pooling operation, represents a concatenation operation along the channel dimension; and / or, The high-frequency information is converted to the frequency domain by fast Fourier transform to globally update the frequency domain data corresponding to the initial visible light image, and the updated frequency domain data is converted to the spatial domain by inverse fast Fourier transform to obtain the first spatial domain data. The corresponding mathematical expression is: in, represents the convolutional layer, represents the fast Fourier transform, represents the inverse Fourier transform; and / or, The steps of extracting the local information through a 3×3 convolution layer, establishing long-distance dependencies using a 5×5 depth convolution and a 5×5 depth dilated convolution, adjusting the relationship between the feature layers through a 1×1 convolution layer, and adding an SE module to adaptively capture key feature layer information to obtain the second spatial domain data, correspond to the following mathematical expressions: in, represents the convolutional layer, represents the depthwise convolution, represents the depth-dilated convolution, represents an SE module; and / or, The step of fusing the first spatial domain data and the second spatial domain data and adjusting the number of channels through a 1×1 convolution layer to obtain and output the feature extraction information corresponds to the following mathematical expression: in, Represents a convolutional layer.

5. The image reconstruction algorithm according to claim 1, characterized in that: According to the feature map to be processed of the specific scale, feature processing is performed using the first decoder to obtain a final processing result, including: Using the first decoder, splicing the feature map to be processed at the specific scale and the visible light feature map of the corresponding scale output by the first encoder to obtain a spliced ​​feature map of the specific scale; For the spliced ​​feature map of the specific scale, feature enhancement is performed through a fusion learning block and / or up-sampling is performed through a deconvolution layer to obtain a reconstructed feature map of the specific scale; Performing feature fusion on the reconstructed feature map of the specific scale and the auxiliary reconstructed feature map of the corresponding scale to obtain a second fused feature map of the specific scale, wherein the auxiliary reconstructed feature map of the corresponding scale is output by the second branch model; Obtaining a to-be-processed reconstructed feature map of a specific scale according to the reconstructed feature map of a specific scale and / or the second fused feature map; The to-be-processed reconstructed feature map of the specific scale is processed in the first branch model to obtain the final processing result.

6. The image reconstruction algorithm according to claim 5, characterized in that: The workflow of the second branch model to obtain the auxiliary feature map includes: In the second branch model, the auxiliary modality initial image obtained by the second encoder is subjected to feature extraction through a residual group and / or down-sampled through a convolutional layer to obtain an auxiliary feature map of the specific scale, wherein the auxiliary modality initial image is an initial auxiliary feature map obtained after shallow feature extraction of the auxiliary modality image, or an auxiliary feature map obtained by the second encoder at a previous scale stage; and / or, The workflow of the second branch model to obtain the auxiliary reconstruction feature map includes: In the second branch model, the auxiliary feature map output by the second encoder obtained by the second decoder is feature enhanced through a residual group and / or up-sampled through a deconvolution layer to obtain the auxiliary reconstructed feature map of the specific scale.

7. The image reconstruction algorithm according to claim 5, characterized in that: The step of performing image reconstruction according to the final processing result to obtain a target image comprises: Performing linear projection on the final processing result to obtain a reconstructed image; According to the reconstructed image, using the minimization L After optimizing the network parameters with a 1 pixel loss, the target image is obtained.

8. An image processing method, characterized in that: include: Acquire a visible light image and an auxiliary modality image to be fused; The visible light image and the auxiliary modality image are fused and reconstructed using an image reconstruction algorithm to obtain a target image, wherein the image reconstruction algorithm includes the step of using a high-frequency attention mechanism and a Fourier convolution mechanism in a high-frequency attention block to complete a feature extraction task associated with the visible light image.

9. A computing device, characterized in that include: A processor and a memory storing a computer program, when the processor runs the computer program, the steps of the image processing method according to claim 8 are implemented.

10. A vehicle, characterized in that: A computing device as claimed in claim 9 is provided.

11. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the steps of the image processing method according to claim 8 are implemented.

Images