Infrared and visible dual-band image registration and fusion detection method based on brain-like computing platform deployment

Abstract

The application discloses an infrared and visible light dual-band image registration and fusion detection method based on a brain-like computing platform, and relates to the technical field of image processing. The method comprises the following steps: acquiring an infrared image and a visible light image of the same scene and the same time; extracting feature points and feature descriptions of the infrared image and the visible light image based on a trained feature extraction network; inputting the feature points and the feature descriptions of the infrared image and the visible light image into a trained feature point matching network to generate a feature point matching result; performing image registration on the infrared image and the visible light image based on the feature point matching result; inputting the infrared image and the visible light image after image registration into a trained image fusion network to obtain a fusion image; and inputting the fusion image into a trained target detection network to obtain target detection information. The application can realize the registration efficiency and accuracy of infrared images and visible light images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to an infrared and visible light dual-band image processing method and a brain-inspired computing platform, or an infrared and visible light dual-band image registration and fusion detection method deployed based on the brain-inspired computing platform. Background Art

[0002] With the continuous development and advancement of optoelectronic imaging technology, image sensors can capture information over a wider spectral range, significantly increasing the richness of information and expanding its application areas. However, due to the limited spectral range of a single image sensor, the information it can capture is limited and increasingly unable to meet the practical requirements for a comprehensive and accurate description of scenes and targets. Therefore, the comprehensive processing and fusion of images acquired by at least two imaging devices operating in different spectral bands or using different imaging principles can further facilitate a comprehensive description of the scene and the detection and identification of targets.

[0003] Infrared and visible light imaging differ significantly in their imaging principles, wavelength range, and information content. While each possesses unique characteristics, they also offer significant complementary advantages. Infrared imaging is performed by infrared sensors based on infrared radiation emitted or reflected by objects, with the imaging band ranging from 0.75 to 14 microns. Infrared light can penetrate smoke, haze, and some non-metallic materials to a certain extent, enabling superior imaging in low-visibility conditions (such as at night or in smog-filled areas). Consequently, infrared imaging is independent of ambient lighting and can be performed day or night, offering full-time operation. However, due to the longer wavelength of infrared light, the resolution of infrared images is typically lower than that of visible light images. Furthermore, the pixel density of infrared sensors is lower, limiting their ability to capture detail. Visible light imaging is passive imaging, with a wavelength range of 370 to 730 nanometers. It incorporates information about the scene's sunlight reflections. These images offer sharp details, high resolution and contrast, and rich color information, making them more easily perceived by the human eye. However, the imaging process is susceptible to interference from environmental factors, leading to distortion and limited 24 / 7 operation.

[0004] Therefore, by using image registration and fusion technology to fuse visible light images and infrared images and comprehensively utilize information, the advantages of the two can be complementary. Not only can the thermal radiation distribution of important targets and backgrounds be obtained from infrared images, but also clear background details and texture information can be obtained from visible light images. This can provide a more comprehensive description of scenes and targets, which is more helpful for environmental monitoring and target detection and identification, and has very important practical needs and application value.

[0005] Currently, traditional image registration methods are mainly divided into registration algorithms based on image grayscale statistics, registration methods based on image transform domains, and registration methods based on image features. However, in the actual registration process, a considerable number of feature points ultimately cannot be matched due to various differences, becoming useless feature points, which consumes a large amount of computational time and reduces registration efficiency. Furthermore, registration algorithms have difficulty in completing multi-source image registration. Many current classic registration methods are designed for images with similar bands, but infrared and visible light images often have large differences in resolution and local grayscale. As a result, many classic methods are basically ineffective when applied to the registration of infrared and visible light images.

[0006] Traditional image fusion methods primarily include transform-domain, spatial-domain, and deep learning-based methods. Traditional algorithms require manual design of fusion rules, resulting in poor applicability, a waste of manpower, and limited real-time performance. Because the fusion task is unsupervised, current deep learning-based algorithms lack appropriate guidance, leading to an imbalance in preserving visible light features while preserving infrared features. This can lead to issues such as insufficient sharpness in visible light details and inconspicuousness of infrared targets. Consequently, traditional image fusion algorithms struggle to strike a balance between preserving infrared target features and visible light background details, hindering the presentation of fusion results and the accuracy of subsequent object detection. Furthermore, traditional fusion and detection algorithms require a large amount of registration data as input and training data. Due to differences in resolution and modality between infrared and visible light images, errors can be significant, either through camera parameters or through traditional registration algorithms, hindering subsequent fusion and detection tasks.

[0007] For target detection tasks, the detection accuracy of single-modal visible light image targets is extremely susceptible to influence in complex backgrounds, such as those with occlusion, complex backgrounds, or weak illumination, and is prone to missed detections or false detections. For multimodal target detection, the information between different modalities cannot be fully utilized, and the detection accuracy may decrease rather than increase compared to single-modal detection. Therefore, how to integrate multimodal information has also become the key to target detection tasks. Summary of the Invention

[0008] In view of this, the present invention provides an infrared and visible light dual-band image processing method and a brain-like computing platform, or an infrared and visible light dual-band image registration and fusion detection method deployed based on a brain-like computing platform, which can achieve the registration efficiency and accuracy of infrared images and visible light images.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] In a first aspect, the present invention provides an infrared and visible light dual-band image processing method, the method comprising:

[0011] Acquire infrared images and visible light images of the same scene at the same time;

[0012] Feature points and feature descriptions of infrared images and visible light images are extracted based on a trained feature extraction network, respectively; wherein the feature extraction network includes a feature encoder, a feature point decoder, and a feature description decoder; the feature encoder includes at least two consecutive downsampling layers; the feature point decoder includes a first convolutional layer, a first activation function layer, and a graph transformation layer connected in sequence; the feature description decoder includes a second convolutional layer, an upsampling layer, and a regularization layer connected in sequence; the input end of the last downsampling layer of the at least two consecutive downsampling layers is used to input the infrared image and the visible light image; the input end of the first convolutional layer and the input end of the second convolutional layer are both connected to the output end of the last downsampling layer of the at least two consecutive downsampling layers;

[0013] The feature points and feature descriptions of the infrared image and the visible light image are input into the trained feature point matching network to generate feature point matching results; wherein the feature point matching network includes at least one continuous feature point matching layer, each feature point matching layer includes two self-attention units, one cross-attention unit and two confidence calculation units; the input end of one of the two self-attention units is used to input the feature points and feature description of the infrared image, and the input end of the other self-attention unit is used to input the feature points and feature description of the visible light image; the output ends of the two self-attention units are connected to the input end of the cross-attention unit, and the two output ends of the cross-attention unit are respectively connected to the input ends of the two confidence calculation units; if the confidence output by the two confidence calculation units is greater than a preset confidence threshold, then exit the current feature point matching layer, calculate the similarity matrix, and determine the matching feature points; otherwise, remove the feature points whose confidence is not greater than the preset confidence threshold, and input the remaining feature points and corresponding feature descriptions into the next feature point matching layer of the current feature point matching layer;

[0014] Perform image registration on the infrared image and the visible light image based on the feature point matching results;

[0015] Inputting the registered infrared image and visible light image into a trained image fusion network to obtain a fused image; wherein the image fusion network includes at least two consecutive fusion submodules, each fusion submodule includes a third convolutional layer and a third activation function layer, and the fusion submodules of the at least two consecutive fusion submodules except the last fusion submodule also include a batch normalization layer, and the batch normalization layer is respectively connected to the third convolutional layer and the third activation function layer;

[0016] The fused image is input into the trained target detection network to obtain target detection information; wherein, the target detection information includes target positioning information and target type information, and the target detection network includes a target positioning network and a type detection network; the target positioning network includes a convolutional neural network, a first region proposal network, a first pooling layer, and a first full convolutional layer connected in sequence; the input end of the convolutional neural network is used to input the fused image, and the output end of the first full convolutional layer is used to output the target positioning information; the type detection network includes a first residual network, a second region proposal network, a second pooling layer, a second residual network, and a second full convolutional layer connected in sequence; the input end of the first residual network inputs the image block, and the second full convolutional layer is used to output the target type information, and the image block is obtained by mapping the target positioning information back to the fused image and cropping it.

[0017] The method provided by the present invention utilizes deep learning algorithms to combine the features of both infrared and visible light modalities, enabling more comprehensive acquisition of useful information from both. This allows for high-precision heterogeneous image registration, high-information fusion, and all-weather, omnidirectional target detection. Forming a task flow for at least two visual tasks—feature point extraction, registration, fusion, and target detection—can improve the accuracy of each task, addressing their respective shortcomings and resolving the issue of large amounts of data required for deep network model training, while also avoiding issues like overfitting and poor generalization performance. Furthermore, the method organically combines at least two visual tasks, enhancing the accuracy of the final detection.

[0018] In a possible implementation, before extracting feature points and feature descriptions of the infrared image and the visible light image based on the trained feature extraction network, the following is further included:

[0019] Get the first training set;

[0020] Training the feature extraction network based on the first training set and the first loss function to obtain a trained feature extraction network;

[0021] Among them, the formula of the first loss function is:

[0022] L(X,X′,D,D′;Y,Y′S)=L p (X, Y) + L p (X′, Y′)+λL d (D, D′, S);

[0023] Where X is the input image; Y is the true label of the feature point; D is the feature description; X′ is the new input image generated by randomly changing the input image by the H matrix; Y′ is the true label of the new feature point generated by randomly changing the input image by the H matrix; D′ is the new feature description generated by randomly changing the input image by the H matrix; L p is the feature point loss; Ld is the feature description loss; λ is the weight parameter; S is the guide term of the homography matrix.

[0024] In a possible implementation, before inputting the feature points and feature descriptions of the infrared image and the visible light image into the trained feature point matching network to generate feature point matching results, the method further includes:

[0025] Get the second training set;

[0026] Training the feature point matching network based on the second training set and the second loss function to obtain a trained feature point matching network;

[0027] Among them, the formula of the second loss function is:

[0028]

[0029] Where σ is the score of the matching probability output by the feature point matching network; l is the number of layers when exiting the current feature point matching layer; L is the output result of the Lth feature point matching layer; M is the matching result of the true label obtained by matching the feature point sets A and B of the infrared image and the visible light image when the transformation matrix or the relative position between the cameras is known; and are the unmatched feature points in the feature point sets A and B of the infrared image and the visible light image respectively; l P ij is the similarity matrix corresponding to the l-th layer feature point matching layer; is the matching probability of the i-th feature point of the image to which the feature point set A belongs in the l-th feature point matching layer; is the matching probability of the jth feature point in the image to which the feature point set B belongs at the lth layer.

[0030] In one possible implementation, performing image registration on the infrared image and the visible light image based on the feature point matching result includes:

[0031] Setting source images and target images for infrared images and visible light images;

[0032] The thin template interpolation method is used to determine the registration parameters that minimize the error between the registration points on the source image and the corresponding points on the target image;

[0033] The source image is mapped to the target image based on the registration parameters to obtain the registration results of the infrared image and the visible light image.

[0034] In a possible implementation, before inputting the registered infrared image and the visible light image into a trained image fusion network to obtain a fused image, the method further includes:

[0035] Get the third training set;

[0036] Training the image fusion network based on the third training set and the third loss function to obtain a trained image fusion network;

[0037] Among them, the formula of the third loss function is:

[0038]

[0039] Where, I1 and I2 are the images to be fused; I f is the fused image; φ j (x),φ j (y) are the jth set of feature maps extracted by the feature encoder; H and W are the length and width of the input image respectively; H j 、W j are the length and width of the jth group of features extracted by the feature encoder; For image I f Structural similarity with image I1; For image I f Structural similarity with image I2; is the loss of structural consistency; is the feature loss; is the gradient loss; are the gradients of input x and y respectively; α, β, w1 and w2 are weight parameters.

[0040] In a possible implementation, before inputting the fused image into the trained object detection network to obtain object detection information, the following steps are further included:

[0041] Get the fourth training set;

[0042] Training the object detection network based on the fourth training set and the fourth loss function to obtain a trained object detection network;

[0043] Among them, the formula of the fourth loss function is:

[0044] Loss = L CIOU +L fl ;

[0045] L CIOU is the position loss of the target box, and the formula is:

[0046]

[0047] Where p is the center point of the predicted box; g is the center point of the real box; c is the diagonal length of the minimum enclosing rectangle of the predicted box and the real box; IoU is the intersection over union of the predicted box and the real box; v is the distance between the aspect ratio of the predicted box and the real box. wg and h g is the length and width of the real frame, w p and h p is the length and width of the prediction box; ρ 2 (p, g) is the Euclidean distance between the center points of the predicted box and the true box; α is the balance coefficient,

[0048] L fl is the classification loss, and the formula is:

[0049] L fl =-((lp) 2 q log(p)+p 2 (1-q)log(1-p));

[0050] Where p and q are the classification prediction vectors of the candidate samples and the corresponding predicted true values.

[0051] In a second aspect, the present invention provides a brain-inspired computing platform, comprising an image acquisition unit, a main control unit, a switching unit, and at least two many-core computing units;

[0052] An image acquisition unit, connected to the main control unit, is used to synchronize and align the collected visible light image and infrared image to form a video stream;

[0053] The main control unit is connected to the image acquisition unit and the switching unit respectively, and is used to send the video stream to the switching unit;

[0054] a switching unit, connected to the main control unit and the at least two many-core computing units, respectively, configured to generate at least two image processing tasks according to the video stream, distribute the at least two image processing tasks to the at least two many-core computing units for image processing, and send image processing results returned by the many-core computing units to the main control unit;

[0055] The many-core computing unit includes at least one brain-like computing chip connected to the switching unit; the brain-like computing chip stores and executes a computer program or computer instructions of the infrared and visible light dual-band image processing method of the first aspect and any possible implementation thereof.

[0056] For the specific description of the second aspect and its various implementations in the present invention, reference can be made to the detailed description of the first aspect and its various implementations; and for the beneficial effects of the second aspect and its various implementations, reference can be made to the analysis of the beneficial effects of the first aspect and its various implementations, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0058] Figure 1 1 is a flow chart of an infrared and visible light dual-band image processing method according to an embodiment of the present invention;

[0059] Figure 2 is a structural diagram of a feature extraction network according to an embodiment of the present invention;

[0060] Figure 3 is a structural diagram of a feature point matching network according to an embodiment of the present invention;

[0061] Figure 4 is a structural diagram of an image fusion network according to an embodiment of the present invention;

[0062] Figure 5 is a structural diagram of a target detection network according to an embodiment of the present invention;

[0063] Figure 6 This is a structural block diagram of a brain-inspired computing platform according to an embodiment of the present invention;

[0064] Figure 7 This is a diagram of the minimum functional core architecture of a brain-inspired computing chip according to an embodiment of the present invention;

[0065] Figure 8 This is a diagram of an artificial neural network algorithm configuration process according to an embodiment of the present invention;

[0066] Figure 9 This is a process diagram of an artificial neural network streaming topology mapping according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0068] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0069] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0070] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0071] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0072] Example 1:

[0073] The embodiment of the present invention provides an infrared and visible light dual-band image processing method or an infrared and visible light dual-band image registration and fusion detection method based on a brain-like computing platform. Figure 1 , which is a flow chart of the infrared and visible light dual-band image processing method provided by an embodiment of the present invention. Figure 1 As shown, the infrared and visible light dual-band image processing method provided by the present invention mainly includes the following steps (steps S101 to S106):

[0074] Step S101, acquiring an infrared image and a visible light image of the same scene at the same time;

[0075] Step S102, extracting feature points and feature descriptions of the infrared image and the visible light image respectively based on the trained feature extraction network;

[0076] Step S103, inputting the feature points and feature descriptions of the infrared image and the visible light image into the trained feature point matching network to generate feature point matching results;

[0077] Step S104, performing image registration on the infrared image and the visible light image based on the feature point matching result;

[0078] Step S105, inputting the registered infrared image and visible light image into the trained image fusion network to obtain a fused image;

[0079] Step S106: input the fused image into the trained target detection network to obtain target detection information.

[0080] In this embodiment, the resolutions of the infrared image and the visible light image are generally different. For example, the resolution of the infrared image is 640*512, and the resolution of the visible light image is 1280*720.

[0081] For step S102, the embodiment of the present invention provides a feature extraction network. Figure 2 , which is a structural diagram of the feature extraction network provided by the embodiment of the present invention. Figure 2 As shown, the feature extraction network includes a feature encoder 201, a feature point decoder 202 and a feature description decoder 203.

[0082] Among them, the structure of the feature encoder 201 is similar to that of vgg16 (Visual Geometry Group 16, a deep convolutional neural network model), which includes at least two consecutive downsampling layers 2011, which are used to reduce the size of infrared images and visible light images, for example, to 1 / 8 of the original input size, so as to reduce the amount of calculation, speed up the operation, and extract multi-scale features to construct a feature pyramid.

[0083] The feature point decoder 202 includes a first convolutional layer 2021, a first activation function layer 2022, and a graph transformation layer 2023, which are connected in sequence. The first convolutional layer 2021 can further extract feature points and feature descriptions from the reduced-size infrared image and visible light image output by the feature encoder 201, and locate the positions of the feature points based on this information, obtaining feature points of dimension D1 (e.g., D1 = 65). After passing through the first activation function layer 2022 and the graph transformation layer 2023, a feature point distribution map with the same size and dimension as the original input is obtained, with a size between 0 and 1. The closer to 1, the greater the probability of representing a feature point.

[0084] Optionally, the first convolution layer 2021 is a single-layer convolution; the first activation function layer 2022 can adopt a softmax activation function; the graphic transformation layer 2023 can adopt resizing, upsampling, etc.

[0085] The feature description decoder 203 includes a second convolutional layer 2031, an upsampling layer 2032, and a regularization layer 2033, which are connected in sequence. The input of the first convolutional layer 2021 and the input of the second convolutional layer 2031 are both connected to the output of the last downsampling layer 2011 of at least two consecutive downsampling layers 2011. The feature description output by the feature encoder 201 passes through the second convolutional layer 2031 of the feature description decoder 203 to obtain a feature description of D2 dimensions (for example, D2 = 256). After passing through the upsampling layer 2032 and the regularization layer 2033, a feature description of D2 dimensions, the same as the original input size, is obtained.

[0086] Optionally, the second convolution layer 2031 is a single-layer convolution; the upsampling layer 2032 can use bicubic interpolation; the regularization layer 2033 can use L2 regularization; the number of downsampling layers is set to 4, that is, N=4.

[0087] It should be noted that the size in this embodiment is the aspect ratio of the feature map.

[0088] In some optional embodiments, for training the feature extraction network, a first training set is first obtained; then the feature extraction network is trained based on the first training set and the first loss function to obtain a trained feature extraction network.

[0089] Optionally, the Synthtic Shapes dataset (a synthetic dataset for shape recognition tasks) can be used as a dataset to train the network to extract points of various corners, lines, and cubes.

[0090] The first loss function mainly includes feature point loss and feature description loss, where X is the input image, Y is the ground-truth label of the feature point, and D is the feature description.

[0091] To improve the robustness of the network, the input image is randomly transformed by the H matrix to obtain the updated input image X′, the ground-truth label Y′ of the feature point, and the feature description D′. The final first loss function can be written as follows:

[0092] L(X,X′,D,D′;Y,Y′,S)=Lp(X,Y)+L p (X′, Y′)+λL d (D, D′, S) Formula (1);

[0093] In formula (1), L p is the feature point loss; L d is the feature description loss; λ is the weight parameter, which can be set to 10 to maintain a balance between feature point loss and feature description loss.

[0094] The feature point loss L is p and feature description loss L d The calculation of is described in detail.

[0095] L p is the image block x hw ∈X, with the full convolutional cross entropy loss of the image patch x hw The corresponding label is y hw (1 if there is a feature point, 0 if there is no feature point), feature point loss L p The formula is:

[0096]

[0097] In the above formula, X hw H is the pixel at position (h, w) output by the feature point decoder; C 、W C The length and width of the feature point decoder output; x hwk The position of the feature point decoder output is (h, w) and the pixel corresponding to the k-th dimension channel; x hwy is x hw The total label (1 for feature points and 0 for no feature points).

[0098] For the feature description loss L d , similarly, in an image block d hw ∈D and d′ h′w′ ∈D′, in the image patch at position (h, w) and the image patch at position (h′, w′), S is calculated by the following formula:

[0099]

[0100] In formula (4), S is the guide term of the homography matrix; H is the homography matrix; P hw is the center point of the image block; is P multiplied by H hw And divided by the final coordinate, the final coordinate refers to the corresponding transformed coordinate, that is, the coordinate of the Euclidean distance is transformed to the secondary coordinate. Then the final feature description loss L d The calculation formula is as follows:

[0101]

[0102] In formula (5), d hw is the feature description of the position (h, w), d′ h′w′ is the feature description at position (h′, w′).

[0103] l d (d, d′; s) = λ d *s*max(0,m p -d T d′)+(1-s)*max(0,d T d′-m n ) formula (6);

[0104] In formula (6), d and d' are the feature descriptions of infrared image and visible light image respectively; m p Taking 1 is the positive margin; m n Taking 0.2 is a negative margin; s is the same as S; λ d is the weight, which can be 250.

[0105] For step S103, the embodiment of the present invention provides a feature point matching network. Figure 3 , which is a structural diagram of the feature point matching network provided by an embodiment of the present invention. Figure 3 As shown, the feature point matching network includes at least one continuous feature point matching layer, each feature point matching layer includes two self-attention units, one cross-attention unit and two confidence calculation units.

[0106] The input end of one of the two self-attention units is used to input the feature point P of the infrared image A and feature description d A , the input of the other self-attention unit is used to input the feature point P of the visible light image B and feature description d B .

[0107] The outputs of the two self-attention units are connected to the input of the cross-attention unit, and the two outputs of the cross-attention unit are connected to the inputs of the two confidence calculation units respectively. The cross-attention unit outputs the feature representation of the new feature point.

[0108] One confidence calculation unit performs a confidence calculation on the feature representation of the new feature point and the feature description of the infrared image to obtain a first confidence level. The other confidence calculation unit performs a confidence calculation on the feature representation of the new feature point and the feature description of the visible light image to obtain a second confidence level. If both the first and second confidence levels are greater than a preset confidence threshold, the current feature point matching layer is exited, and a similarity matrix is ​​calculated to determine the matching feature points. Otherwise, feature points with confidence levels less than the preset confidence threshold are removed, and the remaining feature points and their corresponding feature descriptions are entered into the next feature point matching layer of the current feature point matching layer.

[0109] For feature point matching networks, in order to reduce errors during training, we can use style transfer to generate strictly matched pseudo infrared images on the COCO dataset (Common Objects in Context, a large-scale target detection, segmentation and description dataset) and the IVS dataset (Iris Verification in the Wild, a dataset for iris recognition) for training. The loss is mainly the loss between the matching matrix P and the true label ground truth. When the transformation matrix or the relative position between cameras is known, we can match the point sets A and B to obtain the matching result M of the true label ground truth. The points in the feature point set that fail to match are denoted as By optimizing the form of log-likelihood probability, the matching can be completed as much as possible in the first layer of the network, allowing the network to predict the corresponding matching relationship as early as possible.

[0110] Specifically, for the training of the feature point matching network, a second training set is first obtained; then the feature point matching network is trained based on the second training set and the second loss function to obtain a trained feature point matching network.

[0111] Among them, the second loss function is defined as follows:

[0112]

[0113] In formula (7), σ is the score of the matching probability output by the feature point matching network; l is the number of layers when exiting the current feature point matching layer; L is the output result of the Lth feature point matching layer; M is the matching result of the real label obtained by matching the feature point sets A and B of the infrared image and the visible light image when the transformation matrix or the relative position between the cameras is known; and are feature points that fail to match in feature point sets A and B of the infrared image and the visible light image, respectively; l P ij is the similarity matrix corresponding to the l-th layer feature point matching layer; is the matching probability of the i-th feature point of the image to which the feature point set A belongs in the l-th feature point matching layer; is the matching probability of the jth feature point in the image to which the feature point set B belongs at the lth layer.

[0114] In step S104, after the feature point matching results are output through the feature point matching network, it is necessary to set the source image and the target image for the infrared image and the visible light image; the thin template interpolation method is used to determine the registration parameters that minimize the error between the registration points on the source image and the corresponding points on the target image; these registration parameters define how to map the source image to the target image to minimize the error between the registration points, so the source image is mapped to the target image based on the registration parameters, that is, the corresponding position of each pixel point in the source image in the target image is calculated according to the registration parameters, and the registration result of the infrared image and the visible light image is obtained.

[0115] After the above steps, the source image will be transformed so that it is aligned with the target image. The final result is the registered versions of the two images, which are spatially consistent and can be used for further image fusion.

[0116] In this embodiment, the heterogeneous registration algorithm based on feature point detection has the characteristics of high robustness, high speed and strong generalization.

[0117] For step S105, this embodiment provides an image fusion network. The image fusion network includes at least two consecutive fusion submodules, each fusion submodule includes a third convolutional layer and a third activation function layer, and the fusion submodules of the at least two consecutive fusion submodules except the last fusion submodule also include a batch normalization layer, and the batch normalization layer is respectively connected to the third convolutional layer and the third activation function layer.

[0118] In some optional embodiments, the last fusion submodule directly outputs the image fusion result. Therefore, the activation function used by the third activation function layer in the last fusion submodule is different from that used by the other fusion submodules. Figure 4 As shown in the figure, the last fusion submodule adopts the tanh activation function (hyperbolic tangent activation function), while the other fusion submodules adopt the Leaky Relu (Leaky Rectified Linear Unit) activation function.

[0119] In this embodiment, the image fusion network adopts an end-to-end network structure design, avoiding the problem of insufficient fusion features caused by manually designed fusion rules, so as to ensure that the fused image retains the useful information of the source image to the greatest extent possible.

[0120] After image registration, the registered infrared and visible light images are fed into the image fusion network, resulting in a fused image that combines the rich texture details of the visible light image with the target characteristics of the infrared image. To obtain a fused image that maximizes background detail and salient objects, a third training set is first obtained. The image fusion network is then supervised trained using this third training set and a third loss function, resulting in a trained image fusion network.

[0121] Specifically, the third loss function is defined as follows:

[0122]

[0123] In the above formula, I1 and I2 are the images to be fused; I f is the fused image; φ j (x),φ j (y) are the jth set of feature maps extracted by the feature encoder; H and W are the length and width of the input image respectively; H j 、W j are the length and width of the jth group of features extracted by the feature encoder; For image I f Structural similarity with image I1; For image I f Structural similarity with image I2; is the loss of structural consistency; is the feature loss; is the gradient loss; are the gradients of input x and y respectively; α, β, w1 and w2 are weight parameters.

[0124] For step S106, the embodiment of the present invention provides a target detection network. Figure 5 , which is a structural diagram of the target detection network provided by the embodiment of the present invention. Figure 5 As shown, the target detection network includes a target positioning network and a type detection network.

[0125] The target localization network consists of a convolutional neural network (CNN), a first region proposal network (RPN), a first pooling layer, and a first fully convolutional layer. The fused image is input into the convolutional neural network in the target localization network, passes through the first region proposal network (RPN), the first pooling layer, and the first fully convolutional layer outputs the target localization information.

[0126] The type detection network consists of a first residual network (RESNET), a second region proposal network (RPN), a second pooling layer, a second residual network (RESNET), and a second fully convolutional layer. The target location information output by the target localization network must first be mapped back to the fused image. After cropping the image blocks, the image blocks are input into the first residual network in the type detection network. After passing through the second region proposal network, the second pooling layer, and the second residual network, the second fully convolutional layer outputs the target type information.

[0127] In this embodiment, after the object detection network introduces RPN, the calculation of candidate regions can be effectively reduced, a set of detection boxes that may contain objects in the image can be generated, and finally the detection results and detection boxes are output.

[0128] Optionally, the first pooling layer and the second pooling layer may adopt ROI pooling.

[0129] In some optional embodiments, for training the target detection network, a fourth training set is first obtained, and then the target detection network is trained based on the fourth training set and the fourth loss function to obtain a trained target detection network.

[0130] The fourth loss function is mainly divided into position loss and classification loss. The former is the loss between the detected target box and the groundtruth, and the latter is the loss of classifying the objects in the target box. The formula of the fourth loss function is as follows:

[0131] Loss = L CIOU +L fl Formula (14);

[0132] In formula (14), L CIOU is the position loss of the target box, L fl is the classification loss.

[0133] For the position loss of the target frame, in order to make the predicted frame as close to the real frame as possible, the overlapping area, center point distance and aspect ratio between the two are measured as the loss function, so that the predicted frame generated by the network is as close to the real frame as possible. Therefore, the position loss of the target frame L CIOU The formula is:

[0134]

[0135] In formula (15), p is the center point of the predicted box; g is the center point of the real box; c is the diagonal length of the minimum enclosing rectangle of the predicted box and the real box. When the distance between the two boxes is infinite, the distance between the center points and the diagonal length of the circumscribed rectangle are infinitely close, R(p, g)→1; IoU is the intersection over union ratio of the predicted box and the real box; v is the distance between the aspect ratio of the predicted box and the real box. w g and h g is the length and width of the real frame, w p and h p is the length and width of the prediction box; ρ 2 (p, g) is the Euclidean distance between the center points of the predicted box and the true box; α is the balance coefficient,

[0136] For classification loss, in order to solve the problem of unbalanced sample classification, an adjustable balanced sparse classification loss L is used. fl The formula is:

[0137] L fl =-((1-p) 2 q log(p)+p 2 (1-q)log(1-p)) Formula (16);

[0138] In formula (16), p and q are the classification prediction vectors of the candidate samples and the corresponding predicted true values.

[0139] It should be noted that the data used for network training in this example consists of a public dataset and images collected by infrared and visible light devices. 80% of the data was used as a training set for training the neural network, and 20% of the data was used as a test set for testing the neural network's performance. The Adam optimizer was used to optimize the network parameters using backpropagation.

[0140] Example 2:

[0141] At present, commonly used computing platforms are usually general-purpose processors based on the von Neumann architecture, namely general-purpose CPU or GPU platforms. They have defects such as serious resource consumption, slow running speed, and high energy consumption. As a result, they can only rely on server clusters to provide cloud computing services, and are difficult to apply to mobile terminals or embedded devices that have high requirements for area, speed and energy consumption.

[0142] Brain-inspired computing is inspired by the brain's nervous system and uses the way the human brain processes information to design computing methods for certain intelligent applications. Brain-inspired computing includes the simulation and application of human brain neuron models, as well as more complex in-depth simulations of human brain information processing to carry out the thinking process of learning, reasoning, and decision-making. Brain-inspired computing platforms refer to integrated circuit chips and systems that draw on the computing principles of the brain. Through a large-scale parallel computing platform, they provide an information processing architecture that provides efficient solutions to general intelligence problems. Compared with traditional CPU and GPU platforms, their energy efficiency is greatly improved, making them more suitable for the application of artificial intelligence algorithms in terminal devices such as embedded and mobile devices.

[0143] Since the network model used in the above embodiment adopts the standard Conv-BN-relu structure, the trained algorithm model can be easily deployed on a high-energy-efficiency brain-like computing platform through a tool chain. In the test, the infrared image and visible light image to be tested are first subjected to feature point extraction and feature point matching to obtain the registered infrared image and visible light image; secondly, the fusion network is used to extract and fuse the features of the two modalities and perform feature fusion; finally, the fused image is predicted using the detection network, and high-confidence candidate samples are selected according to the set classification confidence threshold. The predicted target position is obtained by fine-tuning the spatial position predicted by the regression vector, and non-maximum suppression is performed on candidate samples that meet the confidence threshold.

[0144] The embodiment of the present invention provides a brain-like computing platform that applies the infrared and visible light dual-band image processing method of the above embodiment. Figure 6 , which is a structural diagram of the brain-like computing platform provided by an embodiment of the present invention. Figure 6 As shown, the brain-like computing platform includes: an image acquisition unit 601, a main control unit 602, a switching unit and at least two many-core computing units 604; wherein,

[0145] The image acquisition unit 601 is connected to the main control unit 602 and is used to synchronize the visible light image captured by the visible light camera 605 and the infrared image captured by the infrared camera 606 to form a video stream;

[0146] The main control unit 602 is connected to the image acquisition unit 601 and the switching unit respectively, and is used to send the video stream to the switching unit;

[0147] a switching unit, connected to the main control unit 602 and the at least two many-core computing units 604, respectively, configured to generate at least two image processing tasks based on the video stream, distribute the at least two image processing tasks to the at least two many-core computing units 604 for image processing, and send image processing results returned by the many-core computing units 604 to the main control unit 602;

[0148] Each of the many-core computing units 604 includes at least one brain-inspired computing chip, and each of the brain-inspired computing chips stores and executes the computer program or computer instructions of the infrared and visible light dual-band image processing method of the above embodiment.

[0149] In this embodiment, image acquisition unit 601 exchanges data with main control unit 602 via network protocols such as IPv4 / IPv6 (Internet Protocol Version 4 / Internet Protocol Version 6), FTP (File Transfer Protocol), RTSP (Real Time Streaming Protocol, a network control protocol), and TCP (Transmission Control Protocol, a connection-oriented, reliable, byte-stream-based transport layer communication protocol). Image acquisition unit 601 connects to the visible light camera and infrared camera via a network port. Its core component is an FPGA (Field-Programmable Gate Array), which enables functions such as image acquisition and image data acceleration.

[0150] The switching unit may be a PCIE (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) switching unit 603 , and the main control unit 602 transmits the video stream data to the PCIE switching unit 603 through the PCIE protocol.

[0151] The brain-inspired computing platform may also include a camera control module 607 connected to the main control unit 602, infrared camera 606, and visible light camera 605, respectively. The camera control module 607 primarily consists of a joystick and a control keyboard. The joystick controls the rotation of the camera pan / tilt, while the control keyboard controls the focus and operating mode of the infrared camera 606 and visible light camera 605. The camera control module 607 can directly control the infrared camera 606 and visible light camera 605 via the RS232 protocol (Recommended Standard 232, a serial communication protocol). Alternatively, the camera control module 607 can connect to the main control unit 602, receiving and responding to camera control commands sent by the main control unit 602 via a network protocol to control the infrared camera 606 and visible light camera 605.

[0152] The brain-inspired computing platform further includes a storage module 608 connected to the main control unit 602 , for receiving and storing the video stream data before and after processing sent by the main control unit 602 .

[0153] Optionally, the storage module 608 may be connected to the main control unit 602 via a SATA (Serial Advanced Technology Attachment) interface to store the video stream data before and after being processed by the many-core computing unit 604 .

[0154] It should be noted that Figure 6 Only four many-core computing units 604 and an example of a many-core computing unit 604 composed of two brain-inspired computing chips are shown. The number of many-core computing units 604 and brain-inspired computing units is set according to actual conditions.

[0155] This embodiment adopts a mapping method based on the minimum functional core of brain-like computing. The minimum functional core architecture of brain-like computing chips is as follows: Figure 7 As shown in the figure, its main structure includes axons, synapses, dendrites, cell bodies, and routing networks. The axon unit is equivalent to the input buffer, mainly used to cache the front-end input information of the neuron and provide the required input data for the neuron calculation in the next time step; the synaptic array mainly completes the storage of connection topology and weight parameters, and establishes an access data interface with other units that need to access connections and weights; the dendrite unit mainly completes the multiplication, accumulation and integration process of input and weight; the cell body unit mainly performs nonlinear processing on the output; the routing unit is mainly used for information transmission between neurons, that is, for configuring the network topology.

[0156] Brain-like computing chips use fixed-point numbers to calculate network parameters to reduce the overly complex floating-point calculation process. At the same time, since it is impossible to assign too large an input weight to a single neuron and the hardware is often constrained by the communication bit width, the artificial neural network often needs to perform a series of operations such as network shaping when it is mapped to the brain-like computing chip for calculation in order to achieve maximum computing efficiency. The mapping process of the artificial neural network (ANN) in the brain-like computing chip is as follows Figure 8 shown.

[0157] The mapping of algorithm networks mainly includes network shaping and network topology mapping, which are mainly used to configure weight parameters such as cell bodies, synapses, routing tables, and timing in brain-like computing chips.

[0158] The network model provided in the above embodiment can be configured and used in a brain-inspired computing chip according to the following steps:

[0159] Step 1: Establish the upper layer original network model.

[0160] Step 2: Parameter Configuration and Chip Layout—This primarily involves network parameter shaping, network topology mapping, and chip physical layout. This includes configuring cell body parameters, synaptic parameters, routing parameters, and operating modes, as well as timing and related parameters. If, after comprehensive consideration of factors such as resource consumption and network performance, the network model does not meet application requirements, return to Step 1 to adjust the network model; otherwise, continue.

[0161] Step 3: Initialize the chip status, generate chip configuration and initialization files, download them to the chip system, and start the chip operation.

[0162] Among them, network integerization is mainly to solve the limitations of storage and communication bit width and calculation accuracy in brain-like computing chips. Normally, the floating-point network model is converted into 8b or 10b signed integer data through table lookup and chain scaling rules to meet the computing limitations of brain-like computing chips. However, due to the centralized multi-core parallel processing architecture, the computing resources of a single core are limited, and the information is local, so the network topology and the calculation process require global control, and the number of nodes in the usual network model far exceeds the range supported by a single minimum computing unit. Therefore, in the process of network mapping, the network needs to be split, and according to the hardware architecture, the split network needs to be mapped to the chip or chip group. The specific convolution mapping process is as follows. Figure 8 and Figure 9As shown, each minimum computing unit (FCore) is divided into three major functions: caching, sorting, and computation. The output of the caching FCore is fed back to the sorting FCore across rows. The sorting FCore is responsible for collecting and sorting the input FM of several rows in the past, and after sorting, it is sent to the computing FCore. The computing FCores share the output of the sorting FCore through direct multicasting and are responsible for completing convolution and pooling operations. All FCores are configured in multi-phase mode, with the phases interconnected. Each FCore's dendritic accumulation and cell body computation can be enabled in the phase where calculation is required and disabled in other phases. This coordination allows the input feature map to be input row by row and output in a pipelined manner.

[0163] In some optional embodiments, the specific algorithm flow of network shaping is as follows:

[0164] Step (1): Determine the weight W of each layer according to the data range of the weight parameters of each layer. (l) The zoom factor The scaled weight parameter is rounded to meet the 8-bit integer limit, that is, and is an integer; determines the scaling factor of the input layer This also ensures that the input meets the 8-bit integer limit after rounding; let l=1.

[0165] Step (2): If all layers are processed, i.e., l = L, then stop; otherwise, scale the input according to the lth layer’s input scaling factor. and weight scaling factor Calculate the integer bias value of layer l

[0166] Step (3), calculate the output of layer l Determine the truncation scaling factor of the layer based on its value range Make the output meet the 10-bit integer limit, that is,

[0167] And it is an integer.

[0168] Step (4): Determine the search compression factor for layer l So that the output of layer l is Y (l) Satisfy the 8-bit integer restriction, that is And it is an integer.

[0169] Step (5), follow Calculate input scaling factor Update l←l+1 and return to step (2).

[0170] In the above algorithm, by determining each scaling factor to satisfy the integer constraint, the parameter or state value after multiplying or dividing by the scaling factor is minimized to the saturation edge of the data range, thereby fully utilizing the chip's data representation capabilities. This algorithm can be used to shape the parameters of the proposed infrared and visible light dual-band registration and detection fusion algorithm network and deploy it on a brain-inspired computing platform, achieving highly energy-efficient algorithm implementation.

[0171] In summary, the brain-like computing platform proposed in this embodiment solves the problems of low computing power density, high power consumption, fixed computing power functions, difficulty in performance expansion and upgrading, and difficult maintenance of traditional computing power platforms such as CPU+GPU. It provides a brain-like processing system with a flexible architecture, passive heat dissipation and waterproofness. The system has the characteristics of high computing power density, high performance, low power consumption, compact structure, scalable and expandable functional performance, and upgradeable system hardware. Functions such as infrared and visible light camera image acquisition, image data storage, image data calculation, and artificial intelligence reasoning acceleration can be realized on a single platform, which can meet the use and deployment requirements of high-performance image acquisition and processing in harsh environments.

[0172] Although some specific embodiments of the present invention have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should be understood by those skilled in the art that modifications may be made to the above embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for processing infrared and visible light dual-band images, characterized in that: include: Acquire infrared images and visible light images of the same scene at the same time; The feature points and feature descriptions of the infrared image and the visible light image are respectively extracted based on a trained feature extraction network; wherein the feature extraction network includes a feature encoder, a feature point decoder, and a feature description decoder; the feature encoder includes at least two consecutive downsampling layers; the feature point decoder includes a first convolutional layer, a first activation function layer, and a graph transformation layer connected in sequence; the feature description decoder includes a second convolutional layer, an upsampling layer, and a regularization layer connected in sequence; the input end of the last downsampling layer of the at least two consecutive downsampling layers is used to input the infrared image and the visible light image; the input end of the first convolutional layer and the input end of the second convolutional layer are both connected to the output end of the last downsampling layer of the at least two consecutive downsampling layers; Input the feature points and feature descriptions of the infrared image and the visible light image into a trained feature point matching network to generate feature point matching results; wherein the feature point matching network includes at least one continuous feature point matching layer, each of which includes two self-attention units, one cross-attention unit and two confidence calculation units; the input end of one of the two self-attention units is used to input the feature points and feature descriptions of the infrared image, and the input end of the other self-attention unit is used to input the feature points and feature descriptions of the visible light image; the output ends of the two self-attention units are connected to the input end of the cross-attention unit, and the two output ends of the cross-attention unit are respectively connected to the input ends of the two confidence calculation units; if the confidences output by the two confidence calculation units are both greater than a preset confidence threshold, exit the current feature point matching layer, calculate the similarity matrix, and determine the matching feature points; otherwise, remove the feature points whose confidences are not greater than the preset confidence threshold, and input the remaining feature points and corresponding feature descriptions into the next feature point matching layer of the current feature point matching layer; performing image registration on the infrared image and the visible light image based on the feature point matching result; Inputting the registered infrared image and visible light image into a trained image fusion network to obtain a fused image; wherein the image fusion network includes at least two consecutive fusion submodules, each of the fusion submodules includes a third convolutional layer and a third activation function layer, and the fusion submodules of the at least two consecutive fusion submodules except the last fusion submodule also include a batch normalization layer, and the batch normalization layer is respectively connected to the third convolutional layer and the third activation function layer; The fused image is input into a trained target detection network to obtain target detection information; wherein, the target detection information includes target positioning information and target type information, and the target detection network includes a target positioning network and a type detection network; the target positioning network includes a convolutional neural network, a first region proposal network, a first pooling layer, and a first full convolutional layer connected in sequence; the input end of the convolutional neural network is used to input the fused image, and the output end of the first full convolutional layer is used to output the target positioning information; the type detection network includes a first residual network, a second region proposal network, a second pooling layer, a second residual network, and a second full convolutional layer connected in sequence; the input end of the first residual network inputs an image block, and the second full convolutional layer is used to output the target type information, and the image block is obtained by mapping the target positioning information back to the fused image and cropping it.

2. The infrared and visible light dual-band image processing method according to claim 1, characterized in that: Before extracting the feature points and feature descriptions of the infrared image and the visible light image based on the trained feature extraction network, the method further includes: Get the first training set; Training the feature extraction network based on the first training set and the first loss function to obtain the trained feature extraction network; The formula of the first loss function is: L(X,X′,D,D′;Y,Y′,S)=L p (X,Y)+L p (X′,Y′)+λL d (D,D′,S); Where X is the input image; Y is the true label of the feature point; D is the feature description; X′ is the new input image generated by randomly changing the input image by the H matrix; Y′ is the true label of the new feature point generated by randomly changing the input image by the H matrix; D′ is the new feature description generated by randomly changing the input image by the H matrix; L p is the feature point loss; L d is the feature description loss; λ is the weight parameter; S is the guide term of the homography matrix.

3. The infrared and visible light dual-band image processing method according to claim 1, characterized in that: Before inputting the feature points and feature descriptions of the infrared image and the visible light image into a trained feature point matching network to generate a feature point matching result, the method further includes: Get the second training set; Training the feature point matching network based on the second training set and the second loss function to obtain the trained feature point matching network; The formula of the second loss function is: Wherein, σ is the score of the matching probability output by the feature point matching network; l is the number of layers when exiting the current feature point matching layer; L is the output result of the Lth feature point matching layer; M is the matching result of the true label obtained by matching the feature point sets A and B of the infrared image and the visible light image when the transformation matrix or the relative position between the cameras is known; and are feature points that fail to match in feature point sets A and B of the infrared image and the visible light image, respectively; l P ij is the similarity matrix corresponding to the l-th layer feature point matching layer; is the matching probability of the i-th feature point of the image to which the feature point set A belongs in the l-th feature point matching layer; is the matching probability of the jth feature point in the image to which the feature point set B belongs at the lth layer.

4. The infrared and visible light dual-band image processing method according to claim 1, characterized in that: The performing image registration on the infrared image and the visible light image based on the feature point matching result includes: Setting a source image and a target image for the infrared image and the visible light image; Determining registration parameters that minimize errors between registration points on the source image and corresponding points on the target image using a thin template interpolation method; The source image is mapped to the target image based on the registration parameters to obtain a registration result of the infrared image and the visible light image.

5. The infrared and visible light dual-band image processing method according to claim 1, characterized in that: Before inputting the registered infrared image and the visible light image into the trained image fusion network to obtain the fused image, the method further includes: Get the third training set; Training the image fusion network based on the third training set and the third loss function to obtain the trained image fusion network; The formula of the third loss function is: Where, I1 and I2 are the images to be fused; I f is the fused image; φ j (x),φ j (y) are the jth set of feature maps extracted by the feature encoder; H and W are the length and width of the input image respectively; H j 、W j are the length and width of the jth group of features extracted by the feature encoder respectively; For image I f Structural similarity with image I1; For image I f Structural similarity with image I2; is the loss of structural consistency; is the feature loss; is the gradient loss; are the gradients of input x and y respectively; α, β, w1 and w2 are weight parameters.

6. The infrared and visible light dual-band image processing method according to claim 1, characterized in that: Before inputting the fused image into a trained target detection network to obtain target detection information, the method further includes: Get the fourth training set; Training the object detection network based on the fourth training set and the fourth loss function to obtain the trained object detection network; Wherein, the formula of the fourth loss function is: Loss=L CIOU +L fl ; L CIOU is the position loss of the target box, and the formula is: Where p is the center point of the predicted box; g is the center point of the real box; c is the diagonal length of the minimum enclosing rectangle of the predicted box and the real box; IoU is the intersection over union ratio of the predicted box and the real box; v is the distance between the aspect ratio of the predicted box and the real box, w g and h g is the length and width of the real frame, w p and h p is the length and width of the prediction box; ρ 2 (p, g) is the Euclidean distance between the center points of the predicted box and the real box; α is the balance coefficient, L fl is the classification loss, and the formula is: L fl =-((1-p) 2 qlog(p)+p 2 (1-q)log(1-p)); Where p and q are the classification prediction vectors of the candidate samples and the corresponding predicted true values.

7. A brain-like computing platform, characterized in that: It includes an image acquisition unit, a main control unit, a switching unit, and at least two many-core computing units; The image acquisition unit is connected to the main control unit and is used to synchronously align the collected visible light image and the infrared image to form a video stream; The main control unit is connected to the image acquisition unit and the switching unit respectively, and is used to send the video stream to the switching unit; The switching unit is connected to the main control unit and the at least two many-core computing units, respectively, and is configured to generate at least two image processing tasks according to the video stream, distribute the at least two image processing tasks to the at least two many-core computing units for image processing, and send image processing results returned by the many-core computing units to the main control unit; The many-core computing unit includes at least one brain-inspired computing chip, which is connected to the switching unit; the brain-inspired computing chip stores and executes the computer program or computer instructions of the infrared and visible light dual-band image processing method as described in any one of claims 1 to 6.

Images