Infrared visible light dual-band image registration and fusion detection method based on brain-like computing platform deployment

Abstract

The invention discloses an infrared visible light dual-band image registration and fusion detection method based on brain-like computing platform deployment, and relates to the technical field of image processing, and the scheme comprises the steps: obtaining an infrared image and a visible light image of the same scene at the same moment; respectively extracting feature points and feature descriptions of the infrared image and the visible light image based on the trained feature extraction network; inputting the feature points and feature descriptions of the infrared image and the visible light image into a trained feature point matching network to generate a feature point matching result; performing image registration on the infrared image and the visible light image based on a feature point matching result; inputting the infrared image and the visible light image after image registration into a trained image fusion network to obtain a fused image; and inputting the fused image into a trained target detection network to obtain target detection information. According to the invention, the registration efficiency and accuracy of the infrared image and the visible light image can be realized.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to an infrared and visible light dual-band image processing method and a brain-like computing platform or an infrared and visible light dual-band image registration and fusion detection method deployed based on a brain-like computing platform. Background Art

[0002] With the continuous development and progress of optoelectronic imaging technology, the spectral range of information that image sensors can obtain is wider, the information richness is greatly increased, and the application field is gradually expanding. However, due to the limited imaging spectral band range of a single image sensor, the information obtained has certain limitations and is increasingly unable to meet people's requirements for obtaining a multi-faceted and more accurate description of scenes and targets in actual use. Therefore, comprehensive processing and fusion of images obtained by at least two imaging devices working in different spectral bands or with different imaging principles will be more conducive to the all-round description of the scene and the detection and identification of targets.

[0003] There are significant differences between infrared images and visible light images in terms of imaging principles, wavelength range, information content, etc. Each has unique characteristics, and the two are obviously complementary. Infrared images are imaged by infrared sensors based on infrared radiation emitted or reflected by objects, and the imaging band is between 0.75-14 microns. Infrared light can penetrate smoke, mist and some non-metallic materials to a certain extent, which enables infrared images to provide better imaging effects under low visibility conditions (such as at night, smoky, etc.). Therefore, infrared imaging does not rely on ambient light, and can be imaged both during the day and at night, with the characteristics of full-time operation. However, due to the longer wavelength of infrared light, the resolution of infrared images is usually lower than that of visible light images, and the pixel density of infrared sensors is low, so the ability to express details is limited. Visible light imaging is passive imaging with a wavelength range of 370 nanometers to 730 nanometers, which contains the sunlight reflection information of the scene. The image has clear details, high resolution and contrast, and contains rich color information, which is more conducive to human eye observation. However, it is easily disturbed by environmental factors and distortion during the imaging process, and cannot work all day.

[0004] Therefore, by using image registration and fusion technology to fuse visible light images and infrared images and comprehensively utilize information, the advantages of the two can be complementary. Not only can the thermal radiation distribution of important targets and backgrounds be obtained from infrared images, but also clear background details and texture information can be obtained from visible light images. This can provide a more comprehensive description of the scene and target, and is more helpful for environmental monitoring and target detection and identification. It has very important practical needs and application value.

[0005] At present, traditional image registration methods are mainly divided into registration algorithms based on image grayscale statistics, registration methods based on image transform domain, and registration methods based on image features. However, in the actual registration process, a considerable number of feature points cannot be matched due to various differences, becoming useless feature points, taking up a lot of computing time and reducing the registration efficiency. In addition, it is difficult for the registration algorithm to complete multi-source image registration. At present, many classic registration methods are designed for images with similar bands, but infrared and visible light images often have large resolution differences and local grayscale differences. Therefore, many classic methods are basically ineffective when applied to the registration of infrared and visible light images.

[0006] Traditional image fusion methods mainly include transform domain-based fusion methods, spatial domain-based fusion methods, and deep learning-based fusion methods. Traditional algorithms require manual design of fusion rules, which have poor applicability, waste manpower, and have certain limitations in real-time performance. Since the fusion task is an unsupervised task, the current deep learning-based algorithm lacks appropriate guidance, and there is an imbalance in the visible light features that retain infrared features, which easily leads to problems such as visible light details not being sharp enough and infrared targets not being prominent enough. Therefore, it is difficult for traditional image fusion algorithms to strike a balance between the retention of infrared target features and the retention of visible light background details, which is not conducive to the display of fusion results and the accuracy of subsequent target detection. In addition, traditional fusion algorithms and detection algorithms require a large amount of registration data as input and training data, and due to the differences in resolution and modality between infrared images and visible light images, the errors through camera parameters or traditional registration algorithms are large, which is not conducive to subsequent fusion and detection tasks.

[0007] For target detection tasks, the detection accuracy of single-modal visible light image target detection is easily affected when the background is complex, such as when there is occlusion, complex background or weak light, and it is easy to miss or misdetect. For multi-modal target detection, it is impossible to fully utilize the information between different modalities, and the detection accuracy may decrease instead of increase compared to single-modal detection. Therefore, how to integrate multi-modal information has also become the key to the target detection task. Summary of the invention

[0008] In view of this, the present invention provides an infrared and visible light dual-band image processing method and a brain-like computing platform, or an infrared and visible light dual-band image registration and fusion detection method deployed based on a brain-like computing platform, which can achieve the registration efficiency and accuracy of infrared images and visible light images.

[0009] In order to achieve the above object, the present invention adopts the following technical scheme:

[0010] In a first aspect, the present invention provides an infrared and visible light dual-band image processing method, the method comprising:

[0011] Acquire infrared images and visible light images of the same scene and at the same time;

[0012] Feature points and feature descriptions of infrared images and visible light images are extracted based on a trained feature extraction network; wherein the feature extraction network includes a feature encoder, a feature point decoder and a feature description decoder; the feature encoder includes at least two consecutive downsampling layers; the feature point decoder includes a first convolutional layer, a first activation function layer and a graphic transformation layer connected in sequence; the feature description decoder includes a second convolutional layer, an upsampling layer and a regularization layer connected in sequence; the input end of the last downsampling layer in at least two consecutive downsampling layers is used to input the infrared image and the visible light image; the input end of the first convolutional layer and the input end of the second convolutional layer are both connected to the output end of the last downsampling layer in at least two consecutive downsampling layers;

[0013] Input the feature points and feature descriptions of the infrared image and the visible light image into the trained feature point matching network to generate feature point matching results; wherein the feature point matching network includes at least one continuous feature point matching layer, each feature point matching layer includes two self-attention units, one cross-attention unit and two confidence calculation units; the input end of one of the two self-attention units is used to input the feature points and feature descriptions of the infrared image, and the input end of the other self-attention unit is used to input the feature points and feature descriptions of the visible light image; the output ends of the two self-attention units are connected to the input end of the cross-attention unit, and the two output ends of the cross-attention unit are respectively connected to the input ends of the two confidence calculation units; if the confidences output by the two confidence calculation units are both greater than a preset confidence threshold, then exit the current feature point matching layer, calculate the similarity matrix, and determine the matching feature points; otherwise, remove the feature points whose confidences are not greater than the preset confidence threshold, and input the remaining feature points and the corresponding feature descriptions into the next feature point matching layer of the current feature point matching layer;

[0014] Perform image registration on the infrared image and the visible light image based on the feature point matching results;

[0015] Inputting the registered infrared image and the visible light image into a trained image fusion network to obtain a fused image; wherein the image fusion network includes at least two consecutive fusion submodules, each fusion submodule includes a third convolution layer and a third activation function layer, and the fusion submodules except the last fusion submodule in the at least two consecutive fusion submodules also include a batch normalization layer, and the batch normalization layer is respectively connected to the third convolution layer and the third activation function layer;

[0016] The fused image is input into a trained target detection network to obtain target detection information; wherein the target detection information includes target positioning information and target type information, and the target detection network includes a target positioning network and a type detection network; the target positioning network includes a convolutional neural network, a first region proposal network, a first pooling layer, and a first full convolutional layer connected in sequence; the input end of the convolutional neural network is used to input the fused image, and the output end of the first full convolutional layer is used to output the target positioning information; the type detection network includes a first residual network, a second region proposal network, a second pooling layer, a second residual network, and a second full convolutional layer connected in sequence; the input end of the first residual network inputs an image block, and the second full convolutional layer is used to output the target type information, and the image block is obtained by mapping the target positioning information back to the fused image and cropping it.

[0017] The method provided by the present invention utilizes a deep learning algorithm to combine the features of infrared and visible light modalities, and can more comprehensively obtain useful information from both modalities, and can achieve high-precision heterogeneous image registration, high-information-richness fusion, and all-weather, all-round target detection. Forming a task flow of at least two visual tasks, namely, feature point extraction, registration, fusion, and target detection, can not only improve the accuracy of each task, make up for their respective shortcomings, solve the problem of collecting a large amount of data during the training of deep network models, avoid problems such as overfitting and poor generalization performance of the model, but also organically combine at least two visual tasks to improve the accuracy of the final detection.

[0018] In a possible implementation, before extracting feature points and feature descriptions of the infrared image and the visible light image based on the trained feature extraction network, the method further includes:

[0019] Get the first training set;

[0020] Training the feature extraction network based on the first training set and the first loss function to obtain a trained feature extraction network;

[0021] Among them, the formula of the first loss function is:

[0022] L(X,X′,D,D′;Y,Y′S)=L p (X, Y)+L p (X′, Y′)+λL d (D, D′, S);

[0023] Where X is the input image; Y is the real label of the feature point; D is the feature description; X′ is the new input image generated by a random H matrix change on the input image; Y′ is the real label of the new feature point generated by a random H matrix change on the input image; D′ is the new feature description generated by a random H matrix change on the input image; L p is the feature point loss; Ld is the feature description loss; λ is the weight parameter; S is the guide term of the homography matrix.

[0024] In a possible implementation, before inputting the feature points and feature descriptions of the infrared image and the visible light image into the trained feature point matching network to generate the feature point matching results, the method further includes:

[0025] Get the second training set;

[0026] Training the feature point matching network based on the second training set and the second loss function to obtain a trained feature point matching network;

[0027] Among them, the formula of the second loss function is:

[0028]

[0029] Wherein, σ is the score of the matching probability output by the feature point matching network; l is the number of layers when exiting the current feature point matching layer; L is the output result of the Lth feature point matching layer; M is the matching result of the real label obtained by matching the feature point sets A and B of the infrared image and the visible light image when the transformation matrix or the relative position between the cameras is known; and are the unmatched feature points in the feature point sets A and B of the infrared image and the visible light image respectively; l P ij is the similarity matrix corresponding to the l-th feature point matching layer; is the matching probability of the i-th feature point of the image to which the feature point set A belongs in the l-th feature point matching layer; is the matching probability of the jth feature point in the image to which the feature point set B belongs at the lth layer.

[0030] In a possible implementation, performing image registration on the infrared image and the visible light image based on the feature point matching result includes:

[0031] Setting source image and target image for infrared image and visible light image;

[0032] The registration parameters that minimize the error between the registration points on the source image and the corresponding points on the target image are determined using a thin sample interpolation method;

[0033] The source image is mapped to the target image based on the registration parameters to obtain the registration results of the infrared image and the visible light image.

[0034] In a possible implementation, before inputting the registered infrared image and the visible light image into a trained image fusion network to obtain a fused image, the method further includes:

[0035] Get the third training set;

[0036] Training the image fusion network based on the third training set and the third loss function to obtain a trained image fusion network;

[0037] Among them, the formula of the third loss function is:

[0038]

[0039] Where I1 and I2 are the images to be fused; I f is the fused image; φ j (x),φ j (y) are the jth feature maps extracted by the feature encoder; H and W are the length and width of the input image respectively; H j , W j are the length and width of the jth group of features extracted by the feature encoder respectively; For image I f Structural similarity with image I1; For image I f Structural similarity with image I2; is the structural consistency loss; is the feature loss; is the gradient loss; are the gradients of input x and y respectively; α, β, w1 and w2 are weight parameters.

[0040] In a possible implementation, before inputting the fused image into the trained target detection network to obtain target detection information, the method further includes:

[0041] Get the fourth training set;

[0042] Training the object detection network based on the fourth training set and the fourth loss function to obtain a trained object detection network;

[0043] Among them, the formula of the fourth loss function is:

[0044] Loss = L CIOU +L fl ;

[0045] L CIOU is the position loss of the target box, and the formula is:

[0046]

[0047] Where p is the center point of the predicted box; g is the center point of the real box; c is the diagonal length of the minimum enclosing rectangle of the predicted box and the real box; IoU is the intersection over union ratio of the predicted box and the real box; v is the distance of the aspect ratio of the predicted box to the real box. wg and h g is the length and width of the real box, w p and h p is the length and width of the prediction box; ρ 2 (p, g) is the Euclidean distance between the center points of the predicted box and the true box; α is the balance coefficient,

[0048] L fl is the classification loss, and the formula is:

[0049] L fl =-((lp) 2 q log(p)+p 2 (1-q)log(1-p));

[0050] Where p and q are the classification prediction vectors of candidate samples and the corresponding predicted true values.

[0051] In a second aspect, the present invention provides a brain-like computing platform, including an image acquisition unit, a main control unit, a switching unit, and at least two many-core computing units;

[0052] An image acquisition unit, connected to the main control unit, is used to synchronize and align the acquired visible light image and infrared image to form a video stream;

[0053] A main control unit is connected to the image acquisition unit and the switching unit respectively, and is used to send the video stream to the switching unit;

[0054] a switching unit, connected to the main control unit and the at least two many-core computing units respectively, for generating at least two image processing tasks according to the video stream, distributing the at least two image processing tasks to the at least two many-core computing units for image processing, and sending the image processing results returned by the many-core computing units to the main control unit;

[0055] The many-core computing unit includes at least one brain-like computing chip connected to the switching unit; the brain-like computing chip stores and executes a computer program or computer instructions of the infrared and visible light dual-band image processing method of the first aspect and any possible implementation thereof.

[0056] For the specific description of the second aspect and its various implementations in the present invention, reference can be made to the detailed description of the first aspect and its various implementations; and for the beneficial effects of the second aspect and its various implementations, reference can be made to the analysis of the beneficial effects of the first aspect and its various implementations, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0058] Figure 1 It is a flowchart of an infrared and visible light dual-band image processing method according to an embodiment of the present invention;

[0059] Figure 2 is a structural diagram of a feature extraction network according to an embodiment of the present invention;

[0060] Figure 3 is a structural diagram of a feature point matching network according to an embodiment of the present invention;

[0061] Figure 4 is a structural diagram of an image fusion network according to an embodiment of the present invention;

[0062] Figure 5 is a structural diagram of a target detection network according to an embodiment of the present invention;

[0063] Figure 6 is a structural block diagram of a brain-like computing platform according to an embodiment of the present invention;

[0064] Figure 7 This is a minimum functional core architecture diagram of a brain-like computing chip according to an embodiment of the present invention;

[0065] Figure 8 is a diagram of an artificial neural network algorithm configuration process of an embodiment of the present invention;

[0066] Fig. 9 It is a process diagram of artificial neural network streaming topology mapping according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0068] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0069] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0070] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0071] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0072] Embodiment 1:

[0073] The embodiment of the present invention provides an infrared and visible light dual-band image processing method or an infrared and visible light dual-band image registration and fusion detection method based on a brain-like computing platform. Figure 1 , which is a flow chart of the infrared and visible light dual-band image processing method provided by an embodiment of the present invention. Figure 1 As shown, the infrared and visible light dual-band image processing method provided by the present invention mainly includes the following steps (steps S101 to S106):

[0074] Step S101, acquiring an infrared image and a visible light image of the same scene and at the same time;

[0075] Step S102, extracting feature points and feature descriptions of the infrared image and the visible light image respectively based on the trained feature extraction network;

[0076] Step S103, inputting the feature points and feature descriptions of the infrared image and the visible light image into the trained feature point matching network to generate feature point matching results;

[0077] Step S104, performing image registration on the infrared image and the visible light image based on the feature point matching result;

[0078] Step S105, inputting the registered infrared image and visible light image into the trained image fusion network to obtain a fused image;

[0079] Step S106, inputting the fused image into the trained target detection network to obtain target detection information.

[0080] In this embodiment, the resolutions of the infrared image and the visible light image are generally different. For example, the resolution of the infrared image is 640*512, and the resolution of the visible light image is 1280*720.

[0081] For step S102, the embodiment of the present invention provides a feature extraction network. Figure 2 , which is a structural diagram of the feature extraction network provided by an embodiment of the present invention. Figure 2 As shown, the feature extraction network includes a feature encoder 201, a feature point decoder 202 and a feature description decoder 203.

[0082] Among them, the structure of the feature encoder 201 is similar to vgg16 (Visual Geometry Group 16, a deep convolutional neural network model), which includes at least two consecutive downsampling layers 2011, which are used to reduce the size of infrared images and visible light images, for example, the size is 1 / 8 of the original input size, so as to reduce the amount of calculation, speed up the operation, and extract multi-scale features to construct a feature pyramid.

[0083] The feature point decoder 202 includes a first convolution layer 2021, a first activation function layer 2022, and a graphic transformation layer 2023 connected in sequence. Among them, the first convolution layer 2021 can further extract the feature points and feature descriptions in the infrared image and the visible light image after the reduced size output by the feature encoder 201, and locate the position of the feature points based on this information to obtain feature points of D1 dimension (for example, D1 = 65); after passing through the first activation function layer 2022 and the graphic transformation layer 2023, a feature point distribution map consistent with the original input size and dimension is obtained, and the size is between 0 and 1, and the closer to 1, the greater the probability of representing a feature point.

[0084] Optionally, the first convolution layer 2021 is a single-layer convolution; the first activation function layer 2022 may adopt a softmax activation function; and the graphic transformation layer 2023 may adopt resizing, upsampling, and the like.

[0085] The feature description decoder 203 includes a second convolution layer 2031, an upsampling layer 2032, and a regularization layer 2033 connected in sequence, and the input end of the first convolution layer 2021 and the input end of the second convolution layer 2031 are both connected to the output end of the last downsampling layer 2011 in at least two consecutive downsampling layers 2011. After the feature description output by the feature encoder 201 passes through the second convolution layer 2031 of the feature description decoder 203, a feature description of D2 dimension (for example, D2=256) is obtained, and after passing through the upsampling layer 2032 and the regularization layer 2033, a feature description of D2 dimension with the same size as the original input is obtained.

[0086] Optionally, the second convolution layer 2031 is a single-layer convolution; the upsampling layer 2032 can use bicubic interpolation; the regularization layer 2033 can use L2 regularization; the number of downsampling layers is set to 4, that is, N=4.

[0087] It should be noted that the size in this embodiment is the aspect ratio of the feature graph.

[0088] In some optional embodiments, for training the feature extraction network, a first training set is first obtained; then the feature extraction network is trained based on the first training set and the first loss function to obtain a trained feature extraction network.

[0089] Optionally, the Synthtic Shapes dataset (a synthetic dataset for shape recognition tasks) can be used as a dataset to train the network to extract various corner points, lines, and cube points.

[0090] The first loss function mainly includes feature point loss and feature description loss, where X is the input image, Y is the ground-truth label of the feature point, and D is the feature description.

[0091] In order to improve the robustness of the network, the input image is randomly transformed by the H matrix to obtain the updated input image X′, the ground-truth label of the feature point Y′ and the feature description D′. The final first loss function can be written as follows:

[0092] L(X,X′,D,D′;Y,Y′,S)=Lp(X,Y)+L p (X′, Y′)+λL d (D, D′, S) Formula (1);

[0093] In formula (1), L p is the feature point loss; L d is the feature description loss; λ is the weight parameter, which can be set to 10 to maintain a balance between feature point loss and feature description loss.

[0094] The feature point loss L is p And the feature description loss L d The calculation of is described in detail.

[0095] L p is the image patch x hw ∈X, with the image patch x hw The corresponding label is y hw (1 if there is a feature point, 0 if there is no feature point), feature point loss L p The formula is:

[0096]

[0097] In the above formula, X hw H is the pixel at position (h,w) output by the feature point decoder; C , W C The length and width of the feature point decoder output; x hwk The position of the feature point decoder output is (h, w) and the pixel corresponding to the k-th dimension channel; x hwy For x hw The total label (1 for feature points and 0 for no feature points).

[0098] For the feature description loss L d , similarly, in an image block d hw ∈D and d′ h′w′ ∈D′, in the image block at position (h, w) and the image block at position (h′, w′), S is calculated by the following formula:

[0099]

[0100] In formula (4), S is the guide term of the homography matrix; H is the homography matrix; P hw is the center point of the image block; is P multiplied by H hw And divided by the final coordinate, the final coordinate refers to the corresponding transformed coordinate, that is, the coordinate of the Euclidean distance is transformed to the secondary coordinate. Then the final feature description loss L d The calculation formula is as follows:

[0101]

[0102] In formula (5), d hw is the feature description of the position (h, w), d′ h′w′ is the feature description at position (h′, w′).

[0103] l d (d, d′; s) = λ d *s*max(0,m p -d T d′)+(1-s)*max(0,d T d′-m n ) formula (6);

[0104] In formula (6), d and d' are the feature descriptions of infrared image and visible light image respectively; m p 1 is the positive margin; m n Taking 0.2 is a negative margin; s is the same as S; λ d is the weight, which can be 250.

[0105] For step S103, an embodiment of the present invention provides a feature point matching network. Figure 3 , which is a structural diagram of a feature point matching network provided by an embodiment of the present invention. Figure 3 As shown, the feature point matching network includes at least one continuous feature point matching layer, each feature point matching layer includes two self-attention units, one cross-attention unit and two confidence calculation units.

[0106] The input end of one of the two self-attention units is used to input the feature point P of the infrared image A and feature description d A , the input of the other self-attention unit is used to input the feature point P of the visible light image B and feature description d B .

[0107] The outputs of the two self-attention units are connected to the input of the cross-attention unit, and the two outputs of the cross-attention unit are connected to the inputs of the two confidence calculation units respectively. The cross-attention unit outputs the feature representation of the new feature point.

[0108] One of the confidence calculation units performs confidence calculation on the feature representation of the new feature point and the feature description of the infrared image to obtain a first confidence; the other confidence calculation unit performs confidence calculation on the feature representation of the new feature point and the feature description of the visible light image to obtain a second confidence. If both the first confidence and the second confidence are greater than the preset confidence threshold, the current feature point matching layer is exited, and the similarity matrix is ​​calculated to determine the matching feature points. Otherwise, the feature points whose confidence is not greater than the preset confidence threshold are removed, and the remaining feature points and the corresponding feature descriptions are input into the next feature point matching layer of the current feature point matching layer.

[0109] For feature point matching networks, in order to reduce errors during training, we can use style transfer to generate strictly matched pseudo infrared images on the COCO dataset (Common Objects in Context, a large-scale target detection, segmentation and description dataset) and the IVS dataset (Iris Verification in the Wild, a dataset for iris recognition) for training. The loss is mainly the loss between the matching matrix P and the real label ground truth. When the transformation matrix or the relative position between cameras is known, we can match the point sets A and B to obtain the matching result M of the real label ground truth. The points that fail to match in the feature point set are recorded as By optimizing the form of log-likelihood probability, the matching can be completed as much as possible in the network layer 1, allowing the network to predict the corresponding matching relationship as early as possible.

[0110] Specifically, for the training of the feature point matching network, the second training set is first obtained; then the feature point matching network is trained based on the second training set and the second loss function to obtain a trained feature point matching network.

[0111] Among them, the second loss function is defined as follows:

[0112]

[0113] In formula (7), σ is the score of the matching probability output by the feature point matching network; l is the number of layers when exiting the current feature point matching layer; L is the output result of the Lth feature point matching layer; M is the matching result of the real label obtained by matching the feature point sets A and B of the infrared image and the visible light image when the transformation matrix or the relative position between cameras is known; and are feature points that fail to match in feature point sets A and B of the infrared image and the visible light image, respectively; l P ij is the similarity matrix corresponding to the l-th feature point matching layer; is the matching probability of the i-th feature point of the image to which the feature point set A belongs in the l-th feature point matching layer; is the matching probability of the jth feature point in the image to which the feature point set B belongs at the lth layer.

[0114] In step S104, after the feature point matching results are output through the feature point matching network, it is necessary to set the source image and the target image for the infrared image and the visible light image; the thin template interpolation method is used to determine the registration parameters that minimize the error between the registration points on the source image and the corresponding points on the target image; these registration parameters define how to map the source image to the target image to minimize the error between the registration points, so the source image is mapped to the target image based on the registration parameters, that is, the corresponding position of each pixel in the source image in the target image is calculated according to the registration parameters, and the registration results of the infrared image and the visible light image are obtained.

[0115] After the above steps, the source image will be transformed so that it is aligned with the target image. The final result is the registered versions of the two images, which are spatially consistent and can be used for further image fusion.

[0116] In this embodiment, the heterogeneous registration algorithm based on feature point detection has the characteristics of high robustness, high speed and strong generalization.

[0117] For step S105, this embodiment provides an image fusion network. The image fusion network includes at least two consecutive fusion submodules, each fusion submodule includes a third convolution layer and a third activation function layer, and the fusion submodules other than the last fusion submodule in the at least two consecutive fusion submodules also include a batch normalization layer, and the batch normalization layer is respectively connected to the third convolution layer and the third activation function layer.

[0118] In some optional embodiments, the last fusion submodule directly outputs the image fusion result. Therefore, the activation function used by the third activation function layer in the last fusion submodule is different from that in other fusion submodules. Figure 4 As shown, the last fusion submodule adopts the tanh activation function (hyperbolic tangent activation function), while the other fusion submodules adopt the Leaky Relu (Leaky Rectified Linear Unit) activation function.

[0119] In this embodiment, the image fusion network adopts an end-to-end network structure design, which avoids the problem of insufficient fusion features caused by manually designed fusion rules, so as to ensure that the fused image retains the useful information of the source image to the maximum extent.

[0120] After completing the image registration, the registered infrared image and visible light image are sent to the image fusion network to obtain a fused image with rich visible light image texture details and infrared image target characteristics. In order to obtain a fused image with as rich background details and prominent targets as possible, the third training set is first obtained, and then the image fusion network is supervised and trained based on the third training set and the third loss function to obtain a trained image fusion network.

[0121] Specifically, the third loss function is defined as follows:

[0122]

[0123] In the above formula, I1 and I2 are the images to be fused; I f is the fused image; φ j (x),φ j (y) are the jth feature maps extracted by the feature encoder; H and W are the length and width of the input image respectively; H j , W j are the length and width of the jth group of features extracted by the feature encoder respectively; For image I f Structural similarity with image I1; For image I f Structural similarity with image I2; is the structural consistency loss; is the feature loss; is the gradient loss; are the gradients of input x and y respectively; α, β, w1 and w2 are weight parameters.

[0124] For step S106, an embodiment of the present invention provides a target detection network. Figure 5 , which is a structural diagram of the target detection network provided by an embodiment of the present invention. Figure 5 As shown, the target detection network includes a target positioning network and a type detection network.

[0125] The target positioning network includes a convolutional neural network (CNN), a first region proposal network (RPN), a first pooling layer, and a first full convolution layer connected in sequence. The fused image is input into the convolutional neural network in the target positioning network, passes through the first region proposal network (RPN), the first pooling layer, and the first full convolution layer outputs the target positioning information.

[0126] The type detection network includes the first residual network (RESNET), the second region proposal network (RPN), the second pooling layer, the second residual network (RESNET) and the second full convolution layer connected in sequence. The target positioning information output by the target positioning network needs to be mapped back to the fused image first, and after the image block is cropped, the image block is input into the first residual network in the type detection network, and then passes through the second region proposal network, the second pooling layer and the second residual network, and the target type information is output by the second full convolution layer.

[0127] In this embodiment, after the object detection network introduces RPN, the calculation of candidate regions can be effectively reduced, a set of detection boxes that may contain objects in the image can be generated, and finally the detection results and detection boxes are output.

[0128] Optionally, the first pooling layer and the second pooling layer may adopt ROI pooling.

[0129] In some optional embodiments, for training the target detection network, a fourth training set is first obtained, and then the target detection network is trained based on the fourth training set and the fourth loss function to obtain a trained target detection network.

[0130] The fourth loss function is mainly divided into position loss and classification loss. The former is the loss between the detected target box and the groundtruth, and the latter is the loss of classifying the objects in the target box. The formula of the fourth loss function is as follows:

[0131] Loss = L CIOU +L fl Formula (14);

[0132] In formula (14), L CIOU is the position loss of the target box, L fl is the classification loss.

[0133] For the position loss of the target box, in order to make the predicted box as close to the real box as possible, the overlapping area, center point distance and aspect ratio between the two are measured as the loss function, so that the predicted box generated by the network is as close to the real box as possible. Therefore, the position loss of the target box L CIOU The formula is:

[0134]

[0135] In formula (15), p is the center point of the predicted box; g is the center point of the real box; c is the diagonal length of the minimum enclosing rectangular box between the predicted box and the real box. When the distance between the two boxes is infinite, the distance between the center points and the diagonal length of the circumscribed rectangular box are infinitely close, and R(p, g)→1; IoU is the intersection over union ratio of the predicted box and the real box; v is the distance of the aspect ratio of the predicted box and the real box. w g and h g is the length and width of the real box, w p and h p is the length and width of the prediction box; ρ 2 (p, g) is the Euclidean distance between the center points of the predicted box and the true box; α is the balance coefficient,

[0136] For classification loss, in order to solve the problem of unbalanced sample classification, an adjustable balanced sparse classification loss L is used. fl The formula is:

[0137] L fl =-((1-p) 2 q log(p)+p 2 (1-q)log(1-p)) Formula (16);

[0138] In formula (16), p and q are the classification prediction vectors of the candidate samples and the corresponding predicted true values.

[0139] It should be noted that the data used for network training in this embodiment consists of public data sets and images collected by infrared and visible light devices. 80% of the data is made into a training set for training the neural network, and 20% of the data is made into a test set for testing the performance of the neural network. The Adam optimizer is used to optimize the network parameters in a back-propagation manner.

[0140] Embodiment 2:

[0141] At present, commonly used computing platforms are usually general-purpose processors based on the von Neumann architecture, namely general-purpose CPU or GPU platforms, which have defects such as serious resource consumption, slow running speed, and high energy consumption. As a result, they can only rely on server clusters to provide cloud computing services, and are difficult to apply to mobile terminals or embedded devices that have high requirements on area, speed and energy consumption.

[0142] Brain-inspired computing is inspired by the brain's nervous system and is a method of designing computing for certain intelligent applications by drawing on the way the human brain processes information. Brain-inspired computing includes the simulation and application of human brain neuron models, as well as more complex deep simulations of how the human brain processes information to carry out the thinking process of learning, reasoning, and decision-making. The brain-inspired computing platform refers to an integrated circuit chip and system that draws on the computing principles of the brain. It is an information processing architecture that provides efficient solutions to general intelligence problems through a large-scale parallel computing platform. Compared with traditional CPU and GPU platforms, the energy efficiency ratio is greatly improved, making it more suitable for the application of artificial intelligence algorithms in terminal devices such as embedded and mobile terminals.

[0143] Since the network model used in the above embodiment adopts the standard Conv-BN-relu structure, the trained algorithm model can be easily deployed on a brain-like computing platform with high energy efficiency ratio through the tool chain. In the test, the infrared image and visible light image to be tested are first subjected to feature point extraction and feature point matching to obtain the registered infrared image and visible light image; secondly, the fusion network is used to extract and fuse the features of the two modalities and perform feature fusion; finally, the fused image is predicted using the detection network, and high-confidence candidate samples are selected according to the set classification confidence threshold, and the predicted target position is obtained by fine-tuning the spatial position predicted by the regression vector, and non-maximum suppression is performed on candidate samples that meet the confidence threshold.

[0144] The embodiment of the present invention provides a brain-like computing platform that applies the infrared and visible light dual-band image processing method of the above embodiment. Figure 6 , which is a structural diagram of the brain-like computing platform provided by an embodiment of the present invention. Figure 6 As shown, the brain-like computing platform includes: an image acquisition unit 601, a main control unit 602, a switching unit and at least two many-core computing units 604; wherein,

[0145] The image acquisition unit 601 is connected to the main control unit 602 and is used to synchronize and align the visible light image acquired by the visible light camera 605 and the infrared image acquired by the infrared camera 606 to form a video stream;

[0146] The main control unit 602 is connected to the image acquisition unit 601 and the switching unit respectively, and is used to send the video stream to the switching unit;

[0147] a switching unit, connected to the main control unit 602 and the at least two many-core computing units 604, respectively, for generating at least two image processing tasks according to the video stream, distributing the at least two image processing tasks to the at least two many-core computing units 604 for image processing, and sending the image processing results returned by the many-core computing units 604 to the main control unit 602;

[0148] Each of the many-core computing units 604 includes at least one brain-like computing chip, and each of the brain-like computing chips stores and executes the computer program or computer instructions of the infrared and visible light dual-band image processing method of the above-mentioned embodiment.

[0149] In this embodiment, the image acquisition unit 601 exchanges data with the main control unit 602 through network protocols, such as IPv4 / IPv6 (Internet Protocol Version 4 / Internet Protocol Version 6), FTP (File Transfer Protocol), RTSP (Real Time Streaming Protocol, a network control protocol), TCP (Transmission Control Protocol, a connection-oriented, reliable, byte stream-based transport layer communication protocol), etc. The image acquisition unit 601 is connected to the visible light camera and the infrared camera through a network port, and its core component is FPGA (Field-Programmable Gate Array, programmable logic gate array), which can realize functions such as image acquisition and image data acceleration.

[0150] The switching unit may be a PCIE (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) switching unit 603 , and the main control unit 602 transmits the video stream data to the PCIE switching unit 603 through the PCIE protocol.

[0151] The brain-like computing platform may also include a camera control module 607 connected to the main control unit 602, the infrared camera 606 and the visible light camera 605 respectively, which is mainly composed of a joystick and a control keyboard. The joystick controls the rotation of the camera pan, and the control keyboard controls the focus and working mode of the infrared camera 606 and the visible light camera 605. The camera control module 607 can directly control the infrared camera 606 and the visible light camera 605 through the RS232 protocol (Recommended Standard 232, a serial communication protocol), and can also be connected to the main control unit 602, receive and respond to the camera control instructions sent by the main control unit 602 through the network protocol, and control the infrared camera 606 and the visible light camera 605.

[0152] The brain-like computing platform also includes a storage module 608 connected to the main control unit 602 , which is used to receive and store the video stream data before and after processing sent by the main control unit 602 .

[0153] Optionally, the storage module 608 may be connected to the main control unit 602 via a SATA (Serial Advanced Technology Attachment) interface to store the video stream data before and after being processed by the many-core computing unit 604 .

[0154] It should be noted that Figure 6 Only four many-core computing units 604 and an example in which the many-core computing unit 604 is composed of two brain-like computing chips are shown. The number of many-core computing units 604 and brain-like computing units is set according to actual conditions.

[0155] This embodiment adopts a mapping method based on the minimum functional core of brain-like computing. The minimum functional core architecture of brain-like computing chips is as follows: Figure 7 As shown in the figure, its main structure includes axons, synapses, dendrites, cell bodies and routing networks. The axon unit is equivalent to the input cache, which is mainly used to cache the front-end input information of neurons and provide the required input data for the neuron calculation in the next time step; the synapse array mainly completes the storage of connection topology and weight parameters, and establishes access data interfaces with other units that need to access connections and weights; the dendrite unit mainly completes the multiplication and accumulation of inputs and weights; the cell body unit mainly performs nonlinear processing on the output; the routing unit is mainly used for information transmission between neurons, that is, for configuring the network topology.

[0156] Brain-like computing chips use fixed-point numbers to calculate network parameters to reduce the overly complex floating-point calculation process. At the same time, since it is impossible to assign too large an input weight to a single neuron and the hardware often has communication bit width constraints, the artificial neural network often needs to perform a series of operations such as network integerization when it is mapped to the brain-like computing chip for calculation in order to achieve maximum computing efficiency. The mapping process of the artificial neural network (ANN) in the brain-like computing chip is as follows Figure 8 shown.

[0157] The mapping of algorithm networks mainly includes network shaping and network topology mapping, which are mainly used to configure weight parameters such as cell bodies, synapses, routing tables and timing in brain-like computing chips.

[0158] The network model provided in the above embodiment can be configured and used in the brain-like computing chip according to the following steps:

[0159] Step 1: Establish the upper original network model.

[0160] Step 2: Parameter configuration and chip layout - mainly includes network parameter shaping, network topology mapping and chip physical layout. Among them, network parameter shaping and network topology mapping include cell body parameter configuration, synapse parameter configuration, routing parameter configuration and working mode, timing and related parameter configuration. If it does not meet the application requirements after comprehensive consideration of factors such as resource consumption and network performance, you need to go back to step 1 to adjust the network model; otherwise, continue.

[0161] Step 3: Initialize the chip status, generate chip configuration and initialization files, download them to the chip system, and start the chip operation.

[0162] Among them, network integerization is mainly to solve the limitations of storage and communication bit width and computing accuracy in brain-like computing chips. Usually, the floating-point network model is converted into 8b or 10b signed integer data through table lookup and chain scaling rules to meet the computing limitations of brain-like computing chips. However, due to the centralized multi-core parallel processing architecture, the computing resources of a single core are limited and the information is local, so the network topology and the calculation process require global control, and the number of nodes in the usual network model far exceeds the range supported by a single minimum computing unit. Therefore, in the process of network mapping, the network needs to be split and, according to the hardware architecture, the split network needs to be mapped to a chip or chip group. The specific convolution mapping process is as follows: Figure 8 and Fig. 9As shown in the figure, each minimum computing unit FCore is divided into three functions: caching, sorting, and computing. The output of the cache FCore is fed back to the sorting FCore across rows, and the sorting FCore is responsible for collecting and sorting the input FM of the historical rows. After sorting, it is sent to the computing FCore. The computing FCores share the output of the sorting FCore in a direct multicast manner and are responsible for completing convolution and pooling operations. All FCores are configured in multi-phase mode, and the phase numbers are connected to each other. Each FCore dendrite accumulation and cell body calculation can be started in the phase that needs to be calculated, and closed in other phases. The cooperation allows the input feature map Feature Map to be input by row and output in a pipeline.

[0163] In some optional embodiments, the specific algorithm flow of network shaping is as follows:

[0164] Step (1): Determine the weight W of each layer according to the data range of the weight parameters of each layer. (l) The zoom factor The scaled weight parameter is rounded to meet the 8-bit integer limit, that is, and is an integer; determines the scaling factor of the input layer This also makes the input satisfy the 8-bit integer limit after rounding; let l=1.

[0165] Step (2): If all layers have been processed, i.e., l = L, then stop; otherwise, scale the input according to the lth layer’s input scaling factor and weight scaling factor Calculate the integer bias value of layer l

[0166] Step (3), calculate the output of layer l Determine the truncation scaling factor of the layer based on its value range Make the output meet the 10-bit integer limit, that is,

[0167] And it is an integer.

[0168] Step (4), determine the search compression factor of the lth layer So that the output of layer l is Y (l) Satisfy the 8-bit integer restriction, that is And it is an integer.

[0169] Step (5), follow Calculate input scaling factor Update l←l+1 and return to step (2).

[0170] In the above algorithm, by determining each scaling factor to satisfy the integer constraint, the parameter or state value after multiplying or dividing the scaling factor tends to the saturation edge of the data range as much as possible, so that the data expression capability of the chip can be fully utilized. The parameters of the proposed infrared and visible light dual-band registration and detection fusion algorithm network can be shaped through the above algorithm, and can be deployed on the brain-like computing platform to achieve high-energy-efficiency algorithm implementation.

[0171] In summary, the brain-like computing platform proposed in this embodiment solves the problems of low computing power density, high power consumption, fixed computing power functions, difficult performance expansion, difficult upgrade and maintenance of traditional computing power platforms such as CPU+GPU, and provides a passive heat dissipation waterproof brain-like processing system with flexible architecture. The system has the characteristics of high computing power density, high performance, low power consumption, compact structure, tailorable and expandable functional performance and upgradeable system hardware. On a single platform, functions such as infrared and visible light camera image acquisition, image data storage, image data calculation and artificial intelligence reasoning acceleration can be realized, which can meet the use and deployment requirements of high-performance image acquisition and processing in harsh environments.

[0172] Although some specific embodiments of the present invention have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will be appreciated by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. An infrared and visible light dual-band image processing method, characterized in that: include: Acquire infrared images and visible light images of the same scene and at the same time; The feature points and feature descriptions of the infrared image and the visible light image are extracted based on the trained feature extraction network; wherein the feature extraction network includes a feature encoder, a feature point decoder and a feature description decoder; the feature encoder includes at least two consecutive downsampling layers; the feature point decoder includes a first convolutional layer, a first activation function layer and a graphic transformation layer connected in sequence; the feature description decoder includes a second convolutional layer, an upsampling layer and a regularization layer connected in sequence; the input end of the last downsampling layer of the at least two consecutive downsampling layers is used to input the infrared image and the visible light image; the input end of the first convolutional layer and the input end of the second convolutional layer are both connected to the output end of the last downsampling layer of the at least two consecutive downsampling layers; Input the feature points and feature descriptions of the infrared image and the visible light image into a trained feature point matching network to generate feature point matching results; wherein the feature point matching network includes at least one continuous feature point matching layer, each of which includes two self-attention units, one cross-attention unit and two confidence calculation units; the input end of one of the two self-attention units is used to input the feature points and feature descriptions of the infrared image, and the input end of the other self-attention unit is used to input the feature points and feature descriptions of the visible light image; the output ends of the two self-attention units are connected to the input end of the cross-attention unit, and the two output ends of the cross-attention unit are respectively connected to the input ends of the two confidence calculation units; if the confidences output by the two confidence calculation units are both greater than a preset confidence threshold, exit the current feature point matching layer, calculate the similarity matrix, and determine the matching feature points; otherwise, remove the feature points whose confidences are not greater than the preset confidence threshold, and input the remaining feature points and the corresponding feature descriptions into the next feature point matching layer of the current feature point matching layer; Performing image registration on the infrared image and the visible light image based on the feature point matching result; Inputting the registered infrared image and the visible light image into a trained image fusion network to obtain a fused image; wherein the image fusion network includes at least two consecutive fusion submodules, each of which includes a third convolutional layer and a third activation function layer, and the fusion submodules except the last fusion submodule in the at least two consecutive fusion submodules also include a batch normalization layer, and the batch normalization layer is respectively connected to the third convolutional layer and the third activation function layer; The fused image is input into a trained target detection network to obtain target detection information; wherein the target detection information includes target positioning information and target type information, and the target detection network includes a target positioning network and a type detection network; the target positioning network includes a convolutional neural network, a first region proposal network, a first pooling layer, and a first full convolutional layer connected in sequence; the input end of the convolutional neural network is used to input the fused image, and the output end of the first full convolutional layer is used to output the target positioning information; the type detection network includes a first residual network, a second region proposal network, a second pooling layer, a second residual network, and a second full convolutional layer connected in sequence; the input end of the first residual network inputs an image block, and the second full convolutional layer is used to output the target type information, and the image block is obtained by mapping the target positioning information back to the fused image and cropping it.

2. The infrared and visible light dual-band image processing method according to claim 1, characterized in that: Before respectively extracting feature points and feature descriptions of the infrared image and the visible light image based on the trained feature extraction network, the method further includes: Get the first training set; Training the feature extraction network based on the first training set and the first loss function to obtain the trained feature extraction network; The formula of the first loss function is: L(X,X′,D,D′;Y,Y′,S)=L p (X,Y)+L p (X′,Y′)+λL d (D,D′,S); Where X is the input image; Y is the real label of the feature point; D is the feature description; X′ is the new input image generated by a random H matrix change on the input image; Y′ is the real label of the new feature point generated by a random H matrix change on the input image; D′ is the new feature description generated by a random H matrix change on the input image; L p is the feature point loss; L d is the feature description loss; λ is the weight parameter; S is the guide term of the homography matrix.

3. The infrared and visible light dual-band image processing method according to claim 1, characterized in that: Before inputting the feature points and feature descriptions of the infrared image and the visible light image into a trained feature point matching network to generate feature point matching results, the method further includes: Get the second training set; Training the feature point matching network based on the second training set and the second loss function to obtain the trained feature point matching network; The formula of the second loss function is: Wherein, σ is the score of the matching probability output by the feature point matching network; l is the number of layers when exiting the current feature point matching layer; L is the output result of the Lth feature point matching layer; M is the matching result of the real label obtained by matching the feature point sets A and B of the infrared image and the visible light image when the transformation matrix or the relative position between cameras is known; and are feature points that fail to match in feature point sets A and B of the infrared image and the visible light image, respectively; l P ij is the similarity matrix corresponding to the l-th feature point matching layer; is the matching probability of the i-th feature point of the image to which the feature point set A belongs in the l-th feature point matching layer; is the matching probability of the jth feature point in the image to which the feature point set B belongs at the lth layer.

4. The infrared and visible light dual-band image processing method according to claim 1, characterized in that: The performing image registration on the infrared image and the visible light image based on the feature point matching result comprises: Setting a source image and a target image for the infrared image and the visible light image; Determining the registration parameters that minimize the error between the registration points on the source image and the corresponding points on the target image using a thin template interpolation method; The source image is mapped to the target image based on the registration parameters to obtain a registration result of the infrared image and the visible light image.

5. The infrared and visible light dual-band image processing method according to claim 1, characterized in that: Before inputting the registered infrared image and the visible light image into the trained image fusion network to obtain the fused image, the method further includes: Get the third training set; Training the image fusion network based on the third training set and the third loss function to obtain the trained image fusion network; Wherein, the formula of the third loss function is: Where I1 and I2 are the images to be fused; I f is the fused image; j (x),φ j (y) are the jth group of feature maps extracted by the feature encoder; H and W are the length and width of the input image respectively; H j , W j are respectively the length and width of the jth group of features extracted by the feature encoder; For image I f Structural similarity with image I1; For image I f Structural similarity with image I2; is the loss of structural consistency; is the feature loss; is the gradient loss; are the gradients of input x and y respectively; α, β, w1 and w2 are weight parameters.

6. The infrared and visible light dual-band image processing method according to claim 1, characterized in that: Before inputting the fused image into a trained target detection network to obtain target detection information, the method further includes: Get the fourth training set; Training the object detection network based on the fourth training set and the fourth loss function to obtain the trained object detection network; Wherein, the formula of the fourth loss function is: Loss=L CIOU +L fl ; L CIOU is the position loss of the target box, and the formula is: Where p is the center point of the prediction box; g is the center point of the real box; c is the diagonal length of the minimum enclosing rectangular box between the prediction box and the real box; IoU is the intersection over union ratio of the prediction box and the real box; v is the distance of the aspect ratio of the prediction box to the real box, w g and h g is the length and width of the real frame, w p and h p is the length and width of the prediction box; ρ 2 (p, g) is the Euclidean distance between the center points of the predicted box and the true box; α is the balance coefficient, L fl is the classification loss, and the formula is: L fl =-((1-p) 2 qlog(p)+p 2 (1-q)log(1-p)); Where p and q are the classification prediction vectors of candidate samples and the corresponding predicted true values.

7. A brain-like computing platform, characterized in that: It includes an image acquisition unit, a main control unit, a switching unit and at least two many-core computing units; The image acquisition unit is connected to the main control unit and is used to synchronously align the acquired visible light image and the infrared image to form a video stream; The main control unit is connected to the image acquisition unit and the switching unit respectively, and is used to send the video stream to the switching unit; The switching unit is connected to the main control unit and the at least two many-core computing units respectively, and is used to generate at least two image processing tasks according to the video stream, distribute the at least two image processing tasks to the at least two many-core computing units for image processing, and send the image processing results returned by the many-core computing units to the main control unit; The many-core computing unit includes at least one brain-like computing chip connected to the switching unit; the brain-like computing chip stores and executes the computer program or computer instructions of the infrared and visible light dual-band image processing method as described in any one of claims 1 to 6.

Images