A method and system for multi-source data fusion image recognition

Abstract

The present application relates to a kind of multi-source data fusion image recognition method and system;The method comprises: infrared image and low-light image are aligned;Respectively, infrared image and low-light image are adaptively multi-scale decomposed;Respectively, k the ak base layer and k the bk detail layer are fused to obtain base fusion layer and detail fusion layer, base fusion layer and detail fusion layer are fused, then using the infrared target area extracted from infrared image, the compensation of fusion result is obtained to obtain the final fusion image;Using the fusion image carries out target identification.The present application can fully utilize the layer difference of infrared image and low-light image in different decomposition process, and the interregional difference information of the area where different pixel points are located, fully utilize the interlayer difference and regional difference, and decomposition efficiency is high.

Description

Technical Field

[0001] This invention belongs to the field of intelligent recognition technology, and in particular relates to a method and system for multi-source data fusion image recognition. Background Technology

[0002] Target extraction refers to the process of separating the target of interest from the background in a single image or a sequence of images, identifying and interpreting meaningful objects, and extracting different image features. Currently, many methods apply infrared target extraction algorithms to image fusion. These methods typically segment infrared images into target and background images, using the background image as the infrared source image and fusing it with the low-light image at multiple scales. The resulting fused image is then superimposed on the target image to obtain the final fused image. However, the image segmentation methods used in these methods are often based on thresholding, resulting in clear target region contours. Since the fused image is derived from the background image, poor segmentation can easily lead to significant artificial artifacts in the target region of the final fused image, affecting its naturalness and realism.

[0003] Due to the relatively underdeveloped image sensor technology in the early stages of image fusion, low-light images often suffered from weaknesses such as low resolution, high noise, and weak contrast. Stretching the image contrast to enhance detail in low-light images often further amplified noise, negatively impacting the quality of the final fused image. While current image sensor technology has advanced to the point where low-light sensors can produce low-noise, high-resolution images rich in detail under low-light conditions, traditional image fusion algorithms can only preserve some detail and texture information in low-light conditions and cannot enhance it. Therefore, existing fusion methods do not fully utilize the detail information in low-light images, resulting in an incomplete representation of detail in the fused image. Since noise levels in existing low-light images have been effectively suppressed, detail enhancement operations do not have significant negative effects. To enhance detail in low-light images and make them easier to recognize and understand, this technology employs an image contrast enhancement algorithm to process the image, allowing the detail information to be more fully revealed. The resulting fused image has deep layers, rich detail, clear texture, and is easier for humans to understand and recognize. Guided filtering can successfully separate the three layers of details that are visually distinct: coarse, medium, and fine. However, its smoothing radius is often fixed or manually adjusted, failing to fully utilize the layer differences between infrared and low-light images in different decomposition processes, as well as the regional differences between the areas where different pixels are located. This results in significant information loss and reduced efficiency. Furthermore, traditional image fusion algorithms do not specifically address this issue and often suffer from insufficient utilization of infrared target information.

[0004] Based on the above problems, the payload of this invention has all-weather imaging capabilities, and can fully utilize the layer differences between infrared and low-light images in different decomposition processes, as well as the inter-regional differences in the regions where different pixels are located. It makes full use of inter-layer and regional differences, resulting in high decomposition efficiency. The final fused image retains both the target information of the infrared image and the detailed information of the low-light image, while enhancing the infrared target to make it more prominent. By extracting the target from the infrared image, the extracted infrared target is then used to weight and enhance the fused image, ultimately making the infrared target stand out more significantly from the background in the fused image. Summary of the Invention

[0005] To address the aforementioned problems in the prior art, this invention proposes a multi-source data fusion image recognition method and system, the method comprising:

[0006] Step S1: Perform joint calibration of the infrared image camera and the low-light image camera, acquire infrared images and low-light images in real time, and align the infrared images and low-light images;

[0007] Step S2: Perform adaptive multi-scale decomposition on the infrared image and the low-light image respectively; specifically: starting from k=1, use the decomposition matrix to perform the k-th scale decomposition on the infrared image to obtain the ak-th base layer and the ak-th detail layer, and perform the k-th multi-scale decomposition on the low-light image to obtain the bk-th base layer and the bk-th detail layer; obtain the 1k-th observation matrix based on the ak-th base layer and the bk-th base layer; obtain the 2k-th observation matrix based on the ak-th detail layer and the bk-th detail layer; determine the decomposition matrix of the (k+1)-th multi-scale decomposition based on the 1k-th observation matrix and the 2k-th observation matrix; determine whether the decomposition cutoff condition is met. If yes, proceed to step S3; otherwise, set k=k+1 and repeat the step.

[0008] Step S3: Fuse the k-th base layer and the k-th detail layer respectively to obtain a base fusion layer and a detail fusion layer. Then, fuse the base fusion layer and the detail fusion layer, and use the infrared target region extracted from the infrared image to compensate the fusion result to obtain the final fused image. Use the fused image for target recognition.

[0009] Furthermore, step S2 specifically includes the following steps:

[0010] Step S21: Perform the k-th scale decomposition on the infrared image to obtain the ak-th base layer and the ak-th detail layer;

[0011] Step S22: Perform the k-th multi-scale decomposition on the low-light image to obtain the bk-th base layer and the bk-th detail layer;

[0012] Step S23: Based on the ak-th base layer and the bk base layer To obtain the 1kth observation matrix Based on the ak-th detail layer and the bk detail layer To obtain the 2kth observation matrix ;

[0013] Step S24: Determine the next decomposition matrix based on the 1kth observation matrix and the 2kth observation matrix;

[0014] Step S25: Determine whether the decomposition cutoff condition is met. If yes, proceed to step S3; otherwise, set k=k+1 and return to step S21 to execute the next scale decomposition.

[0015] The determination of whether the decomposition cutoff condition is met specifically involves: calculating the mean of the elements in the decomposition matrix. When the mean of the element is greater than or equal to the first preset value or k is greater than or equal to the second preset value, the decomposition cutoff condition is determined to be satisfied.

[0016] Furthermore, step S21 specifically involves: setting the current image based on the infrared image. Obtain the decomposition matrix Based on the decomposition matrix, the current image Perform smoothing to obtain the ak-th base layer. ; Set the ak detail layer And so on.

[0017] Furthermore, we set the initial value of k to 1; when k=1, we decompose the matrix. The initial values ​​for the decomposition matrix are the current image. Let Ia be the initial value of the infrared image; when k>=1, the decomposition matrix is... The current image is calculated in step S24. This is the ak-th base layer of the infrared image.

[0018] Furthermore, step S22 specifically involves: setting the current image based on the low-light image. Obtain the decomposition matrix Based on the decomposition matrix of the current image Perform smoothing to obtain the bk-th base layer. Set the bk detail layer And so on.

[0019] Furthermore, the current image based on the decomposition matrix... Smoothing is performed; specifically, the following steps are included:

[0020] Step S2A1: Normalize the values ​​of the decomposed matrix to the range [0, 1].

[0021] Step S2A2: Instantiate the decomposition matrix based on the decomposition scenario;

[0022] Step S2A3: Based on the elements in the decomposition matrix Construct a dynamic smoothing window and use it to smooth the current image. The corresponding pixel cr in Perform smoothing to obtain .

[0023] A multi-source data fusion image recognition platform is provided, the platform being used to implement the multi-source data fusion image recognition method.

[0024] A server for multi-source data fusion image recognition includes a processor coupled to a memory, the memory storing program instructions, and the multi-source data fusion image recognition method is implemented when the program instructions stored in the memory are executed by the processor.

[0025] A multi-source data fusion image recognition system is provided, the system being used to implement the multi-source data fusion image recognition method.

[0026] A computer-readable storage medium includes a program that, when run on a computer, causes the computer to perform the multi-source data fusion image recognition method.

[0027] The beneficial effects of this invention include:

[0028] (1) Based on the real-time differences of the acquired images, the number of decompositions is dynamically determined. The differences in the layer structure of infrared images and low-light images during different decomposition processes, as well as the differences between regions where different pixels are located, are fully utilized. By utilizing the differences between layers and regions, the differences in local consistency between basic layers are fully discovered, and the differences in activity and intensity between detail layers are discovered. The differences in the differences between different regions are used to guide whether to perform the next decomposition for that region to reduce computational overhead. The decomposition matrix is ​​instantiated based on the decomposition scene, which can adapt to the hardware and software characteristics of the scene and has high decomposition efficiency.

[0029] (2) Fit different rotation relationships to each part of the two images to adapt to the error caused by the inconsistency between the model assumptions and the actual situation; the fused image retains both the infrared target information and the detail information of the low-light image, while enhancing the infrared target to make the target more prominent; by extracting the target from the infrared image, the extracted infrared target is then used to weight the fused image, ultimately making the infrared target in the fused image stand out more significantly from the background. Attached Figure Description

[0030] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to unduly limit the invention. In the drawings:

[0031] Figure 1 This is a schematic diagram of the multi-source data fusion image recognition method provided by the present invention.

[0032] Figure 2 This is a schematic diagram illustrating the mapping relationship between a fisheye image and a three-dimensional sphere provided by the present invention.

[0033] Figure 3 This is a schematic diagram of the PTZ camera parameter mapping relationship provided by the present invention. Detailed Implementation

[0034] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions are only used to explain the present invention and are not intended to limit the present invention.

[0035] As attached Figure 1 As shown in the attached figure, this invention proposes a multi-source data fusion image recognition method and system, as follows. Figure 1 As shown, the method includes the following steps:

[0036] Step S1: Joint calibration of the infrared image camera and the low-light image camera to acquire infrared image Ia and low-light image Ib in real time, and alignment of the infrared image and the low-light image; specifically: obtaining the mapping relationship between the coordinates of the infrared image camera and the rotation angle of the low-light image camera through joint calibration, acquiring the infrared image and the low-light image in real time, reprojecting the low-light image onto the plane where the infrared image is located based on the mapping relationship; aligning the infrared image and the low-light image through pixel interpolation;

[0037] Preferably: a panoramic camera is used to acquire infrared images; a PTZ camera is used to acquire low-light images; step S1 specifically includes the following steps:

[0038] Step S11: Based on the intrinsic parameters of the two cameras, convert their respective image coordinates and gimbal parameters into spherical coordinates; the mapping relationship between the corresponding spheres is a three-dimensional rotation relationship; further: fit different rotation relationships to each local part of each image to adapt to the error caused by the inconsistency between the model assumptions and the actual situation;

[0039] Preferably, the panoramic camera is a fisheye panoramic camera; the intrinsic parameter model of the fisheye panoramic camera depends on the optical imaging principle of the fisheye lens; according to the equidistant model of the fisheye image, the pixels of the fisheye image are mapped one-to-one with the three-dimensional spherical points; the parameters of the PTZ camera indicate the rotation parameters of the PTZ camera and the mapping relationship between the image coordinates and the three-dimensional sphere.

[0040] Step S12: Solve for the rotational relationship between the two spherical coordinates using a rotation matrix. Calculations: When the shooting scene is large and the installation distance between cameras is relatively small or very close, it can be assumed that the optical centers of the two cameras coincide. In this case, the relationship between the two cameras can be assumed to be a rotational relationship. The camera's parametric model indicates the mapping relationship between the camera and the three-dimensional sphere. The solution is the rotational relationship between the two spheres. Based on the assumption, the three-dimensional coordinate system where the panoramic camera is located... To the 3D coordinate system of the PTZ camera The mapping relationship is a three-dimensional rotation relationship, assuming a coordinate system Circling in sequence The axes rotate respectively radian can be compared with If they coincide, then the rotation relationship is as follows: Formula (1): Formula (2) is the rotation matrix. ;

[0041] , , (1);

[0042] (2)

[0043] Step S13: Reproject the PTZ image based on the mapping relationship; calculate the panoramic image coordinates based on the PTZ image coordinates; reproject the PTZ image onto the plane of the panoramic image, and then match and align the feature points of the reprojected image with the panoramic image.

[0044] Preferred method: Aligning infrared and low-light images by pixel interpolation, contrast stretching, scaling, translation, and rotation;

[0045] Step S2: Acquire infrared image Ia and low-light image Ib in real time, and perform adaptive multi-scale decomposition on the infrared image and the low-light image respectively; specifically: starting from k=1, use the decomposition matrix to perform the k-th scale decomposition on the infrared image to obtain the ak-th base layer and the ak-th detail layer, and perform the k-th multi-scale decomposition on the low-light image to obtain the bk-th base layer and the bk-th detail layer; obtain the 1k-th observation matrix based on the ak-th base layer and the bk-th base layer; obtain the 2k-th observation matrix based on the ak-th detail layer and the bk-th detail layer; determine the decomposition matrix of the (k+1)-th multi-scale decomposition based on the 1k-th observation matrix and the 2k-th observation matrix; determine whether the decomposition cutoff condition is met. If yes, proceed to step S3; otherwise, set k=k+1 and repeat the step.

[0046] Step S2 specifically includes the following steps:

[0047] Step S21: Perform the k-th scale decomposition on the infrared image to obtain the ak-th base layer and the ak-th detail layer; specifically: set the current image based on the infrared image. Obtain the decomposition matrix Based on the decomposition matrix, the current image Perform smoothing to obtain the ak-th base layer. ; Set the ak detail layer And so on; It contains the most significant edge information;

[0048] Preferred method: Set the initial value of k to 1; when k=1, decompose the matrix. The initial values ​​for the decomposition matrix are the current image. The initial value of the infrared image is Ia; the initial value of the decomposition matrix is ​​a preset value; in the initial state, it is set... ; Conversely, use the decomposition matrix determined in step S24; when k>=1, the decomposition matrix is... The current image is calculated in step S24. This is the ak-th base layer of the infrared image;

[0049] in: To obtain infrared images Dimensions; To find the integer sign upwards;

[0050] Preferred: Set the initial value of k to 1; obviously, when k=1, Used for identification Used for identification And so on;

[0051] The current image is based on the decomposition matrix. Smoothing is performed; specifically, the following steps are included:

[0052] Step S2A1: Normalize the values ​​of the decomposed matrix to the range [0, 1].

[0053] Preferably, the normalization is a local normalization relative to the decomposition matrix, or a global normalization relative to the historical decomposition matrix;

[0054] Step S2A2: Instantiate the decomposition matrix based on the decomposition scenario; specifically: obtain the minimum decomposition value corresponding to the decomposition scenario. and maximum value ,set up ;

[0055] Preferred: Decomposition of minimum value and maximum value These are preset values ​​set based on the decomposed scenario; for example: ; ;

[0056] Step S2A3: Based on the elements in the decomposition matrix Construct a dynamic smoothing window and use it to smooth the current image. The corresponding pixel cr in Perform smoothing to obtain Specifically: Set the smooth window to... Centered on, size is A square window is defined; within this square window, the linear coefficients a1 and a2 of the guided filter are calculated, and the linear coefficients are used to calculate the pixel cr. The output value; repeat this step until every pixel in the current image has been processed; set the current image based on the infrared image. At that time, pixel cr The output value is The value is used to obtain the ak-th basic layer. ; Set the current image based on low-light image At that time, pixel cr The output value is The numerical value yields the bk-th base layer. ;

[0057] Step S22: Perform the k-th multi-scale decomposition on the low-light image to obtain the bk-th base layer and the bk-th detail layer; specifically: set the current image based on the low-light image. Obtain the decomposition matrix Based on the decomposition matrix of the current image Perform smoothing to obtain the bk-th base layer. Set the bk detail layer ; It contains the most significant edge information in low-light images, and so on; as you can see, the processing method for low-light images here is similar to that for infrared images;

[0058] The acquisition of the decomposition matrix Specifically, when k=1, the decomposition matrix is ​​set to a preset initial value; otherwise, the decomposition matrix determined in step S24 is used; the initial value of the decomposition matrix is... ;

[0059] Step S23: Based on the ak-th base layer and the bk base layer To obtain the 1kth observation matrix Based on the ak-th detail layer and the bk detail layer To obtain the 2kth observation matrix ;in: For the ak base layer Middle position Pixel value at; For the bk-th base layer Middle position Pixel value at;

[0060] The base layer based on the ak layer and the bk base layer To obtain the 1kth observation matrix Specifically, this involves calculating the 1kth observation matrix. Used to represent the dynamic local consistency of the background portion between the ak-th base layer and the bk-th base layer; further: calculate the 1k-th observation matrix based on the following equation (3). ;in: It is a zero-crossing residual; Between 0 and 1, when the brightness of two base layers is very close at a certain position... A value close to 1 indicates high consistency; when the difference is large, A value close to 0 indicates a strong potential for conflict.

[0061] (3);

[0062] The level based on the ak detail layer and the bk detail layer To obtain the 2kth observation matrix Specifically: Calculate the 2kth observation matrix. Dynamic local consistency used to represent the difference in activity or intensity between the ak-th detail layer and the bk-th detail layer; further: the 2k-th observation matrix is ​​calculated based on the following equation (4) or (5). ;in: It is a zero-crossing residual; the 2Kth observation matrix indicates the local consistency of detail intensity at that location or the local consistency of maximum metric. Between 0 and 1, when two details are very close in activity or intensity at a certain location. A value close to 1 indicates high consistency; when the difference is large, A value close to 0 indicates poor consistency in activity or intensity of details;

[0063] (4);

[0064] (5);

[0065] Step S24: Determine the next decomposition matrix based on the 1kth observation matrix and the 2kth observation matrix; specifically: set , , This is a weighted average or minimum value function; it can be seen that, in In regions with small values, i.e., regions where the difference between the infrared and low-light base layers is large, significant conflicts exist. To better handle this conflict in the next layer, a smaller smoothing radius is used for decomposition to separate them at a finer scale; In regions with large values, indicating strong consistency and low conflict, a larger smoothing radius is used for coarse-grained smoothing to improve efficiency; Regions with small values ​​indicate important details. When decomposing to the next level, a larger radius is used to protect these details and avoid over-smoothing. Conversely, regions with large values ​​use a smaller radius.

[0066] Replaceable: Settings ;

[0067] Replaceable: Settings ;

[0068] Step S25: Determine whether the decomposition cutoff condition is met. If yes, proceed to step S3; otherwise, set k=k+1 and return to step S21 to execute the next scale decomposition. This method observes the conflict areas and important detail areas that appear in k rounds of fusion. Based on the differences in the infrared and low-light images of each area, while reducing the overall scale, it adaptively guides the k+1 rounds of decomposition based on the differentiated characteristics of local areas, making the multi-scale decomposition process more targeted and adaptive, and providing better fusion results.

[0069] The determination of whether the decomposition cutoff condition is met specifically involves: calculating the mean of the elements in the decomposition matrix. When the mean of the element is greater than or equal to the first preset value or k is greater than or equal to the second preset value, the decomposition cutoff condition is determined to be satisfied.

[0070] Preferably: the first preset value is 0.9; the second preset value is equal to 5~10;

[0071] Alternatively: Determining whether the decomposition cutoff condition is met specifically involves calculating the coefficient of variation of element deviations. If the deviation coefficient of variation is less than or equal to the third preset value, the decomposition cutoff condition is determined to be met. The deviation coefficient of variation is used to consider the saliency between regions within the detail layer and the base layer. Wherein, I and J are the maximum values ​​of rows and columns in the image, respectively. When the saliency and difference weaken between different regions as the decomposition process progresses, it indicates that the decomposition process is nearing completion. Therefore, in local regions with strong differences, even if the decomposition process has been performed multiple times, it may still be necessary to perform another decomposition for that region. Of course, it is possible to select the change in differences between different regions to guide whether to perform another decomposition for that region to reduce computational overhead.

[0072] Preferred value: The third preset value is 0.1~0.3;

[0073] Step S3: The base layer and detail layer are fused separately to obtain a base fusion layer and a detail fusion layer. The base fusion layer and detail fusion layer are then fused, and the fusion result is compensated using the infrared target region extracted from the infrared image to obtain the final fused image. Target recognition is then performed using this fused image. Specifically: the infrared target region is extracted from the infrared image to obtain the infrared target region; the 1k detail layer and 2k detail layer obtained from the k-th scale decomposition are fused to obtain the detail fusion layer; the 1k base layer and 2k base layer are fused to obtain the base fusion layer; the detail fusion layer and base fusion layer are fused, and the fused image is compensated using the infrared target region to obtain the final fused image; target recognition is performed based on the final fused image. By compensating for the infrared target region, it is ensured that the most critical thermal target information in the infrared image is not diluted or lost during the fusion process, and it can be enhanced in the final stage to make it stand out more in the final image; it prevents it from being overwhelmed by the background details of the low-light image; and it improves the reliability of the fusion result in subsequent target recognition tasks.

[0074] Preferred approach: When fusing detail layers, consider using the maximum region energy as the fusion rule to avoid isolated points and noise points being selected as detail information and fused into the final image;

[0075] Preferably, during fusion, the base layer generates weights for the base fusion layer using saliency-based rules to enhance image saliency; the detail layer preserves detail features using a maximum value rule; the base fusion layer contains a wide-range, smooth brightness distribution and main contours in the infrared image; the detail fusion layer contains high-frequency detail information such as edges and textures in the low-light image.

[0076] The process of fusing the 1k-th base layer and the 2k-th base layer to obtain the base fusion layer involves: sequentially fusing the 1k-th base layer and the 2k-th base layer corresponding to each k value to obtain the k-th sub-base fusion layer; then fusing the k k-th sub-base fusion layers based on different weights or weight matrices to obtain the base fusion layer; a similar approach can be used to fuse the 1k-th detail layer and the 2k-th detail layer obtained from the k-th scale decomposition to obtain the detail fusion layer.

[0077] The fusion of the detail fusion layer and the basic fusion layer is specifically performed by directly adding the fused detail fusion layer and the basic fusion layer together to reconstruct a preliminary fused image. This process typically does not involve complex fusion rules; it is simply a superposition.

[0078] The process of using the fused image for target recognition specifically involves: scaling the pixel values ​​in the fused image from [0, 255] to the range of [0, 1] or [-1, 1] to make the distribution of the input data consistent with the data distribution used during model training; using gamma correction and histogram equalization to further highlight the difference between the target and the background; using Gaussian filtering, median filtering, or more advanced nonlocal mean denoising to suppress noise that may be introduced during the fusion process and prevent the model from misclassifying noise as a target; and directly inputting the preprocessed fused image into the loaded target detection model.

[0079] Preferably, the target detection model is a target detection model such as YOLO or Faster R-CNN; deep features of the image are automatically extracted through the convolutional neural network backbone to obtain target classification;

[0080] This invention utilizes a saliency-based infrared target extraction algorithm to extract infrared targets, and then uses these targets with certain weights to compensate the fused image. Salient regions in an image often contain image information that differs from the surrounding environment, making them more likely to attract the attention of the human visual system. This is why saliency methods are highly useful in image understanding and target detection. By using saliency detection, salient regions in an image can be effectively and quickly identified, which contain target information. Finally, weighted compensation of the fused image can highlight salient regions in the infrared image. Even when there are no prominent targets in the image, the saliency detection algorithm will detect other salient regions. Finally, weighted compensation enhances the fused image, which can, to some extent, avoid the unnatural and unrealistic image appearance that can occur with the methods mentioned above. Furthermore, saliency detection is a detection method based on human visual habits, which allows the target and background in the enhanced image to blend more naturally, conforming to human visual habits and better helping the human eye recognize and understand information in the image.

[0081] By adaptively decomposing the image and dynamically comparing local regions across different images, an infrared target extraction algorithm is used to extract the infrared target region. Then, contrast stretching is used to enhance the low-light image. The enhanced low-light image and infrared image are decomposed through multi-layer guided filtering to obtain a base layer and multiple detail layers. The base layer retains a large amount of basic brightness information. By using a saliency detection algorithm to generate weights for the base layer, the saliency information in the image is effectively enhanced, while maintaining the gradient information of the image's grayscale values. Figure 1 In the detail layer, the maximum regional energy is used as the fusion rule to avoid isolated points and noise points being selected as detail information and fused into the final image. Finally, the extracted infrared target regions are given certain weights to enhance the fused image, so that more infrared saliency information is retained in the fused image.

[0082] Based on the same inventive concept, the present invention also provides a multi-source data fusion image recognition system, which is used to implement the above-mentioned multi-source data fusion image recognition method;

[0083] Based on the same inventive concept, the present invention also provides a multi-source data fusion image recognition server, the server being used to implement the above-mentioned multi-source data fusion image recognition method;

[0084] Based on the same inventive concept, the present invention also provides a multi-source data fusion image recognition device, which is used to implement the above-mentioned multi-source data fusion image recognition method;

[0085] Based on the same inventive concept, the present invention also provides a multi-source data fusion image recognition platform, which is used to implement the above-mentioned multi-source data fusion image recognition method;

[0086] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including assembly or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to said program, or in multiple co-located files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communications network.

[0087] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0089] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0090] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multi-source data fusion image recognition method, characterized in that, Include: Step S1: Perform joint calibration on the infrared image camera and the low-light image camera to obtain the mapping relationship between the coordinates of the infrared image camera and the rotation angle of the low-light image camera; acquire infrared images and low-light images in real time and align the infrared images and low-light images. Step S2: Perform adaptive multi-scale decomposition on the infrared image and the low-light image respectively; specifically: starting from k=1, use the decomposition matrix to perform the k-th scale decomposition on the infrared image to obtain the ak-th basic layer and the ak-th detail layer, and perform the k-th multi-scale decomposition on the low-light image to obtain the bk-th basic layer and the bk-th detail layer. The 1k observation matrix is ​​obtained based on the ak-th and bk-th basic layers; The 2k-th observation matrix is ​​obtained based on the ak-th detail layer and the bk-th detail layer; The decomposition matrix of the (k+1)th multi-scale decomposition is determined based on the 1kth observation matrix and the 2kth observation matrix. Determine whether the decomposition cutoff condition is met. If yes, proceed to step S3; otherwise, set k=k+1 and repeat the step. Step S3: Fuse the k-th base layer and the k-th detail layer respectively to obtain a base fusion layer and a detail fusion layer. Fuse the base fusion layer and the detail fusion layer, and then use the infrared target region extracted from the infrared image to compensate the fusion result to obtain the final fused image. Use the fused image for target recognition. Step S2 specifically includes the following steps: Step S21: Perform the k-th scale decomposition on the infrared image to obtain the ak-th base layer and the ak-th detail layer; Step S22: Perform the k-th multi-scale decomposition on the low-light image to obtain the bk-th base layer and the bk-th detail layer; Step S23: Based on the ak-th base layer and the bk base layer To obtain the 1kth observation matrix Based on the ak-th detail layer and the bk detail layer To obtain the 2kth observation matrix ; Step S24: Determine the next decomposition matrix based on the 1kth observation matrix and the 2kth observation matrix; Step S25: Determine whether the decomposition cutoff condition is met. If yes, proceed to step S3; otherwise, set k=k+1 and return to step S21 to execute the next scale decomposition. The determination of whether the decomposition cutoff condition is met specifically involves: calculating the mean of the elements in the decomposition matrix. When the mean of the element is greater than or equal to the first preset value or k is greater than or equal to the second preset value, the decomposition cutoff condition is determined to be satisfied.

2. The multi-source data fusion image recognition method according to claim 1, characterized in that, Step S21 specifically involves: setting the current image based on the infrared image. Obtain the decomposition matrix ; Based on the decomposition matrix of the current image Perform smoothing to obtain the ak-th base layer. ; Set the ak detail layer And so on.

3. The multi-source data fusion image recognition method according to claim 2, characterized in that, Set the initial value of k to 1; when k=1, decompose the matrix. The initial values ​​for the decomposition matrix are the current image. Let Ia be the initial value of the infrared image; when k>=1, the decomposition matrix is... The current image is calculated in step S24. This is the ak-th base layer of the infrared image.

4. The multi-source data fusion image recognition method according to claim 3, characterized in that, Step S22 specifically involves: setting the current image based on the low-light image. Obtain the decomposition matrix Based on the decomposition matrix of the current image Perform smoothing to obtain the bk-th base layer. Set the bk detail layer And so on.

5. The multi-source data fusion image recognition method according to claim 4, characterized in that, The current image is based on the decomposition matrix. Smoothing is performed; specifically, the following steps are included: Step S2A1: Normalize the values ​​of the decomposed matrix to the range [0, 1]. Step S2A2: Instantiate the decomposition matrix based on the decomposition scenario; Step S2A3: Based on the elements in the decomposition matrix Construct a dynamic smoothing window and use it to smooth the current image. The corresponding pixel cr in Perform smoothing to obtain .

6. A platform for multi-source data fusion image recognition, characterized in that, The platform is used to implement the multi-source data fusion image recognition method according to any one of claims 1-5.

7. A server for multi-source data fusion image recognition, characterized in that, The method includes a processor coupled to a memory, the memory storing program instructions, which, when executed by the processor, implement the multi-source data fusion image recognition method according to any one of claims 1-5.

8. A multi-source data fusion image recognition system, characterized in that, The system is used to implement the multi-source data fusion image recognition method according to any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, Includes a program that, when run on a computer, causes the computer to perform the multi-source data fusion image recognition method according to any one of claims 1-5.

Images