Multi-spectral target detection method, system, device and storage medium in smoke and dust environment
The multi-spectral target detection method addresses blurred images and low detection rates in smoke and dust environments by employing real-time image acquisition, guided filtering, and multispectral image fusion, enhancing recognition speed and accuracy while reducing power consumption.
Patent Information
- Application Number
- CN202111417592.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-25
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-11-25
AI Technical Summary
Existing image processing technologies struggle with blurred image target information, low detection rates, and slow recognition speeds in smoke and dust environments, particularly in battlefield conditions, and require high power consumption.
A multi-spectral target detection method utilizing real-time image acquisition, guided filtering, weight map construction, and image fusion to enhance image processing efficiency and accuracy, employing FPGA for low-power operation and multispectral image fusion using CMOS, LD-SW640, and Tigris-640 cameras.
The method achieves rapid, accurate target recognition in smoke environments with improved detection rates and reduced power consumption, suitable for real-time target tracking and recognition in intelligent ammunition guidance.
Smart Images

Figure CN114067134B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology and target tracking, and in particular to a multi-spectral target detection method, system, device and storage medium in a smoke and dust environment. Background Art
[0002] Images are important carriers of information. Active and passive interference in smoke and dust environments and battlefield environments will cause problems such as blurred image target information, missing details, and low detection rate, which seriously affects the imaging quality. For target recognition in smoke and dust environments, the image information obtained by a single sensor in the existing technology can no longer meet the target recognition work in smoke and dust fields, and multispectral image fusion technology has emerged.
[0003] Visible light images are formed by reflecting visible light, and the texture details and contrast are more suitable for human visual perception, but the imaging effect of visible light is poor under smoke and dust conditions. The image formed by short-wave infrared is very similar to that of visible light images. Compared with thermal imaging, short-wave infrared has better detail resolution and analysis capabilities, and can better identify the category of the target. Infrared images mainly rely on the thermal radiation of the object itself for imaging, highlighting the hidden targets in the background, and are not affected by lighting conditions and weather. However, when the temperature difference of the object is not obvious, the imaging details are poor. Multi-spectral fusion technology can be used to obtain high-quality images of clear targets in smoke and dust environments. For target tracking and recognition, it is often necessary to process a large amount of image or video stream information in a short time. The real-time performance of image processing using classic methods such as MATLAB is poor and it is difficult to meet the computing requirements. Field Programmable Gate Array (FPGA) is a semi-custom circuit in the field of application-specific integrated circuits. It has the characteristics of high performance and low power consumption, and can obtain clear and high-quality images with faster processing speed and wider application range. Summary of the invention
[0004] In view of the problems existing in the prior art, the present invention provides a multi-spectral target detection method, system, equipment and storage medium in a smoke environment, which can effectively and accurately identify targets under the smoke background in real time, and solve the problems of fuzzy target information, low detection rate, slow recognition speed and high power consumption under the existing smoke background.
[0005] The present invention is achieved through the following technical solutions:
[0006] The multi-spectral target detection method in a smoke environment is characterized by comprising the following steps:
[0007] S1: Real-time image or video acquisition of the target area, and conversion and storage of the acquired data in different formats;
[0008] S2: read out the converted and stored data, perform guided filtering on the data of each channel, determine the weight parameters of different image information through weight map construction, and perform image fusion and arbitration;
[0009] S3: Perform corresponding command control operations on the fused and arbitrated images, and select image processing algorithms accordingly to obtain image data streams;
[0010] S4: The image information stream is read out through the asynchronous FIFO and displayed on the corresponding display screen position, completing the multi-spectral target detection in the smoke background.
[0011] Furthermore, in step 1, the target area is photographed in real time to obtain real-time video and image sequences, and the videos and images are converted into RGB565 format and stored.
[0012] Furthermore, in step 2, the video and image are processed by guided filtering using a Lab-guided filter fusion method to convert the RGB video and image into a Lab format, wherein the Lab-guided filter fusion includes a bright channel L and two color channels a and b.
[0013] Further, the steps of converting RGB video and image to Lab format are as follows:
[0014] With the help of XYZ space,
[0015]
[0016] Where: RGB is the value of the original channel; Lab is the value after conversion; Y n , X n , Z n Usually 100, 95.047 and 108.883; M is a matrix;
[0017] The image data input from each channel is processed by improved guided filtering. The input image is p. There is a linear relationship between the input guided image I and the output image q. The least squares method is used to minimize the square sum of noise and solve the coefficients as follows:
[0018]
[0019] Where: a k and b k represents a constant parameter, a ω k Corresponds to only one pair of constant parameters;
[0020] Introduce two variable parameters A and B, and use the covariance matrix of the previous layer to linearly adjust the two constant coefficients estimated by the least squares method, as follows:
[0021]
[0022] b is the base layer image, d is the detail layer image, and the guided filter decomposition formula is as follows:
[0023]
[0024] I=b 0 =b n +d n +d n-1 +...+d 1 ;
[0025] Where: GF is the guided filter, b 0 The layer is the source image, the basic image of the nth layer is obtained by guided filtering of the n-1th layer image, and the detail image of the nth layer is the difference between the basic image of the previous layer and the basic image of this layer.
[0026] Furthermore, the step of introducing the weight graph construction into the obtained Lab file in step S2 is:
[0027] The image is decomposed by guided filtering, and there is a difference between the detail layer and the visible image information;
[0028] The relative weight of the separated detail images of each layer is calculated to fuse the detail images; among them, short-wave infrared can obtain more target information and details in a smoke environment. In the fusion rule, the detail information of short-wave infrared is added again, and the information of each layer is added to obtain the final detail layer image; the visible light and short-wave infrared basic layers can better reflect the background, and the basic layer fusion rule is further improved, and the background information of visible light and short-wave infrared is introduced again to obtain the basic layer image.
[0029] Furthermore, the calculation steps of the weight graph construction introduced in the Lab file are:
[0030]
[0031] Where: Represents the detail layer image fused by weight; Denotes the detail layer image after the addition of the near infrared image; D F Represents the fused detail layer image; B F ' represents the base layer image fused by weight; B F Represents the base layer image after the addition of visible light and infrared;
[0032] The base layer and the detail layer are superimposed to obtain a fused image, as follows:
[0033] F=B F +D F ;
[0034] Finally, the fused image data is again put into the acquisition data buffer unit for buffering via the acquisition data arbitration unit.
[0035] Multi-spectral target detection system in smoke and dust environment, including:
[0036] The data signal acquisition module is used to acquire real-time video images of the target area under the smoke background. By configuring the sensor, the acquired data is converted into 16-bit color data of RGB565 through the data capture unit, and the cross-clock domain processing of the data bus and the address bus is realized through the asynchronous FIFO. The light, near-infrared and infrared data are sequentially cached into the acquisition data cache unit through the acquisition data arbitration unit;
[0037] The multispectral image fusion module under smoke and dust background is used to perform guided filtering on the cached data, determine the fusion weights of the base layer and the detail layer, fuse the images in the multispectral image fusion unit, and output the fused data through the acquisition data arbitration unit and the acquisition data cache unit;
[0038] Serial port control module, used to control and analyze instructions, and to control the selection of corresponding image algorithms under different smoke environments;
[0039] The data signal display module is used to control the information flow sent by the display module control unit VGA to continuously read the data flow from the data buffer unit and display it on the HDMI display.
[0040] Further, the data signal acquisition module includes a CMOS visible light camera, a LD-SW640 near-infrared camera, a Tigris-640 infrared camera, a data capture unit, a collection data arbitration unit, and a collection data cache unit;
[0041] The multispectral image fusion module under smoke and dust background includes a guided filtering unit, a weight map construction unit and a multispectral image fusion unit;
[0042] The serial port control module includes a host computer unit, a serial port protocol unit, an instruction analysis unit and a control image algorithm unit;
[0043] The data signal display module includes a display data buffer unit, a display module control unit and an HDMI display.
[0044] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-spectral target detection method in a smoke environment when executing the computer program.
[0045] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the multi-spectral target detection method in a smoke environment.
[0046] Compared with the prior art, the present invention has the following beneficial technical effects:
[0047] The present invention provides a multi-spectral target detection method, system, device and storage medium in a smoke environment, which fully utilizes the computing resources of FPGA and the fusion of image information collected by multiple sensors under different spectra in the smoke environment, more quickly completes the recognition of moving targets under the smoke background, better achieves real-time target recognition and low-power operation under the smoke background, and is applied to the recognition of flight trajectory and posture during the terminal guidance process of intelligent ammunition in the smoke environment of a shooting range; the application has simple steps, improves the recognition speed and recognition accuracy of moving targets, and can be widely used in actual production detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 It is a flow chart of a multi-spectral target detection method in a smoke environment in the implementation of the present invention;
[0049] Figure 2 Schematic diagram of a multi-spectral target detection system in a smoke and dust environment in the implementation of the present invention;
[0050] Figure 3 A visible light channel image data cache diagram in the implementation of the present invention; DETAILED DESCRIPTION
[0051] The present invention is further described in detail below in conjunction with specific embodiments, which are intended to explain the present invention rather than to limit it.
[0052] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0054] The present invention provides a multi-spectral target detection method in a smoke environment, such as Figure 1 As shown, the following steps are included:
[0055] S1: Real-time image or video acquisition of the target area, and conversion and storage of the acquired data in different formats;
[0056] S2: read out the converted and stored data, perform guided filtering on the data of each channel, determine the weight parameters of different image information through weight map construction, and perform image fusion and arbitration;
[0057] S3: Perform corresponding command control operations on the fused and arbitrated images, and select image processing algorithms accordingly to obtain image data streams;
[0058] S4: The image information stream is read out through the asynchronous FIFO and displayed on the corresponding display screen position, completing the multi-spectral target detection in the smoke background.
[0059] A preferred embodiment provided by the present invention is that in step 1, the target area is photographed in real time to obtain a real-time video and image sequence, and the video and image are converted into RGB565 format and stored.
[0060] A preferred embodiment provided by the present invention is that in the step 2, the video and image are guided filtered by a Lab-guided filter fusion method to convert the RGB video and image into a Lab format, and the Lab-guided filter fusion includes a bright channel L and two color channels a and b.
[0061] Furthermore, the steps of converting the RGB video and image into Lab format are as follows:
[0062] With the help of XYZ space,
[0063]
[0064] Where: RGB is the value of the original channel; Lab is the value after conversion; Y n , X n , Z n Usually 100, 95.047 and 108.883; M is a matrix;
[0065] The image data input from each channel is processed by improved guided filtering. The input image is p. There is a linear relationship between the input guided image I and the output image q. The least squares method is used to minimize the square sum of noise and solve the coefficients as follows:
[0066]
[0067] Where: a k and b k represents a constant parameter, a ω k Corresponds to only one pair of constant parameters.
[0068] Introduce two variable parameters A and B, and use the covariance matrix of the previous layer to linearly adjust the two constant coefficients estimated by the least squares method, as follows:
[0069]
[0070] b is the base layer image, d is the detail layer image, and the guided filter decomposition formula is as follows:
[0071]
[0072] I=b 0 =b n +d n +d n-1 +...+d 1 ;
[0073] Where: GF is the guided filter, b 0 The layer is the source image, the basic image of the nth layer is obtained by guided filtering of the n-1th layer image, and the detail image of the nth layer is the difference between the basic image of the previous layer and the basic image of this layer.
[0074] Another preferred embodiment provided by the present invention is that the step of introducing the weight graph structure into the obtained Lab file in step S2 is:
[0075] The image is decomposed by guided filtering, and there is a difference between the detail layer and the visible image information;
[0076] The relative weight of the separated detail images of each layer is calculated to fuse the detail images; among them, short-wave infrared can obtain more target information and details in a smoke environment. In the fusion rule, the detail information of short-wave infrared is added again, and the information of each layer is added to obtain the final detail layer image; the visible light and short-wave infrared basic layers can better reflect the background, and the basic layer fusion rule is further improved, and the background information of visible light and short-wave infrared is introduced again to obtain the basic layer image.
[0077] Furthermore, the calculation steps of the weight graph construction introduced in the Lab file are:
[0078]
[0079] Where: Represents the detail layer image fused by weight; Denotes the detail layer image after the addition of the near infrared image; D F Represents the fused detail layer image. B F ' represents the base layer image fused by weight; B F Represents the base layer image after the addition of visible light and infrared;
[0080] The base layer and the detail layer are superimposed to obtain a fused image, as follows:
[0081] F=B F +D F ;
[0082] Finally, the fused image data is again put into the acquisition data buffer unit for buffering via the acquisition data arbitration unit.
[0083] The present invention provides a multi-spectral target detection system in a smoke environment, such as Figure 2 and Figure 3 As shown, including:
[0084] The data signal acquisition module is used to acquire real-time video images of the target area under the smoke background. By configuring the sensor, the acquired data is converted into 16-bit color data of RGB565 through the data capture unit, and the cross-clock domain processing of the data bus and the address bus is realized through the asynchronous FIFO. The light, near-infrared and infrared data are sequentially cached into the acquisition data cache unit through the acquisition data arbitration unit;
[0085] The multispectral image fusion module under smoke and dust background is used to perform guided filtering on the cached data, determine the fusion weights of the base layer and the detail layer, fuse the images in the multispectral image fusion unit, and output the fused data through the acquisition data arbitration unit and the acquisition data cache unit;
[0086] Serial port control module, used to control and analyze instructions, and to control the selection of corresponding image algorithms under different smoke environments;
[0087] The data signal display module is used to control the information flow sent by the display module control unit VGA to continuously read the data flow from the data buffer unit and display it on the HDMI display.
[0088] Another preferred embodiment provided by the present invention is that the data signal acquisition module includes a CMOS visible light camera, a LD-SW640 near-infrared camera, a Tigris-640 infrared camera, a data capture unit, an acquisition data arbitration unit and an acquisition data cache unit;
[0089] The multispectral image fusion module under smoke and dust background includes a guided filtering unit, a weight map construction unit and a multispectral image fusion unit;
[0090] The serial port control module includes a host computer unit, a serial port protocol unit, an instruction analysis unit and a control image algorithm unit;
[0091] The data signal display module includes a display data buffer unit, a display module control unit and an HDMI display.
[0092] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0093] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0094] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A multi-spectral target detection method in a smoke environment, characterized in that: The following steps are involved: S1: Real-time image or video acquisition of the target area, and conversion and storage of the acquired data in different formats; S2: read out the converted and stored data, perform guided filtering on the data of each channel, determine the weight parameters of different image information through weight map construction, and perform image fusion and arbitration; The guided filtering method is used to perform guided filtering processing on the video and the image, and the RGB video and the image are converted into the Lab format, wherein the Lab-guided filtering fusion includes a bright channel L and two color channels a and b; Convert RGB videos and images to Lab format. The steps are as follows: With the help of XYZ space, Where: RGB is the value of the original channel; Lab is the value after conversion; Y n , X n , Z n Usually 100, 95.047 and 108.883; M is a matrix; The image data input from each channel is processed by improved guided filtering. The input image is p. There is a linear relationship between the input guided image I and the output image q. The least squares method is used to minimize the square sum of noise and solve the coefficients as follows: Where: a k and b k represents a constant parameter, a ω k Corresponds to only one pair of constant parameters; Introduce two variable parameters A and B, and use the previous layer covariance matrix to linearly adjust the two constant coefficients estimated by the least squares method, as follows: b is the base layer image, d is the detail layer image, and the guided filter decomposition formula is as follows: I=b 0 =b n +d n +d n-1 +...+d 1 ; Where: GF is the guided filter, b 0 The layer is the source image, the base image of the nth layer is obtained by guided filtering of the n-1th layer image, and the detail image of the nth layer is the difference between the base image of the previous layer and the base image of this layer; S3: Perform corresponding command control operations on the fused and arbitrated images, and select image processing algorithms accordingly to obtain image data streams; S4: The image information stream is read out through the asynchronous FIFO and displayed on the corresponding display screen position, completing the multi-spectral target detection in the smoke background.
2. The multi-spectral target detection method in smoke and dust environment according to claim 1 is characterized in that: In the step 1, the target area is photographed in real time to obtain a real-time video and image sequence, and the video and image are converted into RGB565 format and stored.
3. The multi-spectral target detection method in smoke and dust environment according to claim 1 is characterized in that: The step of introducing the weight graph construction into the obtained Lab file in step S2 is: The image is decomposed by guided filtering, and there is a difference between the detail layer and the visible image information; The relative weight of the separated detail images of each layer is calculated to fuse the detail images; among them, short-wave infrared can obtain more target information and details in a smoke environment. In the fusion rule, the detail information of short-wave infrared is added again, and the information of each layer is added to obtain the final detail layer image; the visible light and short-wave infrared basic layers can better reflect the background, and the basic layer fusion rule is further improved, and the background information of visible light and short-wave infrared is introduced again to obtain the basic layer image.
4. The multi-spectral target detection method in smoke and dust environment according to claim 3 is characterized in that: The calculation steps of the weight graph construction introduced in the Lab file are: Where: Represents the detail layer image fused by weight; Denotes the detail layer image after the addition of the near infrared image; D F Represents the fused detail layer image; B F ' represents the base layer image fused by weight; B F Represents the base layer image after the addition of visible light and infrared; The base layer and the detail layer are superimposed to obtain a fused image, as follows: F=B F +D F ; Finally, the fused image data is again put into the acquisition data buffer unit for buffering via the acquisition data arbitration unit.
5. Multi-spectral target detection system in smoke and dust environment, characterized by: A multi-spectral target detection method in a smoke environment according to any one of claims 1 to 4, comprising: The data signal acquisition module is used to acquire real-time video images of the target area under the smoke background. By configuring the sensor, the acquired data is converted into 16-bit color data of RGB565 through the data capture unit, and the cross-clock domain processing of the data bus and the address bus is realized through the asynchronous FIFO. The light, near-infrared and infrared data are sequentially cached into the acquisition data cache unit through the acquisition data arbitration unit; The multispectral image fusion module under smoke and dust background is used to perform guided filtering on the cached data, determine the fusion weights of the base layer and the detail layer, fuse the images in the multispectral image fusion unit, and output the fused data through the acquisition data arbitration unit and the acquisition data cache unit; Serial port control module, used to control and analyze instructions, and to control the selection of corresponding image algorithms under different smoke environments; The data signal display module is used to control the information flow sent by the display module control unit VGA to continuously read the data flow from the data buffer unit and display it on the HDMI display.
6. The multi-spectral target detection system in smoke and dust environment according to claim 5, characterized in that: The data signal acquisition module includes a CMOS visible light camera, a LD-SW640 near-infrared camera, a Tigris-640 infrared camera, a data capture unit, a data acquisition arbitration unit and a data acquisition cache unit; The multispectral image fusion module under smoke and dust background includes a guided filtering unit, a weight map construction unit and a multispectral image fusion unit; The serial port control module includes a host computer unit, a serial port protocol unit, an instruction analysis unit and a control image algorithm unit; The data signal display module includes a display data buffer unit, a display module control unit and an HDMI display.
7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the multi-spectral target detection method in a smoke environment as described in any one of claims 1 to 4 are implemented.
8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the multi-spectral target detection method in a smoke environment as described in any one of claims 1 to 4 are implemented.
