Image and sound sensor-based fusion target scoring method, storage medium and equipment

Abstract

The invention relates to a target scoring method based on image and sound sensor fusion, a storage medium and equipment. The target scoring method is used for achieving multi-sensor fusion processing and positioning of newly-added bullet point target scoring in the outdoor live ammunition shooting process. The multi-sensor fusion target scoring device comprises a sound and near-infrared image sensor and a processing program module. Before live ammunition shooting, the near-infrared module positions the four edges and the four corners of the target sheet through the brightness difference between the target sheet and the target plate. During live ammunition shooting, the near-infrared image acquisition and recognition method is triggered through sound acquisition and recognition, and the reliability of near-infrared image processing and recognition of ammunition points is improved. The device is used for outdoor live ammunition shooting target scoring, and the problems of misrecognition and the like caused by mosquito falling and target shaking in near-infrared image processing target scoring are solved.

Description

[0001] This application is a divisional application of the Chinese invention patent application. The original application date is December 8, 2022, application number 202211581513.7, the name of the invention: Method, storage medium and device based on multi-sensor fusion reporting, publication number: CN116026190A; the applicant took the initiative to file a divisional application. Technical Field

[0002] The invention belongs to the technical field of light weapon shooting training, and specifically relates to a method, storage medium and device for reporting targets based on multi-sensor fusion, which can be used for the accurate reporting target technology in light weapon shooting training. Background Art

[0003] Nowadays, it is common to use video target reporting equipment for target reporting. Most of them use visible light or near-infrared image acquisition and processing principles to report targets. When using visual target reporting equipment outdoors to report live ammunition, the video target reporting equipment will encounter many interferences, such as interference from flying insects falling on the target, interference from the target surface being shaken by force, interference from overlapping bullet holes, etc. When used outdoors, because the target is bright and hot under the sunlight, and insects such as mosquitoes, flies, and ladybugs have the habit of being attracted to light and heat, they will stop moving after flying onto the target surface. At present, all video target reporting equipment reports targets by collecting and identifying bullet holes. It is difficult to identify insects such as flies and ladybugs that remain stationary on the target surface after flying, and it is easy to produce false target reports. Targets outdoors are easily affected by strong winds, causing the target surface to shake. When bullets hit targets or target stands, the target surface will also be subjected to force and shake. When the target surface is shaking, especially irregularly, the positions of bullet holes on the target paper image collected by the video target reporting equipment will be offset compared with the positions of old bullet holes on the actual target paper. When using frame difference method or pattern recognition and other methods to identify new bullet holes, the new bullet holes and old bullet holes will partially overlap due to displacement, making it difficult to distinguish between the new and old bullet holes, resulting in reduced target reporting accuracy. When the shooter has high shooting accuracy and the shooting scatter points are densely distributed, identification through bullet hole images alone will result in missed reports when the overlap between bullet holes is high.

[0004] In order to solve the above problems, it is necessary to add other technical means that can identify bullet points, and this means will not cause misidentification of mosquitoes and shaking. The above problems can be solved through the comprehensive use of multiple means.

[0005] Among the available means are voice recognition and thermal imaging recognition.

[0006] Among them, the purpose of sound recognition is to collect sound as a trigger signal. The traditional method is to collect the sound of the weapon at the shooter end, and use the recognized sound signal to trigger the video target reporting equipment to collect the target image and identify the bullet point to complete the target reporting. However, this method is easily affected by external factors such as on-site personnel talking, coughing, and weapon operation, resulting in false triggering.

[0007] When choosing a far-infrared thermal imager to identify live ammunition shooting, the traditional method is to use a high-density natural rubber curtain as the target medium. It has good airtightness, good shrinkage, and easy polymerization. In particular, the heat source characteristics are obvious after being hit by a bullet, which makes it easy for the far-infrared thermal imager to capture the bullet point. However, when the far-infrared thermal imager is applied to outdoor target reporting, especially in field training grounds where the site can be changed at any time, natural rubber has the disadvantages of being heavy, unportable, and high cost. In terms of the selection of thermal imagers, high-pixel far-infrared thermal imagers are expensive. If only far-infrared thermal imagers are used for live ammunition target reporting and natural rubber is used as the target medium, the entire set of equipment will be difficult to effectively promote and use. Low-cost far-infrared thermal imagers have low pixels, and the positioning accuracy of target reporting is low when used alone. Moreover, when the heat characteristics of the bullet hole dissipate, the target reporting results are undocumented and easily questioned. In addition, when simply using a far-infrared thermal imager to identify bullet points, it will also be affected by changes in the heat of the target surface such as changes in the weather and raindrops falling on the target surface, resulting in misidentification. Summary of the invention

[0008] In order to solve the above problems in the prior art, the present invention discloses a device and method for reporting targets based on multi-sensor fusion. The purpose is to provide a device and a method for reporting targets based on the fusion processing of sound, near-infrared image and far-infrared image, a storage medium and a device, which respectively obtain the sound generated by the weapon firing and hitting the target board, and the characteristics of the newly added bullet holes in the target paper, and perform fusion processing according to preset steps to accurately locate the newly added bullet holes, so as to solve the problems raised in the above background technology.

[0009] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0010] The method for reporting targets based on multi-sensor fusion is used to realize multi-sensor fusion processing and positioning of newly added bullet points and report targets accurately in real time during outdoor live-fire shooting; the method comprises the following steps:

[0011] Execute the target calibration steps of combining far and near infrared: before outdoor live-fire shooting, identify the target plate and obtain the target paper area based on the temperature difference and brightness difference of each area in the collected target area image for the far and near infrared image data, and match the pixel coordinates of the far and near infrared images;

[0012] Perform far-infrared bullet point positioning steps: during outdoor live-fire shooting, continuously collect far-infrared image data including the target paper area; locate the heat source area of ​​the candidate new bullet point according to the heat source characteristics;

[0013] Execute the near-infrared bullet point positioning step: collect near-infrared image data including the target paper area, perform secondary positioning in the candidate new bullet point heat source area according to the pixel ratio within the corresponding pixel range by frame difference method, and obtain the precise coordinates of the new bullet point according to the brightness and area thresholds; at the same time, transmit the local image of the new bullet point to the display end for target reporting.

[0014] The step of target calibration of the combination of far-infrared and near-infrared comprises:

[0015] a. Collect far-infrared image data of the target area, identify the target board boundary for the first time based on the temperature difference between the foreground and background of each area in the image, distinguish the white edge of the target paper and the black EVA target board based on the temperature difference in the target board boundary area, and locate the four edges and four corners of the target paper;

[0016] b. Collect near-infrared image data of the target area, identify the target board boundary for the first time based on the brightness difference between the foreground and background of each area in the image, accurately distinguish the white edge of the target paper and the black EVA target board within the target board boundary based on the brightness difference, locate the four edges and four corners of the target paper, and obtain the accurate area of ​​the target paper;

[0017] c. Using the four corners of the target paper located by the far and near infrared images as the reference, and according to the pixel ratio of the far and near infrared images, the far and near infrared image coordinate data are matched with corresponding pixel coordinates.

[0018] The target reporting is to update the local image of the newly added bullet point on the target paper in real time, or to report the target by voice.

[0019] The storage medium based on multi-sensor fusion target reporting stores a program of the method steps as described above, which is used to realize the fusion processing of far-infrared images and near-infrared images to locate new bullet points and report targets accurately in real time during outdoor live-fire shooting.

[0020] The device for reporting targets based on multi-sensor fusion includes: a processor and a far-infrared image sensor, a near-infrared image sensor and a memory respectively connected thereto; the far-infrared image sensor is used to collect far-infrared images of targets; the near-infrared image sensor is used to collect near-infrared images of targets; a program is stored in the memory; the processor loads the program to execute the method steps as described above, so as to realize the fusion processing of far-infrared and near-infrared sensors to locate newly added bullet points and report targets accurately in real time during outdoor live-fire shooting.

[0021] The target comprises a target plate and a target paper. The target plate is a sun board pasted with black EVA, and the target paper is printed on a self-adhesive paper with a matte film on the surface.

[0022] The far-infrared image sensor and processing program module include a lens, a far-infrared thermal imaging sensor and a processing chip; the near-infrared image sensor and processing program module include a lens, a CMOS sensor and a processing chip.

[0023] The method for reporting targets based on multi-sensor fusion is used to realize multi-sensor fusion processing and positioning of newly added bullet points and report targets accurately in real time during outdoor live-fire shooting; the method comprises the following steps:

[0024] Execute the target calibration steps of combining far and near infrared: before outdoor live-fire shooting, identify the target plate and obtain the target paper area based on the temperature difference and brightness difference of each area in the collected target area image for the far and near infrared image data, and match the pixel coordinates of the far and near infrared images;

[0025] Sound triggering step: During outdoor live-fire shooting, the sound signal of the bullet hitting the target is collected, and a trigger signal is generated based on the frequency and decibel of the sound;

[0026] Execute the far-infrared bullet point positioning step: according to the trigger signal, trigger the acquisition of far-infrared image data including the target paper area; locate the heat source area of ​​the candidate newly added bullet point according to the heat source characteristics;

[0027] Execute the near-infrared bullet point positioning step: collect near-infrared image data including the target paper area, perform secondary positioning in the candidate new bullet point heat source area according to the pixel ratio within the corresponding pixel range by frame difference method, and obtain the precise coordinates of the new bullet point according to the brightness and area thresholds; at the same time, transmit the local image of the new bullet point to the display end for target reporting.

[0028] The step of target calibration of the combination of far-infrared and near-infrared comprises:

[0029] a. Collect far-infrared image data of the target area, identify the target board boundary for the first time based on the temperature difference between the foreground and background of each area in the image, distinguish the white edge of the target paper and the black EVA target board based on the temperature difference in the target board boundary area, and locate the four edges and four corners of the target paper;

[0030] b. Collect near-infrared image data of the target area, identify the target board boundary for the first time based on the brightness difference between the foreground and background of each area in the image, accurately distinguish the white edge of the target paper and the black EVA target board within the target board boundary based on the brightness difference, locate the four edges and four corners of the target paper, and obtain the accurate area of ​​the target paper;

[0031] c. Using the four corners of the target paper located by the far and near infrared images as a reference, the corresponding pixel coordinates of the far and near infrared image data coordinates are matched to obtain the pixel ratios at two different resolutions.

[0032] The target reporting is to update the local image of the newly added bullet point on the target paper in real time, or to report the target by voice.

[0033] The storage medium based on multi-sensor fusion target reporting stores a program of the method steps as described above, which is used to realize the fusion processing of sound signals, far-infrared images, and near-infrared images to locate new bullet points and report targets in real time and accurately during outdoor live-fire shooting.

[0034] The device for reporting targets based on multi-sensor fusion comprises: a processor and a sound collection sensor, a far-infrared image sensor, a near-infrared image sensor and a memory respectively connected thereto; the sound collection sensor is used to collect sound signals of bullets hitting targets; the far-infrared image sensor is used to collect far-infrared images of targets; the near-infrared image sensor is used to collect near-infrared images of targets; the memory stores a program; the processor loads the program to execute the method steps as described above, so as to realize the fusion processing of sound, far-infrared and near-infrared sensors to locate newly added bullet points and report targets accurately in real time during outdoor live-fire shooting.

[0035] The target comprises a target plate and a target paper. The target plate is a sun board pasted with black EVA, and the target paper is printed on a self-adhesive paper with a matte film on the surface.

[0036] The sound collection sensor and processing program module includes a microphone and a processing chip; the far-infrared image sensor and processing program module includes a lens, a far-infrared thermal imaging sensor and a processing chip; the near-infrared image sensor and processing program module includes a lens, a CMOS sensor and a processing chip.

[0037] The method for reporting targets based on multi-sensor fusion is used to realize multi-sensor fusion processing and positioning of newly added bullet points and report targets accurately in real time during outdoor live-fire shooting; the method comprises the following steps:

[0038] Near-infrared target calibration steps: Before outdoor live-fire shooting, collect near-infrared image data of the target area, identify the target board boundary for the first time based on the brightness difference between the foreground and background of each area in the image, accurately distinguish the white edge of the target paper and the black EVA target board within the target board boundary based on the brightness difference, locate the four edges and four corners of the target paper, and obtain the target paper area;

[0039] Sound triggering step: During outdoor live-fire shooting, the sound signal of the bullet hitting the target is collected, and a trigger signal is generated based on the frequency and decibel of the sound;

[0040] Execute the near-infrared bullet point positioning step: according to the trigger signal, trigger the acquisition of near-infrared image data including the target paper area, perform calculations in the target paper area through the frame difference method, and obtain the precise coordinates of the newly added bullet points according to the brightness and area thresholds; at the same time, transmit the local image of the newly added bullet point to the display end for target reporting.

[0041] The target reporting is to update the local image of the newly added bullet point on the target paper in real time, or to report the target by voice.

[0042] The storage medium based on multi-sensor fusion target reporting stores a program of the method steps as described above, which is used to realize the fusion processing of sound signals and near-infrared images to locate new bullet points and report targets accurately in real time during outdoor live-fire shooting.

[0043] The device for reporting targets based on multi-sensor fusion includes: a processor and a sound collection sensor, a near-infrared image sensor, and a memory respectively connected thereto; the sound collection sensor is used to collect sound signals of bullets hitting targets; the near-infrared image sensor is used to collect near-infrared images of targets; the memory stores programs; the processor loads the program to execute the method steps described above, so as to realize the fusion processing of sound and near-infrared sensors to locate newly added bullet points and report targets accurately in real time during outdoor live-fire shooting.

[0044] The target comprises a target plate and a target paper. The target plate is a sun board pasted with black EVA, and the target paper is printed on a self-adhesive paper with a matte film on the surface.

[0045] The sound collection sensor and processing program module include a microphone and a processing chip; the near-infrared image sensor and processing program module include a lens, a CMOS sensor and a processing chip.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] 1. The present invention has three acquisition sensors and corresponding processing program modules, namely, a sound acquisition sensor and processing program module, a far-infrared image sensor and processing program module, and a near-infrared image sensor and processing program module. The far-infrared image sensor and processing program module and the near-infrared image sensor and processing program module are fixed on a platform, which is conducive to the consistency of the coordinate data calibration of the two sensors and processing program modules. Moreover, both sensors and processing program modules can work in the infrared band and have the characteristics of working both day and night. At the same time, the present invention uses these two sensors and combines them to greatly improve the target recognition rate of the equipment, so that historical results can be traced through the stored near-infrared images of the real target paper. Moreover, the method of local image transmission and update greatly compresses the amount of image transmission data and reduces the pressure of network communication. At the same time, the sound acquisition sensor and processing program module are also integrated into this platform. When fused with the other two sensors and processing program modules, the data transmission speed is fast and the delay is small.

[0048] 2. The invention uses the principle of far-infrared thermal characteristics for identification, and its working wavelength is 8μm-14μm. As cold-blooded animals, insects will not generate thermal friction with the target and do not have significant heat source characteristics. Even if they fly onto the target paper, they will not be misidentified as thermal characteristics by the far-infrared image processing program module, eliminating the misidentification problem of reporting targets through near-infrared images alone; the shaking of the target paper caused by wind or other external forces also does not generate thermal friction, does not have heat source characteristics, and will not be recognized as thermal characteristics by the far-infrared image data acquisition sensor and processing program module. Even if the target surface shakes, it will not affect the reporting; because the far-infrared image processing program module can locate thermal characteristics, once the bullet penetrates the target, it will generate thermal friction and have heat source characteristics, so even if the bullet holes overlap, they can be accurately identified, which not only reduces the missed reporting rate, but also improves the accuracy of reporting targets. In addition, by identifying through sound, there will be no misidentification caused by the flying of insects and the shaking of the target. Therefore, the invention effectively solves the impact of flying insects falling on the target and the shaking of the target surface on the identification of bullet holes.

[0049] 3. When using thermal imagers for target reporting in the traditional method, the targets are made of rubber material, which can enhance the frictional heat generated by the bullet penetrating the target plate, so as to capture the brightness characteristics of the bullet point. In this patent, in order to reduce the cost and weight of consumables, the targets used are made of ordinary EVA and solar panels. While the cost is much lower than that of bullet-resistant rubber, it also causes the heat generated by the bullet rubbing against EVA to be lower than the heat generated by the bullet rubbing against rubber. To address this problem, this patent, based on the high sensitivity of the thermal imager to temperature, obtains the temperature data of the thermal imager and processes it according to the temperature threshold, which can avoid the problem of missed recognition caused by unclear image brightness differences when the infrared image is processed by the thermal imager.

[0050] 4. The targets used in the present invention include target plates and target paper. The targets are made of black EVA pasted on a polycarbonate sheet. Black EVA is used to paste the target paper. The polycarbonate sheet is used as the backing sheet. Black EVA has good light and heat absorption properties and can form a temperature difference compared to the air, which is convenient for the thermal imager to identify and locate the target plate. The target paper is a matte film covered on the printed target paper. Due to the significant difference in reflected light between the target plate and the target paper, a temperature contrast with the target plate can be formed, which is convenient for the thermal imager to identify and locate the target paper within the target plate contour. The targets, target plates and target paper used in the present invention are all common consumables with low cost and stable and reliable operation.

[0051] 5. When the patent of the present invention is in use, there is no need to install sensors or equipment as trigger switches on the weapon end or the shooter end. Instead, a sound collection and processing program module is installed on the sensor fusion reporting device, and it is placed in the target area during live-fire shooting. This not only avoids the influence of irrelevant sound factors such as on-site personnel and weapon operations, but also reduces the support equipment of the shooting range, simplifies the training preparation process, speeds up the transmission and processing of multi-sensor information, and is more conducive to the realization of related functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of using the device of the present invention in an outdoor shooting range;

[0053] Figure 2 It is a schematic diagram of the structure of the device of the present invention;

[0054] Figure 3 This is a flow chart of Example 1 of the method of the present invention;

[0055] Figure 4 This is a flow chart of Example 2 of the method of the present invention;

[0056] Figure 5 This is a flow chart of Example 3 of the method of the present invention;

[0057] Figure 6 Schematic diagram of pixels of a far-infrared image of the present invention;

[0058] Figure 7 Schematic diagram of pixels of a near infrared image of the present invention;

[0059] Figure 8 A schematic diagram of corresponding pixels of bullet hole heat source positioning in far-infrared image data after a live-fire shooting hit according to the present invention;

[0060] Fig. 9 A schematic diagram of the accurate positioning pixels of bullet holes in near-infrared image data after a live-fire shooting hit according to the present invention;

[0061] In the figure, 1 is a black EVA target plate, 2 is a target paper with a pattern printed on it, and 3 is a multi-sensor platform device. DETAILED DESCRIPTION

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The present invention is further described in detail below in conjunction with the accompanying drawings and examples.

[0064] 1. Based on multi-sensor fusion target reporting equipment, such as Figure 1-2 As shown, it mainly includes a sound acquisition sensor and processing program module, a far-infrared image acquisition sensor and processing program module, and a near-infrared image acquisition sensor and processing program module. These three are fixed on a platform device to realize data interaction between the three sensors and program modules, and complete target reporting according to the work flow and steps.

[0065] (1) Sound collection sensor and processing program module: The sound collection sensor and processing program module includes a microphone and a sound processing chip. Its main function is to collect and process the sound data of the bullet hitting the target according to the frequency and decibel of the sound. The collector is a microphone and the processing chip is an ARM processor.

[0066] (2) Far-infrared image data acquisition sensor and processing program module: It includes a far-infrared image acquisition lens, a far-infrared thermal imaging sensor and a far-infrared image processing chip. The thermal imaging sensor has an operating wavelength of 8μm-14μm and its main function is to collect and process the temperature data and image of the target area. The processing chip is an ARM processor.

[0067] (3) Near-infrared image sensor and processing program module: It includes an optical lens, a CMOS sensor and a near-infrared image processing module. The working wavelength is 0.78μm-0.94μm. Its main function is to collect near-infrared bullet point images of the target paper. The processing chip is an ARM processor.

[0068] The target in the present invention mainly comprises a target plate and a target paper. The target plate is made of a polycarbonate sheet pasted with black EVA, and the target paper is printed with self-adhesive paper and the surface is covered with a matte film.

[0069] In the process of selecting the target material, in order to ensure that the heat source characteristics of the bullet point are obvious enough, natural rubber is first considered. This is mainly because the thermal friction effect of the bullet after penetrating natural rubber is significant, and the change in the heat source characteristics of the bullet hole can reach 2°C, which can be quickly and accurately captured and identified by the far-infrared image sensor and processing program module. However, natural rubber is heavy, not portable, and expensive.

[0070] After several tests, it was found that the black EVA pasted on the four-layer polycarbonate sheet had a significant thermal friction effect after the bullet penetrated. The heat source characteristic change of the bullet hole could reach 0.5°C, and the temperature change could also be quickly captured and identified by the far-infrared image data acquisition sensor and processing program module. In addition, after the bullet hit the polycarbonate sheet pasted with EVA, it would make a clear sound, which could be identified by the sound sensor. Therefore, the black EVA with relatively low cost but satisfactory thermal friction effect was selected as the bullet-bearing target.

[0071] The sun board is a hard plastic material made of ABS and PC materials, also known as a hollow board. It has strong shaping ability and cavities, which can effectively reduce the penetration speed of bullets. Moreover, the sound after the bullet penetrates is crisp and easy to catch. Because the black EVA material is soft, the edges are easily twisted, the bullet-bearing capacity is poor, and it is easy to bend and deform after being hit by bullets. Therefore, environmentally friendly benzene-free glue is used to stick the sun board behind the black EVA, which can not only shape the black EVA, but also extend its service life.

[0072] The target paper is printed with self-adhesive paper and covered with a matte film on the surface, which can both prevent rain and scatter sunlight, increasing the temperature difference between the target paper and the edge of the black EVA, making it easier for the far-infrared image acquisition sensor and processing program module to identify and locate the target paper.

[0073] 2. Method flow: The present invention patent proposes three technical methods for outdoor live-fire shooting:

[0074] Example 1: Figure 3 As shown, it is a flow chart of Example 1 of the method of the present invention;

[0075] The far-infrared image sensor and the processing program module cooperate with the near-infrared image sensor and the processing program module to report the target.

[0076] ① Target calibration: Place the multi-sensor fusion target reporting device within 3 meters in front of the target. The far-infrared image sensor and processing program module and the near-infrared image sensor and processing program module of the device will automatically complete the target paper positioning.

[0077] Among them, the far-infrared image sensor and processing program module first automatically identifies and selects the target plate position through the temperature difference between the black EVA target plate and the air, and then automatically identifies and locates the four sides and four corners of the target paper through the temperature difference between the edge of the target paper and the black EVA target plate, and finally realizes the automatic positioning of the target paper; the near-infrared image sensor and processing program module locates the four sides and four corners of the target paper according to the image brightness difference between the white edge of the target paper and the black EVA target plate, realizes the automatic positioning of the target paper, and after positioning is completed, automatically matches the corresponding pixel coordinates with the target paper located by the far-infrared image sensor and processing program module.

[0078] ② Far-infrared bullet point positioning: During outdoor live-fire shooting, when the bullet hits the target paper on the target and penetrates the black EVA target board, the thermal friction generated by the bullet head and the black EVA target board will cause local heating of the bullet hole position, and a heat source feature will be formed at the bullet hole position. The far-infrared image sensor and processing program module will identify the position coordinates of the newly added bullet point through the heat source feature based on the heating of each pixel point in the previous and next two frames of images, and send the position coordinates to the near-infrared image sensor and processing program module.

[0079] ③ Near-infrared bullet point positioning: The near-infrared image sensor and processing program module perform image recognition within the matching corresponding coordinate pixel range according to the bullet point position coordinates provided by the far-infrared image sensor and processing program module, and perform calculations through the frame difference method to obtain the accurate coordinates of the newly added bullet points according to the brightness and area thresholds. At the same time, the local image of the bullet hole area is transmitted to the display end to update the real-time image of the target paper, and the bullet point coordinates are transmitted back to the display end to realize voice target reporting or bullet point display target reporting. The frame difference method is: the brightness value range of the near-infrared image pixel points collected by the near-infrared image sensor is 0-255. The pixels of the newly added bullet holes or the expanded bullet hole area after shooting are black. The two adjacent frames of images are processed by the frame difference method. Since the pixels in other areas remain unchanged, the brightness of the difference image at the bullet hole position will be significantly greater than that at other positions. Then, based on the brightness and area thresholds of the difference, the newly added bullet holes can be determined.

[0080] The invention uses the principle of far-infrared thermal characteristics for identification, and its working wavelength is 8μm-14μm. Insects, as cold-blooded animals, will not generate thermal friction with the target and do not have heat source characteristics. Even if they fly onto the target paper, they will not be mistakenly identified as thermal characteristics by the far-infrared image sensor and the processing program module, eliminating the false alarm problem of reporting targets through near-infrared images alone; the shaking of the target paper caused by wind or other external forces also does not generate heat source characteristics, and will not be identified as thermal characteristics by the far-infrared image sensor and the processing program module. Even if the target surface shakes, it will not affect the target reporting; because the far-infrared image sensor and the processing program module can locate the thermal characteristics, once the bullet penetrates the target, it will generate thermal friction and have heat source characteristics, so even if the bullet holes partially overlap, they can be accurately identified, which not only reduces the missed alarm rate, but also effectively solves the impact of flying insects falling on the target, target surface shaking, and bullet hole overlap on bullet hole identification.

[0081] Specific implementation method: The far-infrared image sensor and processing program module locate the bullet point according to the heat source characteristics of the bullet point, and transmit the coordinates to the near-infrared image sensor and processing program module. The near-infrared image sensor and processing program module accurately identify and locate the bullet point according to the relevant position coordinates, and transmit the coordinates and local image information back to the display end for target reporting.

[0082] ① Equipment calibration of target:

[0083] Place the multi-sensor fusion target reporting device within 3 meters in front of the target, and the device's far-infrared image sensor and processing program module and near-infrared image sensor and processing program module will automatically complete the target paper positioning.

[0084] Far infrared image sensor and processing program module: First, the target plate position is automatically identified and framed through the temperature difference between the black EVA target plate and the air. Then, the four corners of the target paper are automatically identified and located through the temperature difference between the white edge of the target paper and the black EVA target plate. Finally, the target paper is automatically positioned.

[0085] Near-infrared image sensor and processing program module: locate the four edges and four corners of the target paper according to the image brightness difference between the white edge of the target paper and the black EVA target board, and realize automatic positioning of the target paper; after positioning is completed, the corresponding pixel coordinates are automatically matched with the target paper located by the far-infrared image sensor and processing program module.

[0086] An example of the ratio of the pixels of the image processed by the far-infrared image sensor and the processing program module to the pixels of the image processed by the near-infrared image sensor and the processing program module is as follows: Figure 6 , 7 As shown, one pixel of far infrared corresponds to 6 pixel areas of near infrared.

[0087] ②Bullet point identification and positioning:

[0088] The far-infrared image sensor and processing program module communicate with the near-infrared image sensor and processing program module. When the bullet hits the target paper on the target and penetrates the black EVA target board, the thermal friction generated by the bullet head and the black EVA target board will cause the local temperature rise at the bullet hole position, and a heat source feature will be formed at the bullet hole position. The far-infrared image sensor and processing program module will identify the position coordinates of the newly added bullet point based on the temperature rise of each pixel point in the two frames of images before and after, that is, the heat source feature, and send the position coordinates to the near-infrared image sensor and processing program module. The near-infrared image sensor and processing program module performs image recognition within the matching corresponding pixel range based on the bullet point position coordinate image provided by the far-infrared image sensor and processing program module, performs calculations through the frame difference method, and obtains the accurate coordinates of the newly added bullet point based on the brightness and area thresholds. At the same time, the local image of the bullet hole area is transmitted to the display end to update the real-time image of the target paper, and the bullet point coordinates are transmitted back to the display end to realize voice target reporting or bullet point display target reporting. The frame difference method is as follows: the brightness value range of the near-infrared image pixel points collected by the near-infrared image sensor is 0-255. The pixels of the newly added bullet holes or the expanded bullet hole areas after shooting are black. The two adjacent frames of images are processed by the frame difference method. Since the pixels in other areas remain unchanged, the brightness of the difference image at the bullet hole position will be significantly greater than that at other positions. Then, the newly added bullet holes can be determined based on the brightness and area threshold of the difference.

[0089] like Figure 8 , 9 The figure shows an example of pixels of an image processed by a far-infrared image sensor and a processing program module after a live-fire hit; and an example of pixels of an image processed by a near-infrared image sensor and a processing program module after a live-fire hit.

[0090] Example 2: Figure 4 As shown, it is a flow chart of Example 2 of the method of the present invention;

[0091] The sound collection sensor and processing program module cooperate with the far-infrared image sensor and processing program module, and the near-infrared image sensor and processing program module to report the target.

[0092] ① Target calibration: Place the multi-sensor fusion target reporting device within 3 meters in front of the target. The far-infrared image sensor and processing program module and the near-infrared image sensor and processing program module of the device will automatically complete the target paper positioning.

[0093] Among them, the far-infrared image sensor and processing program module first automatically identifies and selects the target plate position through the temperature difference between the black EVA target plate and the air, and then automatically identifies and locates the four sides and four corners of the target paper through the temperature difference between the white edge of the target paper and the black EVA target plate, and finally realizes the automatic positioning of the target paper; the near-infrared image sensor and processing program module locates the four sides and four corners of the target paper according to the image brightness difference between the white edge of the target paper and the black EVA target plate, realizes the automatic positioning of the target paper, and after positioning is completed, automatically matches the corresponding pixel coordinates with the target paper located by the far-infrared image sensor and processing program module.

[0094] ② Triggering by shooting sound signal: Due to the special material structure of the solar panel, the crisp sound emitted when the bullet penetrates is very easy to identify. When the sound collection sensor and processing program module collect and identify that a bullet hits the target, it will trigger the far-infrared image sensor and processing program module to collect the target image.

[0095] ③ Far-infrared bullet point positioning: When the bullet hits the target paper on the target and penetrates the black EVA target plate, the thermal friction generated by the bullet head and the black EVA target plate will cause local heating at the bullet hole position, and a heat source feature will be formed at the bullet hole position. The far-infrared image sensor and processing program module will identify the position coordinates of the newly added bullet point based on the heating of each pixel point in the previous and next two frames of images, that is, the heat source feature, and send the position coordinates to the near-infrared image sensor and processing program module.

[0096] ④ Near-infrared bullet point positioning: The near-infrared image sensor and processing program module perform image recognition within the matching corresponding coordinate pixel range according to the bullet point position coordinates provided by the far-infrared image sensor and processing program module, and perform calculations through the frame difference method to obtain the accurate coordinates of the newly added bullet points according to the brightness and area thresholds. At the same time, the local image of the bullet hole area is transmitted to the display end to update the real-time image of the target paper, and the bullet point coordinates are transmitted back to the display end to realize voice target reporting or bullet point display target reporting. The frame difference method is: the brightness value range of the near-infrared image pixel points collected by the near-infrared image sensor is 0-255. The pixels of the newly added bullet holes or the expanded bullet hole area after shooting are black. The two adjacent frames of images are processed by the frame difference method. Since the pixels in other areas remain unchanged, the brightness of the difference image at the bullet hole position will be significantly greater than that at other positions. Then, based on the brightness and area thresholds of the difference, the newly added bullet holes can be determined.

[0097] On the basis of Example 1, a sound collection sensor and a processing program module are added, so that recognition can be performed through sound, and sound is used as a signal to trigger image acquisition. When insects fly down and the target shakes, no loud sound will be generated, thus effectively solving the problem of misidentification caused by insects flying down and target shaking. Compared with method 1, it can also solve the influence of the change of weather, raindrops falling on the target surface and other heat changes on thermal imaging recognition, and further improve the reliability of recognition. In addition, through the confirmation of the bullet point by sound and far-infrared, and the positioning of the bullet point by far-infrared, the recognition accuracy of partially overlapping bullet holes by near-infrared image processing can be significantly improved.

[0098] Specific implementation method: The sound collection sensor and processing program module collects and processes the sound signal of the bullet hitting the target, triggers the far-infrared image sensor and processing program module to identify and locate the bullet point according to the temperature change, and transmits it to the near-infrared image sensor and processing program module. The near-infrared image sensor and processing program module processes the relevant position coordinates, accurately identifies and locates the bullet point, and transmits the coordinates and local images back to the display end and reports the target.

[0099] The specific steps are as follows:

[0100] ① Equipment calibration of target:

[0101] Place the multi-sensor fusion target reporting device within 3 meters in front of the target, and the device's far-infrared image sensor and processing program module and near-infrared image sensor and processing program module will automatically complete the target paper positioning.

[0102] Far infrared image sensor and processing program module: First, the target plate position is automatically identified and framed through the temperature difference between the black EVA target plate and the air. Then, the four corners of the target paper are automatically identified and located through the temperature difference between the white edge of the target paper and the black EVA target plate. Finally, the target paper is automatically positioned.

[0103] Near-infrared image sensor and processing program module: locate the four edges and four corners of the target paper according to the image brightness difference between the white edge of the target paper and the black EVA target board, and realize automatic positioning of the target paper; after positioning is completed, the corresponding pixel coordinates are automatically matched with the target paper located by the far-infrared image sensor and processing program module.

[0104] The ratio of the pixels of the image processed by the far-infrared image sensor and the processing program module to the pixels of the image processed by the near-infrared image sensor and the processing program module is as follows: Figure 8 , 9 shown.

[0105] ②Bullet point identification and positioning:

[0106] The sound collection sensor and processing program module communicate with the far-infrared image sensor and processing program module and the near-infrared image sensor and processing program module. Due to the special material structure of the solar panel, the crisp sound emitted when the bullet penetrates is very easy to identify. When the sound collection sensor and processing program module collects and identifies that the target has been hit by a bullet, it will trigger the far-infrared image sensor and processing program module to collect the target image. When the bullet hits the target paper on the target and penetrates the black EVA target board, the thermal friction generated by the bullet head and the black EVA target board will cause the local temperature of the bullet hole position to rise, and a heat source feature will be formed at the bullet hole position. The far-infrared image sensor and processing program module will identify the position coordinates of the newly added bullet point based on the temperature rise of each pixel point in the two frames of images before and after, that is, the heat source feature, and send the position coordinates to the near-infrared image sensor and processing program module. The near-infrared image sensor and processing program module perform image recognition within the matching corresponding pixel range based on the bullet point position coordinates provided by the far-infrared image sensor and processing program module, perform calculations through the frame difference method, and obtain the precise coordinates of the newly added bullet point based on the brightness and area thresholds. At the same time, the local image of the bullet hole area is transmitted to the display end to update the real-time image of the target paper, and the bullet point coordinates are transmitted back to the display end to realize voice target reporting or bullet point display target reporting. The frame difference method is as follows: the brightness value range of the near-infrared image pixel points collected by the near-infrared image sensor is 0-255. The pixels of the newly added bullet holes or the expanded bullet hole areas after shooting are black. The two adjacent frames of images are processed by the frame difference method. Since the pixels in other areas remain unchanged, the brightness of the difference image at the bullet hole position will be significantly greater than that at other positions. Then, the newly added bullet holes can be determined based on the brightness and area threshold of the difference.

[0107] After the live ammunition is fired, the ratio of the pixels of the image processed by the far infrared image sensor and the processing program module to the pixels of the image processed by the near infrared image sensor and the processing program module is as follows: Figure 8 , 9 shown.

[0108] Example 3: Figure 6As shown, it is a flow chart of Example 3 of the method of the present invention;

[0109] The sound collection sensor and processing program module cooperate with the near-infrared image sensor and processing program module to report the target.

[0110] ① Target calibration: Place the multi-sensor fusion target reporting device within 3 meters in front of the target. The near-infrared image sensor and processing program module of the device will locate the four sides and four corners of the target paper according to the image brightness difference between the white edge of the target paper and the black EVA target board, thereby realizing automatic positioning of the target paper.

[0111] ② Triggering by shooting sound signal: Due to the special material structure of the solar panel, the crisp sound emitted when the bullet penetrates is very easy to be identified. When the sound collection sensor and processing program module collect and identify that a bullet hits the target, it will trigger the near-infrared image sensor and processing program module to collect the target image.

[0112] ③ Near-infrared bullet point positioning: The near-infrared image sensor and the processing program module perform calculations through the frame difference method, obtain the precise coordinates of the newly added bullet point according to the brightness and area thresholds, and transmit the local image near the bullet point to the display end to update the real-time image of the target paper, and transmit the bullet point coordinates back to the display end to realize voice target reporting or bullet point display reporting. The frame difference method is: the brightness value range of the near-infrared image pixel points collected by the near-infrared image sensor is 0-255. The pixels of the newly added bullet holes or the expanded bullet hole area after shooting are black. The two adjacent frames of images are processed by the frame difference method. Since the pixels in other areas remain unchanged, the brightness of the difference image at the bullet hole position will be significantly greater than that at other positions. Then, based on the brightness and area threshold of the difference, the newly added bullet hole can be determined. The method of fusion processing of the sound acquisition sensor and processing program module and the near-infrared image sensor and processing program module requires the use of sound to identify the signal of the bullet hitting the target, triggering the near-infrared image to collect and identify. Since no recognizable sound signals are generated when insects fly and the target shakes, this method effectively solves the problem of misidentification caused by insects flying and target shaking when only near-infrared image processing is used. At the same time, for the case where the frame difference method is difficult to distinguish between overlapping bullet holes and target shaking, using sound as a signal to trigger image acquisition increases the confirmation method of bullet point identification, which can improve the recognition accuracy of overlapping bullet holes by near-infrared image processing.

[0113] Specific implementation method: The sound acquisition sensor and processing program module collect and process the sound signal of the bullet hitting the target, trigger the near-infrared image data acquisition sensor and processing program module to identify the bullet point on the target paper, accurately identify the position of the bullet point, and transmit the coordinates and local image information back to the display end and report the target.

[0114] ① Target calibration:

[0115] Place the multi-sensor fusion target reporting device within 3 meters in front of the target. The near-infrared image sensor and processing program module of the device will locate the four corners of the target paper according to the image brightness difference between the white edge of the target paper and the black EVA target board, thereby realizing automatic positioning of the target paper.

[0116] ②Bullet point identification and positioning:

[0117] The sound collection sensor and processing program module communicate with the near-infrared image sensor and processing program module. Due to the special material structure of the solar panel, the crisp sound emitted when the bullet penetrates is very easy to be identified. When the sound collection sensor and processing program module collects and identifies that a bullet hits the target, it will trigger the near-infrared image sensor and processing program module to collect the target image, calculate through the frame difference method, obtain the accurate coordinates of the newly added bullet point according to the brightness and area thresholds, and transmit the local image near the bullet point to the display end to update the real-time image of the target paper, and transmit the bullet point coordinates back to the display end to realize voice target reporting or bullet point display target reporting. The frame difference method is: the brightness value range of the near-infrared image pixel point collected by the near-infrared image sensor is 0-255. The pixels of the newly added bullet hole or the expanded bullet hole area after shooting are black. The adjacent two frames of images are processed by the frame difference method. Since the pixels in other areas remain unchanged, the brightness of the difference image at the bullet hole position will be significantly greater than that at other positions. Then, according to the brightness and area thresholds of the difference, the newly added bullet hole can be determined.

[0118] Performance data:

[0119] During the development of the patent for this invention, it was found through experiments that during live-fire shooting, there was no interference from human activities, talking, or other sounds in the target area. When the multi-sensor fusion target reporting device was placed within 3 meters in front of the target, the accuracy of sound recognition could reach more than 99.5% when the bullet hit the black EVA with four layers of sun panels.

[0120] During the development of the patent for this invention, it was found through experiments that when using the 95-1 rifle and the 92 pistol to shoot at the target, the target was reported by multi-sensor fusion, and the misidentification and missed identification rate was ≤ 1‰. There was no false alarm of flying insects. In addition, the recognition rate of bullet holes with an overlap of more than 50% under natural dispersion conditions reached more than 80%.

[0121] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for reporting targets based on the fusion of image and sound sensors, which is used to realize multi-sensor fusion processing and positioning of newly added bullet points during outdoor live-fire shooting, and to report targets accurately in real time; it is characterized in that: The method comprises the following steps: Near-infrared target calibration steps: Before outdoor live-fire shooting, collect near-infrared image data of the target area, identify the target board boundary for the first time based on the brightness difference between the foreground and background of each area in the image, accurately distinguish the white edge of the target paper and the black EVA target board within the target board boundary based on the brightness difference, locate the four edges and four corners of the target paper, and obtain the target paper area; Sound triggering step: During outdoor live-fire shooting, the sound signal of the bullet hitting the target is collected, and a trigger signal is generated based on the frequency and decibel of the sound; Execute the near-infrared bullet point positioning step: according to the trigger signal, trigger the acquisition of near-infrared image data including the target paper area, perform calculations in the target paper area through the frame difference method, and obtain the precise coordinates of the newly added bullet points according to the brightness and area thresholds; at the same time, transmit the local image of the newly added bullet point to the display end for target reporting.

2. The method for targeting based on image and sound sensor fusion according to claim 1, characterized in that: The target reporting is to update the local image of the newly added bullet point on the target paper in real time, or to report the target by voice.

3. A storage medium based on image and sound sensor fusion reporting, characterized in that: The medium stores a program of the method steps as described in claims 1-2, which is used to realize the fusion processing of sound signals and near-infrared images to locate new bullet points and report targets in real time and accurately during outdoor live-fire shooting.

4. A device based on image and sound sensor fusion targeting, characterized in that: include: A processor and a sound collection sensor, a near infrared image sensor, and a memory respectively connected thereto; The sound collection sensor is used to collect the sound signal of the bullet hitting the target; the near-infrared image sensor is used to collect the near-infrared image of the target; the memory stores a program; the processor loads the program to execute the method steps as described in claims 1-2, which is used to realize the fusion processing of sound and near-infrared sensors to locate the newly added bullet points during outdoor live-fire shooting, and report the target accurately in real time.

5. The device based on image and sound sensor fusion targeting according to claim 4 is characterized in that: The target includes a target plate and a target paper. The target plate is a sun board pasted with black EVA, and the target paper is printed on self-adhesive paper with a matte film on the surface.

6. The device based on image and sound sensor fusion targeting according to claim 4 is characterized in that: The sound collection sensor and processing program module include a microphone and a processing chip; the near-infrared image sensor and processing program module include a lens, a CMOS sensor and a processing chip.

Images