Projectile drop point detection method, system and equipment based on dual-light decision fusion, and medium

Abstract

The invention discloses a projectile drop point detection method, system and equipment based on dual-light decision fusion, and a medium. The method comprises the following steps: collecting visible light and thermal infrared imaging data; preprocessing the data to obtain visible light and thermal infrared images of corresponding target region time-space registration; projecting a ground target area on the obtained image after space-time registration to a corresponding reference projection target surface; respectively inputting the registered images into a typical target detection model for target detection, judging whether an explosion phenomenon is detected in the images frame by frame, obtaining positioning information and category information in respective modes, dividing detection results into two categories of matched targets and unmatched targets, and outputting an explosion bounding box after secondary decision making; calculating and correcting the position of the drop point according to the explosion bounding box, and determining the image coordinate of the drop point; and mapping the coordinate of the drop point image to the reference projection target surface to obtain the position coordinate information of the drop point in the reference projection target surface. The method improves the detection precision, and better completes the target scoring task.

Description

Technical Field

[0001] The present invention relates to the technical field of impact point detection, and particularly to a method, system, device and medium for detecting the impact point of a projectile based on dual-light decision fusion. Background Art

[0002] In the field of shooting detection, especially in the outdoor long-distance large-caliber projectile training scenario, for the positioning of the impact point, there are currently methods such as manual target inspection and sighting target inspection, with poor accuracy and low efficiency. Other target reporting methods that do not require human participation, such as radar, sound waves, and light curtains, each have their limitations. Radar technology is relatively complex, vulnerable to electromagnetic interference, and has a high cost; the sound wave detection method is greatly affected by factors such as terrain and noise, and the positioning accuracy is largely affected by the layout scheme; the light curtain has problems such as complex layout and many interferences in a large field of view. The image-based method is based on the projectile explosion phenomenon and conforms to human intuition. The traditional binocular positioning method has high requirements for the site and a complex layout; the image detection method of single visible light or thermal infrared is affected by conditions such as the size and shape of the projectile explosion, with a high false alarm rate or prone to missed detections, making it difficult to accurately detect or locate the impact point of the projectile. For example, the existing patent CN202210189027.4 discloses a method based on a visible light camera using traditional image processing methods, but this method is greatly affected by environmental factors and background complexity in actual applications; the existing technology CN202311144943.7 discloses a method based on binocular vision, which automatically detects the projectile explosion area and obtains the coordinates of the projectile impact point in the image, and calculates the coordinates and distance of the projectile impact point relative to the target; however, this existing technology has a complex layout, and the selection of key parameters such as camera calibration, baseline length measurement, and angle measurement has a great impact on the results, and it is greatly affected by environmental factors.

[0003] Therefore, it is urgent to solve the above problems. Summary of the Invention

[0004] Object of the Invention: The first object of the present invention is to provide a method for detecting the impact point of a projectile based on dual-light decision fusion, which improves the detection accuracy and better completes the target reporting task.

[0005] The second object of the present invention is to provide a system for detecting the impact point of a projectile based on dual-light decision fusion.

[0006] The third object of the present invention is to provide an electronic device.

[0007] The fourth object of the present invention is to provide a computer-readable storage medium.

[0008] Technical Solution: To achieve the above objects, the present invention discloses a method for detecting the impact point of a projectile based on dual-light decision fusion, including the following steps:

[0009] S1. Collect visible light imaging data and thermal infrared imaging data of all target areas;

[0010] S2. Preprocess the visible light imaging data and thermal infrared imaging data to obtain visible light images and thermal infrared images with spatio-temporal registration for the corresponding target areas;

[0011] S3. Project the ground target areas on the spatio-temporally registered visible light images and thermal infrared images onto the corresponding reference projection target surfaces;

[0012] S4. Input the registered visible light images and thermal infrared images into a typical target detection model for target detection, frame by frame, to determine whether an explosion phenomenon is detected in the images, obtain the positioning information and category information in their respective modalities, use the result matching algorithm based on IoU to perform data association on the positioning information in different modalities, and divide the detection results into two major categories: matching targets and non-matching targets; for the matching targets, use the evidence combination rule to fuse the category and positioning information, for the non-matching targets, use the evidence supplementation rule to supplement the category and positioning information, and after secondary decision-making, output the final detection result, that is, obtain the bounding box corresponding to the explosion target in the image;

[0013] S5. According to the explosion bounding box, use the impact point calculation algorithm to calculate and correct the position of the impact point, and determine the impact point image coordinates in the explosion area; according to the coordinate mapping relationship between the reference point on the target area image and the reference projection target surface, map the impact point image coordinates to the reference projection target surface to obtain the position coordinate information of the impact point within the reference projection target surface, and record and output the relative position information of the impact point on the reference projection target surface.

[0014] Optionally, step S2 specifically includes the following steps: Configure the acquisition time sequence for the visible light imaging data and thermal infrared imaging data, so that each frame of visible light image and thermal infrared image is synchronously aligned in time, and then register and align each frame of visible light image and thermal infrared image in space to obtain visible light images and thermal infrared images with spatio-temporal registration for the corresponding target areas.

[0015] Optionally, step S3 specifically includes the following steps:

[0016] Pre-establish and store the mapping relationship of the position information between the ground target areas on the spatio-temporally registered visible light images and thermal infrared images and the reference projection target surface, and the position information is coordinate information;

[0017] Select a number of reference points in the target area in advance. The number of reference points is greater than 3, and the connected area formed by the reference points should cover the entire target area. The reference points can be selected around the target area, and the reference points show obvious features in both visible light and thermal infrared images. The obvious features include, but are not limited to, morphological features, texture features, edge features, or color features. Or prepare a calibration reference object in advance, whose surface has recognizable marking features, and select the corresponding feature points.

[0018] Record the relative position information of a number of reference points, and establish a reference projection target surface based on this information. Denote the position coordinates of the reference points on the reference projection target surface as target position information. Determine the image coordinate positions of a number of reference points in the target area image, denoted as image position information. According to the image position information and the target position information, obtain the coordinate mapping relationship of a number of reference points between the target area image and the reference projection target surface.

[0019] Optionally, the specific steps of the result matching algorithm in step S4 are as follows: Given an IoU threshold, match the detection results of the visible light modality and the thermal infrared modality. If no target is detected in both modalities, no processing is required. If a target is detected in only one modality, the detection result of this modality is taken as the unmatched target, and the matched target is empty. If targets are detected in both modalities, first calculate the IoU matrix of the detection results of different modalities, traverse the IoU matrix, and judge whether the maximum IoU value in each row exceeds the given threshold: If it exceeds, find the index corresponding to the maximum IoU value, and the detection results of different modalities corresponding to this index are both recorded as matched targets. If the maximum IoU value is still less than the threshold, it is considered that the detection result corresponding to the current row index is an unmatched target. The detection results are finally divided into two categories: matched targets and unmatched targets.

[0020] Optionally, the specific steps of fusing the category and localization information of the matched targets using the evidence combination rule in step S4 are as follows:

[0021] For the matched targets, perform category information fusion and localization information fusion respectively. When performing category information fusion, use the Dempster-Shafer evidence theory. When performing localization information fusion, integrate the localization information of the detection results of the two modalities through set operations.

[0022] The category information includes the category probabilities and confidence levels of different categories. For the fusion of category information, the category probabilities output by each modality are used as the basic probability assignments in the Dempster-Shafer evidence theory, and the probability assignment functions of the visible light modality and the thermal infrared modality are constructed. Then, according to the Dempster-Shafer evidence theory synthesis formula, the probability assignment function of the thermal infrared modality and the probability assignment function of the visible light modality are synthesized into a fused probability assignment function. The expression of the fused probability assignment function is:

[0023]

[0024] In the formula, A i and B j represent the class hypotheses of different modalities. K is the conflict coefficient, representing the sum of the conflicting parts, and its magnitude reflects the strength of the conflict between each hypothesis; the closer K is to 1, the more serious the conflict between the evidence sources, and the closer K is to 0, the more consistent the evidence sources are; m vis is the probability assignment function for the visible light modality, and m inf is the probability assignment function for the thermal infrared modality. A is the intersection of A i and B j , is an empty set;

[0025] By calculating through the fusion probability assignment function, the fusion result of the probability corresponding to each class can be obtained. What is finally obtained is the probability assignment of each class after decision fusion. According to the fusion probability assignment, the class with the highest probability is selected as the class decision output of the matching target;

[0026] According to its IoU value, the corresponding positioning information fusion strategy is determined. When 0.3 ≤ IoU < 0.7, the smallest union circumscribed rectangle covering the two detection and positioning regions is selected as the positioning information of the final output; when IoU ≥ 0.7, the intersection region is used as the final result.

[0027] Optionally, the specific steps for supplementing the class and positioning information of the unmatched target using the evidence supplement rule in step S4 are as follows:

[0028] For the unmatched target, the positioning information uses the detection and positioning region of the current unmatched target. By extracting the local contrast and local entropy features of the detection and positioning region as the evidence supplement rule, the confidence is enhanced or punished;

[0029] The local contrast reflects the brightness difference between the target region and the background region. Analyzing the change degree of the gray value between the detection target region and the background region in the thermal infrared image is:

[0030]

[0031] where, F1 is the local contrast value; δ in is the standard deviation of the gray value of the target region, reflecting the brightness of the target region; δ out is the standard deviation of the gray value of the background region, reflecting the brightness of the background region.

[0032] To correct the confidence with the local contrast on the same scale, normalization processing is performed, that is:

[0033]

[0034] Among them, Conf is the initial detection confidence, and Conf' is the corrected detection confidence; α is the contrast adjustment coefficient, T low and T high are the thresholds for low contrast and high contrast respectively; for the target with F1 < T low , it is considered that the target may be a false detection or insignificant, and the confidence is penalized; otherwise, the target is considered significant and the confidence of the detection box is increased;

[0035] Then, calculate the local entropy feature of the detection and localization region in the visible light modality, and correct the confidence; analyze the degree of change in the entropy values of the detection target region and the background region in the visible light image, that is:

[0036]

[0037] Among them, F2 is the local entropy value, H in is the entropy value of the target region, reflecting the information complexity of the target region; H out is the entropy value of the background region, reflecting the information complexity of the background region;

[0038] Similarly, perform normalization processing:

[0039]

[0040] Among them, Conf is the initial detection confidence; Conf′ is the corrected detection confidence; β is the entropy adjustment coefficient, H low and H high are the thresholds for low entropy and high entropy respectively; for the target with F2 < H low , it is considered that the target may be a false detection or insignificant, and the confidence is penalized; otherwise, the target is considered significant and the confidence of the detection box is increased;

[0041] Jointly use the local contrast feature of the thermal infrared image and the local entropy feature of the visible light image to correct the confidence. When the final confidence is greater than the given confidence threshold, the detection result is considered valid, and the unmatched target is also added to the final detection result. If the final confidence is less than the given confidence threshold, the detection result is considered invalid, and the detection result is excluded.

[0042] Optionally, step S5 specifically includes the following steps: Obtain and store the bounding boxes and corresponding category information containing explosion events in consecutive frames, analyze the rate of change and stability of the recorded bounding boxes and category information, calculate the change trends of the positions, sizes, and category confidence levels of the bounding boxes between frames, and at the same time, by statistically analyzing the consecutive frame data, eliminate abnormal results caused by instantaneous noise or misdetection; According to the bounding boxes and category information in the candidate frames, screen for the moment when a frame first satisfies the conditions of a stable bounding box and a clear category as the initial explosion moment, and extract the bounding box in this frame as the basis for impact point calculation; Use the determined bounding box to calculate the center point as the preliminary impact point image coordinates; With the help of additional information detected in multiple subsequent frames, supplement and correct the preliminary calculated impact point image coordinates, finally obtain the corrected impact point image coordinates, and output the corrected impact point image coordinates as the position basis corresponding to the initial explosion moment.

[0043] Based on the same inventive concept, the present invention discloses a projectile impact point detection system based on dual - optical decision fusion, including: A data acquisition module for acquiring visible - light imaging data and thermal - infrared imaging data of the entire target area;

[0044] A pre - processing module for pre - processing the visible - light imaging data and thermal - infrared imaging data to obtain visible - light images and thermal - infrared images with spatio - temporal registration of the corresponding target area;

[0045] A target - area mapping module for projecting the ground target area on the spatio - temporally registered visible - light image and thermal - infrared image onto the corresponding reference projection target surface;

[0046] A decision - fusion detection module for respectively inputting the registered visible - light image and thermal - infrared image into a typical target detection model for target detection, judging frame by frame whether an explosion phenomenon is detected in the image, obtaining the positioning information and category information in their respective modalities, using an IoU - based result matching algorithm to perform data association on the positioning information of different modalities, and dividing the detection results into two major categories: matching targets and non - matching targets; For matching targets, use an evidence combination rule to fuse the category and positioning information, and for non - matching targets, use an evidence supplementation rule to supplement the category and positioning information. After secondary decision - making, output the final detection result, that is, obtain the bounding box corresponding to the explosion target in the image;

[0047] An impact - point calculation module for calculating and correcting the position of the impact point using an impact - point calculation algorithm according to the explosion bounding box, and determining the impact point image coordinates in the explosion area; According to the coordinate mapping relationship between the reference point on the target - area image and the reference projection target surface, map the impact point image coordinates to the reference projection target surface to obtain the position coordinate information of the impact point within the reference projection target surface, and record and output the relative position information of the impact point on the reference projection target surface.

[0048] Optionally, in the preprocessing module, the acquisition timing of visible light imaging data and thermal infrared imaging data is configured so that each frame of visible light image and thermal infrared image is synchronized and aligned in time, and then each frame of visible light image and thermal infrared image is registered and aligned in space to obtain visible light images and thermal infrared images that are spatio-temporally registered with respect to the target area.

[0049] Optionally, in the target area mapping module, a mapping relationship between the position information of the ground target area on the visible light image and the thermal infrared image after spatio-temporal registration and the reference projection target surface is established and stored in advance, and the position information is coordinate information;

[0050] Several reference points are selected in advance in the target area. The number of reference points is greater than 3, and the connection area formed by the reference points should cover the entire area of the target area; the reference points can be selected around the target area, and the reference points have obvious features in both visible light and thermal infrared images. The obvious features include, but are not limited to, morphological features, texture features, edge features, or color features; or a calibration reference object is pre-made, and it has identifiable marking features on its surface, and corresponding feature points are selected;

[0051] Record the relative position information of several reference points, and establish a reference projection target surface based on this information. Denote the position coordinates of the reference points on the reference projection target surface as target position information; determine the image coordinate positions of several reference points in the target area image, denoted as image position information; according to the image position information and the target position information, obtain the coordinate mapping relationship of several reference points on the target area image and the reference projection target surface.

[0052] Optionally, the specific result matching algorithm in the decision fusion detection module is as follows: Given an IoU threshold, match the detection results of the visible light modality and the thermal infrared modality. If no target is detected in both, no processing is required; if a target is detected in only one modality, the detection result of this modality is regarded as an unmatched target, and the matched target is empty; if targets are detected in both modalities, first calculate the IoU matrix of the detection results of different modalities, traverse the IoU matrix, and for each row, judge whether the maximum IoU value exceeds the given threshold: If it exceeds, find the index corresponding to the maximum IoU value, and the detection results of different modalities corresponding to this index are both regarded as matched targets; if the maximum IoU value is still less than the threshold, it is considered that the detection result corresponding to the current row index is an unmatched target; the detection results are finally divided into two categories: matched targets and unmatched targets.

[0053] Optionally, the specific method for fusing the category and localization information of the matched targets using the evidence combination rule in the decision fusion detection module is as follows:

[0054] For the matching targets, category information fusion and location information fusion are performed separately. When performing category information fusion, the Dempster-Shafer evidence theory is adopted. When performing location information fusion, the location information of the two-modal detection results is integrated through set operations;

[0055] The category information includes the category probabilities and confidence levels of different categories. For the fusion of category information, the category probabilities output by each modality are used as the basic probability assignments in the Dempster-Shafer evidence theory, and the probability assignment functions of the visible light modality and the thermal infrared modality are constructed; then, according to the Dempster-Shafer evidence theory synthesis formula, the probability assignment function of the thermal infrared modality and the probability assignment function of the visible light modality are synthesized into a fusion probability assignment function. The expression of the fusion probability assignment function is:

[0056]

[0057] In the formula, A i and B j represent category hypotheses of different modalities. K is the conflict coefficient, which represents the sum of the contradictory parts, and its magnitude reflects the strength of the contradiction between each hypothesis; the closer K is to 1, the more serious the conflict between the evidence sources, and the closer K is to 0, the more consistent the evidence sources are; m vis is the probability assignment function of the visible light modality, m inf is the probability assignment function of the thermal infrared modality, A is the intersection of A i and H j , is the empty set;

[0058] Through the calculation of the fusion probability assignment function, the fusion results of the corresponding probabilities of each category can be obtained. What is finally obtained is the probability assignment of each category after decision fusion. According to the fusion probability assignment, the category with the highest probability is selected as the category decision output of the matching target;

[0059] According to its IoU value, the corresponding location information fusion strategy is determined. When 0.3 ≤ IoU < 0.7, the smallest union circumscribed rectangle covering the two detection and location regions is selected as the finally output location information; when IoU ≥ 0.7, the intersection region is used as the final result.

[0060] Optionally, for the unmatched targets in the decision fusion detection module, the specific method for supplementing category and location information using the evidence supplement rule is:

[0061] For the unmatched targets, the location information uses the detection and location region of the current unmatched target. By extracting the local contrast and local entropy features of the detection and location region as the evidence supplement rule, the confidence level is enhanced or punished;

[0062] The local contrast reflects the brightness difference between the target area and the background area. Analyzing the change degree of the gray values of the target area and the background area in the thermal infrared image is as follows:

[0063]

[0064] Among them, F1 is the local contrast value; δ in is the standard deviation of the gray values of the target area, reflecting the brightness of the target area; δ out is the standard deviation of the gray values of the background area, reflecting the brightness of the background area.

[0065] To correct the confidence level with the local contrast on the same scale, normalization processing is carried out, which is as follows:

[0066]

[0067] Among them, Conf is the initial detection confidence level, and Conf ′ is the corrected detection confidence level; α is the contrast adjustment coefficient, and T low and T high are the low-contrast and high-contrast thresholds respectively; for the target with F1 < T low , it is considered that the target may be misdetected or insignificant, and the confidence level is punished; otherwise, the target is considered significant and the confidence level of the detection box is increased;

[0068] Then, calculate the local entropy feature for the detection and positioning area in the visible light modality to correct the confidence level; analyze the change degree of the entropy values of the detection target area and the background area in the visible light image, which is as follows:

[0069]

[0070] Among them, F2 is the local entropy value, and H in is the entropy value of the target area, reflecting the information complexity of the target area; H out is the entropy value of the background area, reflecting the information complexity of the background area;

[0071] Similarly, normalization processing is carried out:

[0072]

[0073] Among them, Conf is the initial detection confidence level; Conf ′ is the corrected detection confidence level; β is the entropy adjustment coefficient, and H low and H high are the low-entropy and high-entropy thresholds respectively; for the target with F2 < H low , it is considered that the target may be misdetected or insignificant, and the confidence level is punished; otherwise, the target is considered significant and the confidence level of the detection box is increased;

[0074] The local contrast feature of the thermal infrared image and the local entropy feature of the visible light image are jointly used to correct the confidence level. When the final confidence level is greater than the given confidence level threshold, the detection result is considered valid, and the unmatched target is also added to the final detection result. If the final confidence level is less than the given confidence level threshold, the detection result is considered invalid, and the detection result is excluded.

[0075] Optionally, in the landing point calculation module, the bounding boxes containing explosion events and the corresponding category information in consecutive frames are acquired and stored, and the change rate and stability of the recorded bounding boxes and category information are analyzed. The change trends of the positions, sizes, and category confidence levels of the bounding boxes between frames are calculated. At the same time, by statistically analyzing the consecutive frame data, the abnormal results caused by instantaneous noise or misdetection are excluded; according to the bounding boxes and category information in the candidate frames, the moment when the bounding box is first stable and the category is clear is selected as the initial explosion moment, and the bounding box in this frame is extracted as the basis for landing point calculation; using the determined bounding box, the center point is calculated as the preliminary landing point image coordinates; with the help of the additional information detected in multiple subsequent frames, the preliminary calculated landing point image coordinates are supplemented and corrected, and finally the corrected landing point image coordinates are obtained, and the corrected landing point image coordinates are output as the position basis corresponding to the initial explosion moment.

[0076] Based on the same inventive concept, the present invention discloses an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the computer program to implement a projectile landing point detection method based on dual-light decision fusion as described above.

[0077] Based on the same inventive concept, the present invention discloses a computer-readable storage medium, on which a computer program is stored, characterized in that the computer program is executed by a processor to implement a projectile landing point detection method based on dual-light decision fusion as described above.

[0078] Advantageous effects: Compared with the prior art, the present invention has the following remarkable advantages: The present invention is simply arranged, and utilizes the complementary characteristics of two image modalities. The infrared thermal imaging modality is more sensitive to smoke and heat radiation, reducing false alarms caused by complex background interference; the visible light modality contains more detailed information, and the combination of the two improves the performance of impact point detection; the detection results of the explosion target of the present invention are divided into matching targets and unmatched targets. For matching targets, the evidence combination rule is used for class and location information fusion, and for unmatched targets, the evidence supplement rule is used for class and location information supplement. After secondary decision-making, the final detection result is output, improving the detection accuracy; for matching targets in the present invention, class information fusion and location information fusion are respectively performed. When performing class information fusion, the Dempster-Shafer evidence theory is used to fully fuse the evidence information of the two modalities in class recognition; when performing location information fusion, the location information of the detection results of the two modalities is integrated through set operations to ensure full coverage of the location information; for unmatched targets in the present invention, the location information uses the detection and location area of the current unmatched target, and the confidence level is calculated to determine whether there is a false detection or a missed detection; since the other modality does not detect this target and lacks multi-source evidence support, a secondary recognition strategy based on regional feature reconstruction is introduced. Specifically, by extracting the local contrast and local entropy features of the detection and location area as the evidence supplement rule, the confidence level is enhanced or punished targetedly, so as to more accurately identify valid targets that may be misjudged as invalid and eliminate unreliable background noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 is a schematic flowchart of the present invention;

[0080] Figure 2 is a schematic reference diagram of the actual layout of the target area in the present invention;

[0081] Figure 3 is a schematic reference diagram of the projection target surface in the present invention.

[0082] Figure 4 is a schematic framework diagram of the system in the present invention;

[0083] Figure 5 is a schematic flowchart of the decision fusion of the detection results in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0084] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0085] It should be understood that the present invention can be implemented in different forms and should not be construed as limited to the embodiments presented herein. On the contrary, providing these embodiments will make the disclosure thorough and complete, and will fully convey the scope of the present invention to those skilled in the art. In the drawings, for clarity, the dimensions and relative dimensions of the components may be exaggerated. The same reference numerals throughout the drawings denote the same components.

[0086] Embodiment 1

[0087] As Figure 1 shown, the present invention discloses a method for detecting the impact point of a projectile based on dual - light decision fusion, including the following steps:

[0088] S1. Collect visible - light imaging data and thermal - infrared imaging data of the entire target area.

[0089] S2. Pre - process the visible - light imaging data and thermal - infrared imaging data to obtain visible - light images and thermal - infrared images with spatio - temporal registration for the corresponding target areas; specifically: configure the acquisition time sequence for the visible - light imaging data and thermal - infrared imaging data so that each frame of visible - light image and thermal - infrared image is synchronized in time, and then register each frame of visible - light image and thermal - infrared image spatially to obtain visible - light images and thermal - infrared images with spatio - temporal registration for the corresponding target areas; the present invention uses a dual - light camera to collect data, and the visible - light lens and thermal - infrared lens have different resolutions and field - of - view angles respectively. Spatio - temporal registration can achieve alignment in terms of resolution and field - of - view angle; however, due to the errors of imaging distortion and pseudo - coaxial error, there are still slight offsets in local areas of the images.

[0090] S3. Project the ground target areas on the spatio - temporally registered visible - light images and thermal - infrared images onto the corresponding reference projection target surfaces;

[0091] Specifically, it includes the following steps:

[0092] Pre - establish and store the mapping relationship of the position information between the ground target areas on the spatio - temporally registered visible - light images and thermal - infrared images and the reference projection target surfaces, where the position information is coordinate information; select several reference points in advance in the target area, the number of reference points is greater than 3, and the connection area formed by the reference points should cover the entire area of the target area;

[0093] The reference points can be selected around the target area, and the reference points show obvious features on both the visible - light image and the thermal - infrared image. The obvious features include but are not limited to morphological features, texture features, edge features, or color features; alternatively, a calibration reference object can be prepared in advance, whose surface has recognizable marking features, and select the corresponding feature points;

[0094] Record the relative position information of several reference points, and establish a reference projection target surface based on this information. Denote the position coordinates of the reference points on the reference projection target surface as the target position information; determine the image coordinate positions of several reference points in the target area image, denoted as the image position information; according to the image position information and the target position information, obtain the coordinate mapping relationship of several reference points on the target area image and the reference projection target surface.

[0095] As Figure 2 and Figure 3 shown, the height of the dual - light camera is set to H. There is no specific height requirement for H. Considering the actual camera focal length, field of view angle, and the distance to the target area, it is better that the field of view of the dual - light camera covers the target area. Theoretically, setting the installation position high and the camera angle tilted downward at a certain inclination can obtain a better view of the target area; it can also be considered to deploy in combination with a tethered drone, which can hover in the air for a long time. Set reference points 51, 52, 53, and 54 around the ground - ring target on the ground plane, and use the dual - light camera to collect images of the target area to form a corresponding reference projection target surface.

[0096] S4. As Figure 5 shown, input the registered visible - light image and thermal - infrared image into the typical target detection model for target detection respectively, judge frame - by - frame whether an explosion phenomenon is detected in the image, obtain the positioning information and category information in their respective modalities, use the result matching algorithm based on IoU to perform data association on the positioning information of different modalities, and divide the detection results into two major categories: matching targets and non - matching targets. Matching targets are those where the same target is detected in both modalities, and non - matching targets are those where a target is detected only by a single modality and not detected by the other modality; for matching targets, use the evidence combination rule to fuse the category and positioning information, and for non - matching targets, use the evidence supplement rule to supplement the category and positioning information. After secondary decision - making, output the final detection result, that is, obtain the bounding box corresponding to the explosion target in the image.

[0097] The specific steps of the result matching algorithm are as follows: Given an IoU threshold, match the detection results of the visible - light modality and the thermal - infrared modality. If no target is detected in both, no processing is required; if a target is detected in only one modality, then the detection result of this modality is regarded as a non - matching target, and the matching target is empty;

[0098] If targets are detected in both modalities, first calculate the IoU matrix of the detection results of different modalities, traverse the IoU matrix, and for each row, judge whether the maximum IoU value exceeds the given threshold: If it exceeds, find the index corresponding to the maximum IoU value, and the detection results of different modalities corresponding to this index are both recorded as matching targets; if the maximum IoU value is still less than the threshold, then the detection result corresponding to the index of the current row is regarded as a non - matching target; the detection results are finally divided into two major categories: matching targets and non - matching targets.

[0099] For the matching targets, category information fusion and location information fusion are respectively carried out to obtain the final detection results of the matching targets in the two modalities, including the optimal category determination and the final bounding box. When performing category information fusion, the Dempster-Shafer evidence theory is adopted to fully fuse the evidence information of the two modalities in category recognition. When performing location information fusion, the location information of the detection results of the two modalities is integrated through set operations to ensure comprehensive coverage of the location information.

[0100] The category information includes the category probabilities and confidences of different categories. For the fusion of category information, the category probabilities output by each modality are used as the basic probability assignments in the Dempster-Shafer evidence theory to construct the probability assignment function of the visible light modality and the probability assignment function of the thermal infrared modality. Then, according to the synthesis formula of the Dempster-Shafer evidence theory, the probability assignment function of the thermal infrared modality and the probability assignment function of the visible light modality are synthesized into a fused probability assignment function, thus realizing decision-level fusion. The expression of the fused probability assignment function is:

[0101]

[0102] In the formula, A i and B j represent the category hypotheses of different modalities. K is the conflict coefficient, indicating the total sum of the contradictory parts, and its magnitude reflects the strength of the contradiction between each hypothesis. The closer K is to 1, the more serious the conflict between the evidence sources; the closer K is to 0, the more consistent the evidence sources are. m vis is the probability assignment function of the visible light modality, m inf is the probability assignment function of the thermal infrared modality, A is the intersection of A i and B j , and is the empty set.

[0103] Through the calculation of the fused probability assignment function, the fusion results of the probabilities corresponding to each category can be obtained. What is finally obtained is the probability assignment of each category after decision fusion. According to the fused probability assignment, the category with the highest probability is selected as the category decision output of the matching target.

[0104] For the fusion of location information, a set operation method is adopted. For the matching targets, the corresponding location information fusion strategy is determined according to their IoU values. When 0.3 ≤ IoU < 0.7, the smallest union circumscribed rectangle covering the two detection location regions is selected as the location information of the final output, which can ensure the integrity of the detection results and avoid missed detections caused by respective location deviations. When IoU ≥ 0.7, since the two detection location regions already highly overlap, in order to obtain a more accurate target boundary, the intersection region is used as the final result to further eliminate redundant regions and improve the location accuracy.

[0105] For unmatched targets, the positioning information can directly adopt the detection and positioning area of the current unmatched target, and calculate the confidence level to determine whether there is a false detection or a missed detection. Since the other modality did not detect this target and there is a lack of multi-source evidence support, a secondary recognition strategy based on regional feature reconstruction is introduced. Specifically, by extracting the local contrast and local entropy features of the detection and positioning area as evidence supplement rules, the confidence level is enhanced or punished specifically, so as to more accurately identify valid targets that may be misjudged as invalid and eliminate unreliable background noise.

[0106] The local contrast reflects the brightness difference between the target area and the background area. The value of the local contrast directly reflects the distinguishability between the target and the background. The higher the contrast, the greater the distinguishability between the target and the background, and the more separable the target is; the lower the contrast, the less prominent the target is and there may even be false detections. In the landing point detection task, the temperature difference between the explosion target and the background in the thermal infrared image will directly cause a change in the contrast. Analyzing the change degree of the gray value between the detection target area and the background area can be used as an auxiliary index for the detection task.

[0107]

[0108] Among them, F1 is the local contrast value; δ in is the standard deviation of the gray value of the target area, reflecting the brightness of the target area; δ out is the standard deviation of the gray value of the background area, reflecting the brightness of the background area.

[0109] Generally, the detection and positioning area with high local contrast corresponds to the explosion target, and on this basis, the confidence level can be further improved; while the detection and positioning area with low local contrast may be background noise or false detection targets, and the confidence level needs to be reduced or eliminated to reduce the risk of false detection. To correct the confidence level of the local contrast on the same scale, it is necessary to normalize it.

[0110]

[0111] Among them, Conf is the initial detection confidence level, Conf' is the corrected detection confidence level; α is the contrast adjustment coefficient, T low and T high are the thresholds for low contrast and high contrast respectively; for the target with F1 < T low , it is considered that the target may be a false detection or not prominent, and the confidence level is punished; otherwise, it is considered that the target is prominent and the confidence level of the detection box is improved.

[0112] In the unmatched targets, calculate the local entropy feature for the detection and localization region of the visible light modality and correct the confidence. High entropy values usually correspond to target regions with rich content or clear boundaries; low entropy values may correspond to the background or noise, and the confidence needs to be penalized. By analyzing the degree of entropy value change between the detected target region and the background region in the visible light image, it can provide a reference for the detection task.

[0113]

[0114] Among them, F2 is the local entropy value, and H in is the entropy value of the target region, reflecting the information complexity of the target region; H out is the entropy value of the background region, reflecting the information complexity of the background region;

[0115] Similarly, perform normalization processing:

[0116]

[0117] Among them, Conf is the initial detection confidence; Conf ′ is the corrected detection confidence; β is the entropy adjustment coefficient, and H low and H high are the thresholds of low entropy and high entropy respectively; for the target with F2 < H low , it is considered that the target may be a false detection or insignificant, and the confidence is penalized; otherwise, the target is considered significant and the confidence of the detection box is increased.

[0118] Combining the local contrast of the thermal infrared image and the local entropy of the visible light image can make full use of the characteristics of the two modalities to more accurately correct the confidence of the detection result; the detection and localization regions with high contrast and high entropy values often correspond to real and significant explosion targets, while the detection and localization regions with low contrast and low entropy values are more likely to be background noise or false detections. Jointly use the local contrast feature of the thermal infrared image and the local entropy feature of the visible light image to adjust the detection result. For the unmatched targets, calculate the local contrast in the corresponding thermal infrared image region to measure the brightness difference between the target region and the background region; calculate the local entropy in the corresponding visible light image region to measure the texture complexity of the target region; the core is to judge whether to enhance or penalize the target confidence by comparing the reconstructed feature differences between the target region and the background region; when the final confidence is greater than the given confidence threshold, it is considered that the detection result is valid, and the unmatched target is also added to the final detection result. If the final confidence is less than the given confidence threshold, it is considered that the detection result is invalid, and the detection result is excluded.

[0119] S5. Determine the landing point image coordinates through the explosion bounding box. According to the coordinate mapping relationship between the landing point image coordinates and the predefined reference points on the target area image and the reference projection target surface, map the calculated landing point image coordinates to the reference projection target surface, and record and output the relative position information of the landing point on the reference projection target surface;

[0120] Adopt a landing point calculation algorithm. Based on the bounding box on the image corresponding to the explosion phenomenon provided by the fusion detection algorithm, correct the position of the landing point, and obtain the image coordinates of the landing point from the explosion area; According to the pre-stored coordinate mapping relationship of the reference points on the target area image and the reference projection target surface, map the image coordinates of the landing point to the reference projection target surface, and obtain the position coordinate information of the landing point in the target area, so as to know the specific coordinates of the landing point compared with the target area plane;

[0121] The landing point detection algorithm includes the following steps: Obtain and store the bounding boxes and corresponding category information of the candidate areas containing explosion phenomena in consecutive frames for subsequent judgment and statistics of the landing points; Analyze the change rate and stability of the recorded bounding boxes and category information, calculate the change trends of the positions, sizes, and category confidence levels of the bounding boxes between frames, and at the same time, by statistically analyzing the consecutive frame data, eliminate the abnormal frames caused by instantaneous noise or misdetection to ensure that only the detection results that truly reflect the initial moment of the explosion are retained; According to the bounding boxes and category information in the candidate frames, select the moment when a certain frame first meets the conditions of a stable bounding box and a clear category as the initial moment of the explosion, and extract the bounding box in this frame as the basis for calculating the landing point; Use the determined bounding box to calculate the center point as the preliminary landing point image coordinates; With the help of the additional information detected in multiple subsequent frames, supplement and correct the coordinates calculated initially, such as analyzing the change trend of the landing point coordinates in consecutive frames through linear regression, or performing an average operation on the coordinates of each frame, or learning and fitting through a machine learning model; Finally, obtain the corrected accurate image coordinates, and output the finally calculated landing point image coordinates as the position basis corresponding to the initial moment of the explosion.

[0122] Embodiment 2

[0123] As Figure 2 shown, the present invention discloses a projectile landing point detection system based on dual - light decision fusion, including: a data acquisition module for acquiring visible light imaging data and thermal infrared imaging data of the entire target area.

[0124] A preprocessing module is used to preprocess visible light imaging data and thermal infrared imaging data to obtain visible light images and thermal infrared images with spatio-temporal registration of the corresponding target areas. In the preprocessing module, the acquisition time sequence is configured for the visible light imaging data and the thermal infrared imaging data, so that each frame of visible light image and thermal infrared image is synchronized and aligned in time, and then each frame of visible light image and thermal infrared image is registered and aligned in space to obtain visible light images and thermal infrared images with spatio-temporal registration of the corresponding target areas. In the present invention, data is collected by a dual-light camera. The visible light lens and the thermal infrared lens have different resolutions and field angles in their respective imaging. Spatio-temporal registration can achieve alignment in terms of resolution and field angle. However, due to the errors of imaging distortion and pseudo-coaxial error itself, there are still slight offsets in local areas of the images.

[0125] A target area mapping module is used to project the ground target areas on the obtained visible light images and thermal infrared images with spatio-temporal registration onto the corresponding reference projection target surfaces. In the target area mapping module, the mapping relationship of the position information between the ground target areas on the visible light images and thermal infrared images with spatio-temporal registration and the reference projection target surfaces is established and stored in advance. The position information is coordinate information.

[0126] Several reference points are selected in advance in the target area. The number of reference points is greater than 3, and the connecting area formed by the reference points should cover the entire area of the target area. The reference points can be selected around the target area. The reference points present obvious features in both visible light and infrared thermal imaging images. The obvious features include, but are not limited to, morphological features, texture features, edge features or color features. Or a calibration reference object is manufactured in advance, and it has marked features convenient for identification, and corresponding feature points are selected.

[0127] Record the relative position information of several reference points, and establish a reference projection target surface according to this information. Denote the position coordinates of the reference points on the reference projection target surface as target position information. Determine the image coordinate positions of several reference points in the target area image, denoted as image position information. According to the target position information and the image position information, obtain the coordinate mapping relationship of several reference points on the target area image and the reference projection target surface.

[0128] As Figure 2 and Figure 3 shown, the height of the dual-light camera is set to H. There is no clear height requirement for the height H. Considering the actual camera focal length, field angle and the distance to the target area, it is better that the field of view of the dual-light camera covers the target area. Theoretically speaking, setting the installation position high and the camera angle downward at a certain inclination angle can obtain a better field of view of the target area. It is also possible to consider deploying in combination with a tethered drone, which can hover in the air for a long time. Set reference point 51, reference point 52, reference point 53 and reference point 54 around the ground ring target on the ground plane, and use the dual-light camera to collect images of the target area to form the corresponding reference projection target surface.

[0129] As Figure 5As shown in the figure, the decision fusion detection module is used to input the registered visible light image and thermal infrared image into the typical target detection model respectively for target detection, and judge frame by frame whether an explosion phenomenon is detected in the image, obtain the positioning information and category information in their respective modalities, and use the result matching algorithm based on IoU to perform data association on the positioning information of different modalities. The detection results are divided into two categories: matching targets and unmatched targets. Matching targets are those where the same target is detected in both modalities, and unmatched targets are those where the target is detected only by a single modality and not detected by the other modality. For matching targets, the evidence combination rule is used to fuse the category and positioning information, and for unmatched targets, the evidence supplementation rule is used to supplement the category and positioning information. After secondary decision-making, the final detection result is output, that is, the bounding box corresponding to the explosion target in the image is obtained.

[0130] The specific steps of the result matching algorithm are as follows: Given the IoU threshold, match the detection results of the visible light modality and the thermal infrared modality. If no target is detected in both, no processing is required. If a target is detected in only one modality, the detection result of that modality is regarded as an unmatched target, and the matching target is empty.

[0131] If targets are detected in both modalities, first calculate the IoU matrix of the detection results of different modalities, traverse the IoU matrix, and judge for each row whether the maximum IoU value exceeds the given threshold: If it exceeds, find the index corresponding to the maximum IoU value, and the detection results of different modalities corresponding to this index are both recorded as matching targets; if the maximum IoU value is still less than the threshold, the detection result corresponding to the index of the current row is regarded as an unmatched target. The detection results are finally divided into two categories: matching targets and unmatched targets.

[0132] For matching targets, category information fusion and positioning information fusion are performed respectively, and the final detection results of the matching targets in both modalities can be obtained, including the optimal category determination and the final bounding box. When performing category information fusion, the Dempster-Shafer evidence theory is used to fully fuse the evidence information of the two modalities in category recognition. When performing positioning information fusion, the positioning information of the detection results of the two modalities is integrated through set operations to ensure comprehensive coverage of the position information.

[0133] The category information includes the category probabilities and confidence levels of different categories. The fusion of category information is to use the category probabilities output by each modality as the basic probability assignment in the Dempster-Shafer evidence theory to construct the probability assignment function of the visible light modality and the probability assignment function of the thermal infrared modality. Then, according to the synthesis formula of the Dempster-Shafer evidence theory, the probability assignment function of the thermal infrared modality and the probability assignment function of the visible light modality are synthesized into a fused probability assignment function, so as to achieve decision-level fusion. The expression of the fused probability assignment function is:

[0134]

[0135] wherein, A i and B j represent class hypotheses of different modalities, K is a conflict coefficient, representing the total sum of conflicting parts, and its magnitude reflects the strength of the conflict between each hypothesis; the closer K is to 1, the more serious the conflict between the evidence sources, and the closer K is to 0, the more consistent the evidence sources are; m vis is the probability assignment function of the visible light modality, m inf is the probability assignment function of the thermal infrared modality, A is the intersection of A i and B j , and is an empty set.

[0136] By calculating through the fusion probability assignment function, the fusion result of the probability corresponding to each class can be obtained. What is finally obtained is the probability assignment of each class after decision fusion. According to the fused probability assignment, the class with the highest probability is selected as the class decision output of the matching target.

[0137] For the fusion of positioning information, a set operation method is adopted. For the matching target, according to its IoU value, the corresponding positioning information fusion strategy is determined. When 0.3 ≤ IoU < 0.7, the smallest union circumscribed rectangle covering the two detection and positioning regions is selected as the finally output positioning information; this ensures the integrity of the detection result and avoids missed detection caused by respective positioning deviations. When IoU ≥ 0.7, since the two detection and positioning regions already highly overlap, in order to obtain a more accurate target boundary, the intersection region is used as the final result to further eliminate redundant regions and improve the positioning accuracy.

[0138] For the un - matched target, the positioning information can directly adopt the detection and positioning region of the current un - matched target, and calculate the confidence level to judge whether there is mis - detection or missed detection. Since the other modality does not detect this target and lacks multi - source evidence support, a secondary recognition strategy based on regional feature reconstruction is introduced. Specifically, by extracting the local contrast and local entropy features of the detection and positioning region as evidence supplement rules, the confidence level is enhanced or punished in a targeted manner, so as to more accurately identify valid targets that may be misjudged as invalid and eliminate unreliable background noise.

[0139] The local contrast reflects the brightness difference between the target region and the background region. The value of the local contrast directly reflects the distinguishability between the target and the background. The higher the contrast, the greater the distinguishability between the target and the background, and the more separable the target is; the lower the contrast, the less prominent the target and there may even be mis - detection. In the landing point detection task, the temperature difference between the explosion target and the background in the thermal infrared image will directly cause a change in the contrast. Analyzing the change degree of the gray - scale values of the detection target region and the background region can be used as an auxiliary index for the detection task.

[0140]

[0141] Among them, F1 is the local contrast value; δ in is the standard deviation of the gray value of the target area, reflecting the brightness of the target area; δ out is the standard deviation of the gray value of the background area, reflecting the brightness of the background area.

[0142] Generally, the detection and localization area with high local contrast corresponds to the explosion target, and the confidence can be further improved on this basis; while the detection and localization area with low local contrast may be background noise or misdetected targets, and the confidence needs to be reduced or eliminated to reduce the risk of misdetection. To correct the confidence with local contrast on the same scale, it is necessary to normalize it.

[0143]

[0144] Among them, Conf is the initial detection confidence, Conf ′ is the corrected detection confidence; α is the contrast adjustment coefficient, T low and T high are the thresholds for low contrast and high contrast respectively; for the target with F1 < T low , it is considered that the target may be misdetected or insignificant, and the confidence is punished; otherwise, the target is considered significant and the confidence of the detection box is increased.

[0145] Among the unmatched targets, calculate the local entropy feature for the detection and localization area of the visible light modality and correct the confidence; a high entropy value usually corresponds to a target area with rich content or clear boundaries; a low entropy value may correspond to the background or noise, and the confidence needs to be punished; by analyzing the change degree of the entropy values of the detected target area and the background area in the visible light image, it can provide a reference for the detection task.

[0146]

[0147] Among them, F2 is the local entropy value, H in is the entropy value of the target area, reflecting the information complexity of the target area; H out is the entropy value of the background area, reflecting the information complexity of the background area;

[0148] Similarly, perform normalization processing:

[0149]

[0150] Among them, Conf is the initial detection confidence; Conf ′ is the corrected detection confidence; β is the entropy adjustment coefficient, H low and H highThresholds for low entropy and high entropy respectively; for F2 < H low For the target, if the target is considered likely to be a false detection or insignificant, the confidence is penalized; otherwise, the target is considered significant and the confidence of the detection box is increased.

[0151] By combining the local contrast of the thermal infrared image and the local entropy of the visible light image, the characteristics of both modalities can be fully utilized to more accurately correct the confidence of the detection results; regions with high contrast and high entropy values in the detection and localization often correspond to real and significant explosion targets, while regions with low contrast and low entropy values are more likely to be background noise or false detections. The local contrast feature of the thermal infrared image and the local entropy feature of the visible light image are jointly used to adjust the detection results. For unmatched targets, the local contrast is calculated in the corresponding thermal infrared image region to measure the brightness difference between the target region and the background region; the local entropy is calculated in the corresponding visible light image region to measure the texture complexity of the target region; the core is to judge whether to enhance or penalize the target confidence by comparing the reconstructed feature differences between the target region and the background region; when the final confidence is greater than the given confidence threshold, the detection result is considered valid and the unmatched target is also added to the final detection results. If the final confidence is less than the given confidence threshold, the detection result is considered invalid and the detection result is excluded.

[0152] The landing point calculation module is used to calculate and correct the position of the landing point according to the explosion bounding box using the landing point calculation algorithm, and determine the landing point image coordinates from the explosion area; according to the coordinate mapping relationship of the reference point on the target area image and the reference projection target surface, map the landing point image coordinates to the reference projection target surface to obtain the position coordinate information of the landing point within the reference projection target surface, and record and output the relative position information of the landing point on the reference projection target surface. In the landing point calculation module, the bounding boxes and corresponding category information of the candidate areas containing explosion phenomena in consecutive frames are obtained and stored, and the change rate and stability of the recorded bounding boxes and category information are analyzed, and the change trends of the positions, sizes and category confidences of the bounding boxes between frames are calculated. At the same time, by statistically analyzing the consecutive frame data, abnormal frames caused by instantaneous noise or false detections are excluded; according to the bounding boxes and category information in the candidate frames, select the moment when a frame first satisfies the conditions of a stable bounding box and a clear category as the initial explosion moment, and extract the bounding box in this frame as the basis for landing point calculation; using the determined bounding box, calculate the center point as the preliminary landing point image coordinates; with the help of additional information from multiple frame detections in subsequent frames, supplement and correct the preliminary calculated landing point image coordinates, and finally obtain the corrected landing point image coordinates, and output the corrected landing point image coordinates as the position basis corresponding to the initial explosion moment.

[0153] Embodiment 3

[0154] Another embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the computer program to implement a method for detecting the impact point of a projectile based on dual-light decision fusion as described above.

[0155] The electronic device may include: a processor, a memory, a bus, and a communication interface. The processor, the communication interface, and the memory are connected through the bus; a computer program executable on the processor is stored in the memory. When the processor runs the computer program, it executes a method for detecting the impact point of a projectile based on dual-light decision fusion provided in any of the foregoing embodiments of the present invention.

[0156] Among them, the memory may include a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface (which may be wired or wireless), a communication connection is established between the device network element and at least one other network element, and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0157] The bus may be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Among them, the memory is used to store the program. After receiving the execution instruction, the processor executes the program. A method for detecting the impact point of a projectile based on dual-light decision fusion disclosed in any of the foregoing embodiments of the present invention can be applied to the processor or implemented by the processor.

[0158] A processor may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor or instructions in the form of software. The above-mentioned processor may be a general-purpose processor, which may include a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute each method, step, and logic block diagram disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, register, and other mature storage media in the art. This storage media is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0159] The electronic device provided by the embodiments of the present application and the method provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run, or implemented by it.

[0160] Embodiment 4

[0161] Another embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed by a processor to implement the method of any of the above embodiments. The computer-readable storage medium is an optical disc, on which a computer program (i.e., a program product) is stored. When the computer program is run by the processor, it will execute the method provided by any of the foregoing embodiments.

[0162] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other optical and magnetic storage media, which will not be elaborated here one by one.

[0163] The computer-readable storage medium provided by the above embodiments of the present application and the method provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run, or implemented by the application program stored on it.

Claims

1. A projectile landing point detection method based on dual - light decision fusion, characterized in that It includes the following steps: S1. Collect visible light imaging data and thermal infrared imaging data of all target areas; S2. Preprocess the visible light imaging data and thermal infrared imaging data to obtain visible light images and thermal infrared images with spatio-temporal registration for the corresponding target areas; S3. Project the ground target areas on the spatio-temporally registered visible light images and thermal infrared images onto the corresponding reference projection target surfaces; S4. Input the registered visible light images and thermal infrared images into a typical target detection model for target detection, frame by frame determine whether an explosion phenomenon is detected in the images, obtain the localization information and category information in their respective modalities, use the result matching algorithm based on IoU to perform data association on the localization information in different modalities, and divide the detection results into two major categories: matching targets and unmatched targets; for the matching targets, use the evidence combination rule to fuse the category and localization information, for the unmatched targets, use the evidence supplementation rule to supplement the category and localization information, and output the final detection result after secondary decision-making, that is, obtain the bounding box corresponding to the explosion target in the image; S5. According to the explosion bounding box, use the landing point calculation algorithm to calculate and correct the position of the landing point, and determine the landing point image coordinates in the explosion area; according to the coordinate mapping relationship between the reference point in the target area image and the reference projection target surface, map the landing point image coordinates to the reference projection target surface to obtain the position coordinate information of the landing point within the reference projection target surface, and record and output the relative position information of the landing point on the reference projection target surface.

2. The method for detecting the projectile landing point based on dual - light decision fusion according to claim 1, characterized in that, The specific steps of step S2 include the following: Configure the acquisition time sequence for the visible light imaging data and thermal infrared imaging data, so that each frame of visible light image and thermal infrared image is synchronously aligned in time, and then register and align each frame of visible light image and thermal infrared image in space to obtain visible light images and thermal infrared images with spatio-temporal registration for the corresponding target areas.

3. A method for detecting the impact point of a projectile based on dual - light decision fusion according to claim 1, characterized in that: The specific steps of step S3 include the following: Pre-establish and store the mapping relationship of the position information between the ground target areas on the spatio-temporally registered visible light images and thermal infrared images and the reference projection target surface, and the position information is coordinate information; Select several reference points in the target area in advance, the number of reference points is greater than 3, and the connection area formed by the reference points should cover the entire area of the target area; the reference points can be selected around the target area, and the reference points show obvious features in both visible light and thermal infrared images, and the obvious features include but are not limited to morphological features, texture features, edge features or color features; or pre-manufacture a reference object for calibration, whose surface has recognizable marking features, and select the corresponding feature points; Record the relative position information of several reference points, and establish a reference projection target surface based on this information, and record the position coordinates of the reference points on the reference projection target surface as the target position information; determine the image coordinate positions of several reference points in the target area image, and record them as the image position information; according to the image position information and the target position information, obtain the coordinate mapping relationship of several reference points between the target area image and the reference projection target surface.

4. A method for detecting the impact point of a projectile based on dual - light decision fusion according to claim 1, characterized in that, The specific steps of the result matching algorithm in step S4 are as follows: Given an IoU threshold, match the detection results of the visible light modality and the thermal infrared modality. If no target is detected in both modalities, no processing is required. If a target is detected in only one modality, the detection result of that modality is regarded as an unmatched target, and the matched target is empty. If targets are detected in both modalities, first calculate the IoU matrix of the detection results of different modalities, traverse the IoU matrix, and judge whether the maximum IoU value in each row exceeds the given threshold: If it exceeds, find the index corresponding to the maximum IoU value, and the detection results of different modalities corresponding to this index are both recorded as matched targets. If the maximum IoU value is still less than the threshold, the detection result corresponding to the index of the current row is regarded as an unmatched target. The detection results are finally divided into two categories: matched targets and unmatched targets.

5. A method for detecting the impact point of a projectile based on dual - light decision fusion according to claim 1, characterized in that, The specific steps of fusing the category and location information of the matched targets using the evidence combination rule in step S4 are as follows: For the matched targets, perform category information fusion and location information fusion respectively. When performing category information fusion, use the Dempster-Shafer evidence theory. When performing location information fusion, integrate the location information of the detection results of the two modalities through set operations. The category information includes the category probabilities and confidence levels of different categories. For the fusion of category information, the category probabilities output by each modality are used as the basic probability assignments in the Dempster-Shafer evidence theory to construct the probability assignment function of the visible light modality and the probability assignment function of the thermal infrared modality. Then, according to the synthesis formula of the Dempster-Shafer evidence theory, synthesize the probability assignment function of the thermal infrared modality and the probability assignment function of the visible light modality into a fused probability assignment function. The expression of the fused probability assignment function is: where A i and B j represent class hypotheses of different modalities, K is the conflict coefficient, representing the sum of the conflicting parts, and its magnitude reflects the strength of the conflict between each hypothesis; the closer K is to 1, the more serious the conflict between the evidence sources, and the closer K is to 0, the more consistent the evidence sources are; m vis is the probability assignment function of the visible light modality, m inf is the probability assignment function of the thermal infrared modality, A is the intersection of A i and B j , and the empty set; The fusion result of the probability corresponding to each category can be obtained by calculating through the fused probability assignment function. Finally, the probability assignment of each category after decision fusion is obtained. According to the fused probability assignment, select the category with the highest probability as the category decision output of the matched target. Determine the corresponding location information fusion strategy according to its IoU value. When 0.3 ≤ IoU < 0.7, select the smallest union bounding rectangle covering the two detection and location regions as the finally output location information. When IoU ≥ 0.7, use the intersection region as the final result.

6. A method for detecting the impact point of a projectile based on dual - light decision fusion according to claim 1, characterized in that, The specific steps of supplementing the category and location information of the unmatched targets using the evidence supplementation rule in step S4 are as follows: For the unmatched targets, the location information uses the detection and location region of the current unmatched target. By extracting the local contrast and local entropy features of the detection and location region as the evidence supplementation rule, enhance or punish the confidence level. The local contrast reflects the brightness difference between the target region and the background region. Analyze the change degree of the gray values of the detection target region and the background region in the thermal infrared image, that is: Among them, F1 is the local contrast value; δ in is the standard deviation of the gray value of the target area, reflecting the brightness of the target area; δ out is the standard deviation of the gray value of the background area, reflecting the brightness of the background area. To correct the confidence level by the local contrast at the same scale, perform normalization processing, that is: Among them, Conf is the initial detection confidence, and Conf′ is the corrected detection confidence; α is the contrast adjustment coefficient, T low and T high are the low-contrast and high-contrast thresholds respectively; for the target with F1 < T low , it is considered that the target may be a false detection or insignificant, and the confidence is penalized; otherwise, the target is considered significant and the confidence of the detection box is increased; Calculate the local entropy feature for the detection and localization region in the visible light modality, and correct the confidence level; analyze the entropy value change degree between the detected target region and the background region in the visible light image, that is: Among them, F2 is the local entropy value, and H in is the entropy value of the target region, reflecting the information complexity of the target region; H out is the entropy value of the background region, reflecting the information complexity of the background region; Perform the same normalization process: Among them, Conf is the initial detection confidence; Conf′ is the corrected detection confidence; β is the entropy adjustment coefficient, H low and H high are the thresholds of low entropy and high entropy respectively; for the target with F2 < H low , it is considered that the target may be a false detection or insignificant, and the confidence is punished; otherwise, the target is considered significant and the confidence of the detection box is increased; Jointly use the local contrast feature of the thermal infrared image and the local entropy feature of the visible light image to correct the confidence level. When the final confidence level is greater than the given confidence level threshold, the detection result is considered valid, and the unmatched target is also added to the final detection result. If the final confidence level is less than the given confidence level threshold, the detection result is considered invalid, and the detection result is excluded.

7. The method for detecting the projectile landing point based on dual - light decision fusion according to claim 1, wherein The specific steps of step S5 include the following steps: Obtain and store the bounding boxes and corresponding category information containing explosion events in consecutive frames, analyze the change rate and stability of the recorded bounding boxes and category information, calculate the change trends of the positions, sizes, and category confidence levels of the bounding boxes between frames, and at the same time, by statistically analyzing the consecutive frame data, eliminate abnormal results caused by instantaneous noise or misdetection; According to the bounding boxes and category information in the candidate frames, screen which frame first meets the moment when the bounding box is stable and the category is clear as the initial explosion moment, and extract the bounding box in this frame as the basis for impact point calculation; Use the determined bounding box to calculate the center point as the preliminary impact point image coordinates; With the help of the additional information detected in multiple frames in the subsequent frames, supplement and correct the preliminary calculated impact point image coordinates, finally obtain the corrected impact point image coordinates, and output the corrected impact point image coordinates as the position basis corresponding to the initial explosion moment.

8. A projectile landing point detection system based on dual - light decision fusion, characterized in that Include: A data acquisition module for acquiring visible light imaging data and thermal infrared imaging data of all target areas; A preprocessing module for preprocessing the visible light imaging data and thermal infrared imaging data to obtain visible light images and thermal infrared images with spatio-temporal registration of the corresponding target areas; A target area mapping module for projecting the ground target area on the obtained spatio-temporally registered visible light image and thermal infrared image onto the corresponding reference projection target surface; A decision fusion detection module for inputting the registered visible light image and thermal infrared image into a typical target detection model for target detection respectively, judging frame by frame whether an explosion phenomenon is detected in the image, obtaining the localization information and category information in their respective modalities, using an IoU-based result matching algorithm to perform data association on the localization information of different modalities, and classifying the detection results into two categories: matching targets and unmatched targets; For matching targets, use the evidence combination rule to fuse the category and localization information, and for unmatched targets, use the evidence supplement rule to supplement the category and localization information, and output the final detection result after secondary decision-making, that is, obtain the bounding box corresponding to the explosion target in the image; An impact point calculation module for calculating and correcting the position of the impact point using an impact point calculation algorithm according to the explosion bounding box, and determining the impact point image coordinates in the explosion area; According to the coordinate mapping relationship between the reference point on the target area image and the reference projection target surface, map the impact point image coordinates to the reference projection target surface to obtain the position coordinate information of the impact point within the reference projection target surface, and record and output the relative position information of the impact point on the reference projection target surface.

9. The bullet impact point detection system based on dual - light decision fusion according to claim 8, wherein: In the preprocessing module, the acquisition timing of visible light imaging data and thermal infrared imaging data is configured so that each frame of visible light image and thermal infrared image is synchronized and aligned in time. Then, each frame of visible light image and thermal infrared image is registered and aligned in space to obtain the visible light image and thermal infrared image with spatio-temporal registration corresponding to the target area.

10. A bullet impact point detection system based on dual - light decision fusion according to claim 8, characterized in that: In the target area mapping module, a mapping relationship of the position information between the ground target area on the visible light image and thermal infrared image after spatio-temporal registration and the reference projection target surface is established and stored in advance. The position information is coordinate information. Several reference points are selected in the target area in advance. The number of reference points is greater than 3, and the connected area formed by the reference points should cover the entire area of the target area. The reference points can be selected around the target area, and the reference points show obvious features in both visible light and thermal infrared images. The obvious features include, but are not limited to, morphological features, texture features, edge features or color features. Or a calibration reference object is made in advance, and it has recognizable marking features on its surface, and the corresponding feature points are selected. Record the relative position information of several reference points, and establish a reference projection target surface based on this information. Denote the position coordinates of the reference points on the reference projection target surface as the target position information. Determine the image coordinate positions of several reference points in the target area image, denoted as the image position information. According to the image position information and the target position information, obtain the coordinate mapping relationship of several reference points on the target area image and the reference projection target surface.

11. A bullet impact point detection system based on dual - light decision fusion according to claim 8, characterized in that: The specific result matching algorithm in the decision fusion detection module is as follows: Given an IoU threshold, match the detection results of the visible light modality and the thermal infrared modality. If no target is detected in both, no processing is required. If a target is detected in only one modality, the detection result of this modality is regarded as an unmatched target, and the matched target is empty. If targets are detected in both modalities, first calculate the IoU matrix of the detection results of different modalities, traverse the IoU matrix, and judge whether the maximum IoU value in each row exceeds the given threshold: If it exceeds, find the index corresponding to the maximum IoU value, and the detection results of different modalities corresponding to this index are both recorded as matched targets. If the maximum IoU value is still less than the threshold, it is considered that the detection result corresponding to the index of the current row is an unmatched target. The detection results are finally divided into two categories: matched targets and unmatched targets.

12. A bullet impact point detection system based on dual - light decision fusion according to claim 8, characterized in that: In the decision fusion detection module, the specific method of using the evidence combination rule for class and location information fusion for the matched targets is as follows: For the matched targets, class information fusion and location information fusion are performed respectively. When performing class information fusion, the Dempster-Shafer evidence theory is used. When performing location information fusion, the location information of the detection results of the two modalities is integrated through set operations. The category information includes the category probabilities and confidences of different categories. The fusion of category information is to use the category probabilities output by each modality as the basic probability distribution in the Dempster-Shafer evidence theory to construct the probability distribution function of the visible light modality and the probability distribution function of the thermal infrared modality. Then, according to the synthesis formula of the Dempster-Shafer evidence theory, the probability distribution function of the thermal infrared modality and the probability distribution function of the visible light modality are synthesized into a fusion probability distribution function. The expression of the fusion probability distribution function is: Wherein, A i and B j represent category hypotheses of different modalities, K is a conflict coefficient, representing the sum of contradictory parts, and its magnitude reflects the strength of the contradiction between each hypothesis; the closer K is to 1, the more serious the conflict between evidence sources, and the closer K is to 0, the more consistent the evidence sources are; m vis is the probability assignment function of the visible light modality, m inf is the probability assignment function of the thermal infrared modality, A is the intersection of A i and B j , is an empty set; The fusion result of the probability corresponding to each category can be obtained by calculating the fusion probability distribution function. The final result is the probability distribution of each category after decision fusion. According to the fused probability distribution, the category with the highest probability is selected as the category decision output of the matching target; The corresponding positioning information fusion strategy is determined according to its IoU value. When 0.3≤IoU<0.7, the minimum union external rectangular box covering the two detection positioning areas is selected as the final output positioning information; when IoU≥0.7, the intersection area is used as the final result.

13. A bullet impact point detection system based on dual - light decision fusion according to claim 8, characterized in that: The specific method for supplementing the category and positioning information of the unmatched target using the evidence supplement rules in the decision fusion detection module is as follows: For unmatched targets, the positioning information uses the detection and positioning area of the current unmatched target, and enhances or penalizes the confidence by extracting the local contrast and local entropy features of the detection and positioning area as evidence supplementary rules; The local contrast reflects the brightness difference between the target area and the background area. The grayscale value change degree of the target area and the background area detected by the thermal infrared image is analyzed as follows: Among them, F1 is the local contrast value; δ in is the standard deviation of the gray value of the target area, reflecting the brightness of the target area; δ out is the standard deviation of the gray value of the background area, reflecting the brightness of the background area. In order to correct the confidence level at the same scale, the local contrast is normalized, which is: Among them, Conf is the initial detection confidence, and Conf′ is the corrected detection confidence; α is the contrast adjustment coefficient, and T low and T high are the low-contrast and high-contrast thresholds respectively; for targets with F1 < T low , it is considered that the target may be a false detection or insignificant, and the confidence is penalized; otherwise, the target is considered significant and the confidence of the detection box is increased; Then calculate the local entropy features of the detection positioning area of the visible light modality and correct the confidence; analyze the degree of change in the entropy value of the detection target area and the background area in the visible light image, which is: Among them, F2 is the local entropy value, and H in is the entropy value of the target region, reflecting the information complexity of the target region; H out is the entropy value of the background region, reflecting the information complexity of the background region; The same normalization is performed: Among them, Conf is the initial detection confidence; Conf′ is the corrected detection confidence; β is the entropy adjustment coefficient, H low and H high are the thresholds of low entropy and high entropy respectively; for the target with F2 < H low , it is considered that the target may be a false detection or insignificant, and the confidence is punished; otherwise, the target is considered significant and the confidence of the detection box is increased; The local contrast features of the thermal infrared image and the local entropy features of the visible light image are used together to correct the confidence. When the final confidence is greater than the given confidence threshold, the detection result is considered valid and the unmatched target is also added to the final detection result. If the final confidence is less than the given confidence threshold, the detection result is considered invalid and is discarded.

14. A bullet impact point detection system based on dual - light decision fusion according to claim 8, characterized in that: The landing point calculation module acquires and stores the bounding boxes containing explosion events and the corresponding category information in consecutive frames, analyzes the rate of change and stability of the recorded bounding boxes and category information, calculates the change trends of the positions, sizes, and category confidence levels of the bounding boxes between frames, and at the same time eliminates abnormal results caused by instantaneous noise or misdetection by statistically analyzing consecutive frame data; according to the bounding boxes and category information in the candidate frames, filters out the moment when a certain frame first meets the conditions of a stable bounding box and a clear category as the initial explosion moment, extracts the bounding box in this frame as the basis for landing point calculation; uses the determined bounding box to calculate the center point as the preliminary landing point image coordinates; with the help of additional information detected in multiple frames in subsequent frames, supplements and corrects the preliminary calculated landing point image coordinates, finally obtains the corrected landing point image coordinates, and outputs the corrected landing point image coordinates as the position basis corresponding to the initial explosion moment.

15. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the computer program to implement a projectile landing point detection method based on dual - optical decision fusion as described in any one of claims 1 - 7.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by the processor to implement a projectile landing point detection method based on dual - optical decision fusion as described in any one of claims 1 - 7.

Images