A bullet hole detection method and system based on vision

Through the deep neural network and bullet hole integral graph post-processing algorithm that integrates convolutional features and texture features, the problem of bullet hole detection being sensitive to light and jitter in the existing technology is solved, and efficient and low-cost accurate bullet hole detection is achieved. Improves detection accuracy and robustness.

CN113763471BActive Publication Date: 2024-11-08ZHEJIANG LAB +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110997123.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-27
Publication Date
2024-11-08
Estimated Expiration
2041-08-27

AI Technical Summary

Technical Problem

The existing bullet hole detection method based on the frame difference method is sensitive to light and jitter, with low accuracy and high resource consumption, making it difficult to achieve efficient and low-cost accurate detection in shooting training target reporting equipment.

Method used

A deep neural detection network that integrates convolutional features and texture features is used to combine single-scale Anchor for coordinate regression, and the impact of illumination changes is reduced through the post-processing algorithm of bullet hole integral graph, improving robustness and detection accuracy.

Benefits of technology

The bullet hole detection with small resource consumption and high accuracy is achieved. The model parameters are only one-fifth of the mainstream model, the time is less than 5ms, and the deployment cost is low. It can effectively filter false detection and reduce jitter interference.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113763471B_ABST
    Figure CN113763471B_ABST
Patent Text Reader

Abstract

The invention discloses a bullet hole detection method and system based on vision, and the method comprises: S1, obtaining a target shooting bullet hole image data set and marking the bullet holes; S2, constructing an ultra-lightweight network with deep fusion of convolutional features and texture features, and performing model training to obtain a detection model based on the bullet hole image data; S3, performing reasoning based on the trained detection model to obtain a single-frame bullet hole detection result; S4, constructing a bullet hole integral map of the current frame based on multi-frame detection results; S5, performing matching and frame difference based on the bullet hole integral map of the current frame and the bullet hole integral map of the previous frame to obtain newly added bullet holes in the current frame; the system comprises: a focusing layer, a nested bottleneck layer, a convolutional layer, and a feature fusion and single-scale target regression module composed of a single-scale target regression subnetwork, a texture feature extraction unit, and a feature fusion and optimization module; the invention reduces resource consumption and has good detection accuracy and robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a vision-based bullet hole detection method and system. Background Art

[0002] In the technical field of shooting training target marking equipment, traditional target marking methods mainly include manual target marking, photoelectric sensor target marking and the image-based target marking system which has been used more recently.

[0003] Manual target reporting is extremely inefficient, and in most cases it is impossible to report targets in real time; although the photoelectric sensor target reporting method has high accuracy, it is expensive and has complex daily maintenance; therefore, the industry currently generally uses a target reporting method based on visual algorithms. The core issue of the visual target reporting solution is how to accurately detect newly added shooting bullet holes. The current mainstream bullet hole detection solutions all use the frame difference method or the multi-frame frame difference method. This method consumes less computing resources, but it is more sensitive to interference factors such as illumination and jitter, resulting in a low accuracy rate for bullet hole detection based on the frame difference method. Bullet hole detection based on deep learning has achieved a good accuracy rate, but because most detection networks are deep and consume a lot of computing resources, they require high-cost GPUs or other AI computing hardware in actual deployment.

[0004] Based on this, it is necessary to provide a bullet hole detection method with low resource consumption and high accuracy and robustness. Summary of the invention

[0005] In order to solve the problems of low detection rate and poor robustness of existing methods, and to address the problem that the frame difference method is sensitive to illumination and jitter, the present invention proposes a deep neural detection network that fuses convolutional features and texture features, uses a single-scale anchor to regress the coordinates of bullet holes, and reduces the impact of illumination changes on the results through multi-sample training and fusion with texture features; at the same time, a post-processing algorithm based on bullet hole integral images is proposed, which utilizes the irreversible characteristics of bullet hole traces, obtains the result quantized grayscale image through the bullet hole integral image, and then obtains the coordinates of the newly added bullet holes by performing frame difference after registration of the integral grayscale image. The specific technical solution is as follows:

[0006] A bullet hole detection method based on vision comprises the following steps:

[0007] S1, obtain a target shooting bullet hole image dataset and mark the bullet holes;

[0008] S2, constructs an ultra-lightweight network that deeply integrates convolutional features and texture features and trains it, including the following steps:

[0009] S21, performing multi-layer slicing on the image tensor of the target shooting image, and then recombining;

[0010] S22, inputting the reassembled image tensor into a residual-like structure to extract high-level semantic information;

[0011] S23, using the coordinates obtained by regression based on the single-scale target regression sub-network, obtains the suspected bullet hole area, extracts texture features from the suspected bullet hole area, and fuses them with the convolutional features after input into the single-scale target regression sub-network. The channel attention mechanism is used to optimize the features, and the optimized features are classified into bullet holes and non-bullet holes. The binary classification results and the labeled bullet holes are used to train the deep fusion network;

[0012] S3, performs inference based on the trained deep fusion network to obtain the bullet hole detection results of a single frame image;

[0013] S4, in the detection in the video, constructing a bullet hole integral map of the current frame based on the detection results of the multiple frames of images;

[0014] S5, based on the current frame bullet hole integral map and the previous frame bullet hole integral map, register and perform frame difference, and finally obtain the newly added bullet holes in the current frame.

[0015] Furthermore, in S1, a rectangular box is used to mark the bullet holes in each target shooting image and marked as category 0; the single-scale target regression subnetwork in S23 determines the suspected bullet hole target box (X, Y, W, H) through the coordinates obtained by regression, wherein X and Y respectively represent the center coordinates of the suspected bullet hole target box, and W and H respectively represent the width and height of the suspected bullet hole target box; texture features are extracted from the suspected bullet hole target box; the optimized features are binary classified to obtain the binary classification confidence of bullet holes and non-bullet holes, the classification result 0 represents a bullet hole, and the classification result 1 represents a non-bullet hole, and the predicted bullet hole target box obtained by the binary classification result and the labeled bullet hole rectangular box are used to train the deep fusion network.

[0016] Further, in the S23, the obtained suspected bullet hole area is expanded and cropped, and texture features are extracted from the cropped suspected bullet hole area. The image of the cropped suspected bullet hole area is represented by f(x,y), [x∈(0,M-1); y∈(0,N-1)], where M is the pixel width of the area, N is the pixel height of the area, and the gray level co-occurrence matrix of the area is represented by P(i,j,d,θ), where i represents the x-axis subscript of the gray level co-occurrence matrix, j represents the y-axis subscript of the gray level co-occurrence matrix, d represents the distance between two pixels, and θ represents the angle between two pixels; extracting texture features includes: extracting texture angular second-order moment, texture entropy, contrast, uniformity, X-axis gradient, and Y-axis gradient;

[0017] The second-order moment of texture angle is expressed as

[0018] Texture entropy is expressed as

[0019] The contrast ratio is expressed as

[0020] Homogeneity using the inverse difference matrix express;

[0021] The X-axis gradient of this region is represented by f5, and the Y-axis gradient is represented by f6;

[0022] The extracted texture features are flattened into one dimension and concatenated to obtain the texture feature matrix f = [f1, f2, f3, f4, f5, f6] of the area.

[0023] Furthermore, in S23, GIOU Loss is used as the loss function of the bullet hole area, and the cross entropy loss function L is used to evaluate the classification of the bullet holes:

[0024]

[0025]

[0026]

[0027] Among them, A and B represent the predicted bullet hole area and the real bullet hole area respectively, C represents the smallest rectangle that can enclose A and B, N represents the number of samples, and y i represents the classification category of the i-th sample, p i It represents the correct prediction probability of the i-th sample. The final total loss function is the sum of the two loss functions.

[0028] Furthermore, in S2, the target image is scaled to a fixed size and three-channel grayscale normalization is performed, and then the image tensor of the target image is converted from RBG space to RGBP space, and the converted image tensor is used as the input of S21.

[0029] Further, in S4, a bullet hole integral map is constructed for the detection results of multiple frames in the video. The bullet hole integral map is obtained by quantizing and accumulating the results of the current frame and the previous frames. During initialization, a grid map of a fixed size is constructed based on the input image. The initial value of each network is 0. The coordinates of the center points of the bullet holes detected in each frame fall within the corresponding grid after quantization, and the grid value is increased by 1. The grid integral map is the cumulative sum of the results of the current frame and the previous l frames. If the current frame number is less than the number of frames in the previous group of frames, the integral map is calculated by accumulating from the first frame to the current frame, and multiplying the cumulative sum by a fixed grayscale coefficient to obtain a gridded grayscale integral map. The integral map formula of the kth frame is expressed as:

[0030]

[0031] where g m (i) represents the number of bullet holes in the (a, b) grid of the mth frame detection result, a represents the subscript of the x-axis in the grid graph, and b represents the subscript of the y-axis in the grid graph. The integral graph method can effectively prevent the situation where the total number of bullet holes in the kth frame is less than the total number of bullet holes in the k-1th frame due to missed detection, thereby avoiding recognition errors caused by missed detections in the detection network.

[0032] Furthermore, in S5, the bullet hole integral map of the current frame is matched with the bullet hole integral map of the previous frame, and the frame difference is performed, and finally the newly added bullet holes of the current frame are obtained;

[0033] The front and back frames of the bullet hole integral image are matched. The commonly used scale-invariant feature transform Sift is used to extract features. Then the registration transposed matrix T(a, b) of the two images is calculated to perform image matching. Since the target surface will shake due to shooting or longitudinal wind during the shooting process, the front and back frame integral images do not correspond to the same position in the original image in the same grid. Therefore, the registration of the integral images of the front and back frames can effectively suppress the false detection caused by image jitter and increase the robustness of the algorithm. In the experiment, it was found that due to the short time interval between the front and back frames, the integral images of the two frames were not distorted. The obtained registration transposed matrix T(a, b) is the grid offset of the k frame based on the k-1 frame on the X-axis and Y-axis. Therefore, the k-1 frame is used as the reference frame. The integral image of the k frame after registration is expressed as

[0034] The frame difference of the bullet hole integral image is obtained by subtracting the k-1 frame integral image from the registered k frame integral image, and the difference image is binarized using a fixed threshold δ:

[0035]

[0036] Among them, δ is the threshold of binarization. The selection of the threshold is related to the cumulative number of frames of the integral image. The grid (a, b) with a grid value of 1 is used as the grid position of the newly added bullet hole, and then the grid position is converted to the original image, and finally the coordinates of the center point of the newly added bullet hole are obtained.

[0037] A bullet hole detection system based on vision, including a deep fusion network of convolutional features and texture features and a bullet hole integral map post-processing module, wherein the fusion network is a lightweight network including a focusing layer, a nested bottleneck layer, a convolutional layer, a feature fusion and a single-scale target regression module;

[0038] The focusing layer performs multi-layer slicing on the image tensor of the target shooting image and then recombines the image;

[0039] The nested bottleneck layer is a residual-like structure, which is used to extract high-level semantic information from the reassembled image tensor;

[0040] The feature fusion and single-scale target regression module includes a single-scale target regression subnetwork, a texture feature extraction unit, and a feature fusion and optimization module;

[0041] The single-scale target regression subnetwork includes a two-dimensional convolution layer and an activation layer. The two-dimensional convolution layer is connected to the nested bottleneck layer, and the convolution feature is output to the activation layer. The activation layer uses the Sigmoid operator, and the calculation formula is: The coordinates output by the activation layer are used to obtain the suspected bullet hole area;

[0042] The texture feature extraction unit extracts texture features from the suspected bullet hole area;

[0043] The feature fusion and optimization module includes a fusion module, a pooling layer, a normalization layer and a fully connected layer. The extracted texture features are combined with the convolution features output by the two-dimensional convolution layer through the fusion module, and the features are optimized through the channel attention mechanism composed of the pooling layer and the normalization layer. The optimized features are passed through the fully connected layer to perform binary classification of bullet holes and non-bullet holes. The deep fusion network is trained based on the binary classification results and the bullet holes annotated in the training set; and reasoning is performed based on the trained deep fusion network to obtain the single-frame bullet hole detection result.

[0044] Furthermore, the focusing layer includes a slicing unit, a combining unit, a two-dimensional convolution layer, a BN layer and an activation layer connected in sequence, the slicing unit performs multi-layer slicing on the image tensor of the target image, the combining unit combines the multi-layer slicing, and the activation layer Where x represents the input image tensor, max(0,x+n) means obtaining the part of the image tensor x+n that is greater than 0, and the part not greater than 0 is equal to 0, min(m,*) means obtaining the part of the image tensor that is less than m, and the part greater than or equal to m is directly equal to m, where m and n are constants obtained based on experience, and m>n.

[0045] Furthermore, the nested bottleneck layer includes a bottleneck module, a convolution layer, a splicing unit, a BN layer and an activation layer. After the image tensor passes through the convolution layer and a set of nested bottleneck modules, it is spliced ​​with the image tensor that only passes through the convolution layer through the splicing unit, and then output through the BN layer, the activation layer and the convolution layer. The structure of the bottleneck module is that the input passes through two one-dimensional convolution layers and is added to the original input, so as to better extract high-level semantic information. Considering that the bullet hole detection scene is relatively simple, only one bottleneck module is set to reduce the amount of calculation.

[0046] The bullet hole integral image post-processing module comprises a bullet hole integral image construction module, an inter-frame integral image registration module and an integral image frame difference module;

[0047] The bullet hole integral map construction module constructs an integral map according to the bullet hole detection results of the current frame and the previous frames;

[0048] The inter-frame integral image registration module registers the integral images of two adjacent frames;

[0049] The integral image frame difference module performs frame difference on the registered integral image to finally obtain the coordinates of the newly added bullet holes in the current frame.

[0050] The advantages and beneficial effects of the present invention are:

[0051] The present invention extracts convolution features by constructing an extremely lightweight convolutional network. The model parameters are only one-fifth of the current mainstream detection model Yolov5-s, and the final mAP is only one percentage point lower than Yolov5-s. The method of optimizing and fusing high-level semantic features such as convolution with feature depths such as texture and grayscale statistics effectively reduces the false detection rate. The coordinate regression subnetwork based on a single-scale anchor is used to regress the coordinates of bullet holes, which can effectively filter out false detections of extremely large or extremely small sizes, accelerate model training, and improve the model convergence speed. The bullet hole grid integral map method can effectively prevent bullet holes with confidence levels near the detection threshold from "flickering"; the bullet hole grid integral map matching method can effectively reduce the interference caused by target surface jitter; the grid integral map frame difference method is used to accurately obtain the coordinates of newly added bullet holes. The detection algorithm takes less than 5ms per frame on NVIDIA2080Ti and less than 40ms per frame on NVIDIA Tx2 embedded platform, and the deployment cost is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The figure is a flow chart of the method of the present invention.

[0053] Figure 2 This is a diagram of the detection network framework in the present invention.

[0054] Figure 3a This is a network framework diagram of the Focus module in the present invention.

[0055] Figure 3b Schematic diagram of tensor slicing of the Focus module in the present invention.

[0056] Figure 4 This is a network framework diagram of the BCSP module in the present invention.

[0057] Figure 5 This is the network flow chart of the Detect module in the present invention.

[0058] Figure 6 This is a framework diagram for extracting texture features in the present invention.

[0059] Figure 7 This is a grid result integration effect diagram of the present invention. DETAILED DESCRIPTION

[0060] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0061] like Figure 1 As shown, a method for detecting bullet holes based on vision includes the following steps:

[0062] S1: Obtain a target shooting bullet hole image dataset and annotate the bullet holes; use a rectangular box to annotate the bullet holes in each target shooting image and mark them as category 0.

[0063] S2: Construct an ultra-lightweight network that deeply integrates convolutional features and texture features, such as Figure 2 As shown, it includes a focusing layer (Focus module), a nested bottleneck layer (BCSP module), a convolutional layer (Conv module), a feature fusion and single-scale target regression module (Detect module), and based on the data set obtained by S1, the current image frame is preprocessed, and then the preprocessing result is sent to the Focus module, Conv convolution module, BCSP module, Conv convolution module, BCSP module, Conv convolution module, and Detect module for training.

[0064] The image frame is preprocessed by scaling the image to a fixed size of 960*960, performing 3-channel grayscale normalization, and then performing color space conversion to convert the image tensor from RBG space to RGBP space.

[0065] The Focus module has the following structure: Figure 3a As shown, the image tensor in the RGBP space is sliced ​​into four layers and then reassembled, then passes through a 32*32 two-dimensional convolution layer, and then input into a Batch Normalization layer, and finally passes through a Hardswish activation layer.

[0066] Four-layer slicing is to downsample a 960*960*3 tensor by interlaced rows, such as Figure 3b As shown, a 4*4*3 tensor is sliced.

[0067] The Hardswish activation layer uses the common activation function of the detection model: Wherein x represents the input tensor, max(0,x+n) represents obtaining the part of the tensor x+n that is greater than 0, and the part that is not greater than 0 is equal to 0, min(m,*) represents obtaining the part of the tensor that is less than m, and the part that is greater than or equal to m is directly equal to m, wherein m and n are constants obtained based on experience, and m>n. In this embodiment, when m=6 and n=3, the effect is optimal.

[0068] BCSP module, its structure is as follows Figure 4 As shown in the figure, it is a residual-like structure BottleneckCSP (CSP bottleneck layer), where Bottleneck belongs to a residual-like structure, where the input passes through two one-dimensional convolutional layers and then adds itself. The BottleneckCSP layer is a nested Bottleneck plus convolution, then concatenated, and finally passes through a convolutional layer, then a BN layer (Batch Normalization), and finally a LeakyRelu activation plus convolutional layer. It should be noted that multiple Bottleneck modules can be nested in the BottleneckCSP layer. The Bottleneck module can better extract high-level semantic information, but considering that the bullet hole detection scenario is relatively simple, in order to reduce the amount of calculation, one Bottleneck is used in this example.

[0069] The feature fusion and single-scale target regression module Detect has the following structure: Figure 5 As shown in the figure, a coordinate regression subnetwork based on a single-scale Anchor is used, and then the coordinates are regressed. The suspected bullet hole area is cropped from the original image and expanded to extract texture features, which are then fused with the convolutional features. The channel attention module is used for feature optimization, and finally the binary classification confidence of the target is obtained through the fully connected layer.

[0070] The coordinate regression sub-network based on a single-scale Anchor consists of a two-dimensional convolution and an activation layer. The activation layer uses the Sigmoid operator calculation formula: The anchor uses a single-scale anchor frame. Taking a 960*960 image as an example, based on prior knowledge, when the camera is within 5 meters of the target surface, the size of the bullet hole in the 960*960 image is approximately between 5*5 and 15*15 pixels. Therefore, the anchor uses three scale anchor frames of [5,5,10,10,15,15]. Through the coordinate regression subnetwork, we can regress the suspected bullet hole target frame, such as Figure 5 The target frame of the bullet hole shown is represented by (X, Y, W, H), where X and Y represent the center coordinates of the bullet hole target frame, and W and H represent the width and height of the bullet hole target frame, respectively.

[0071] Texture features, such as Figure 6 As shown, the original image is expanded and cropped based on the suspected bullet hole target frame, and the expansion ratio is 1:1.2. The texture angle second moment, texture entropy, contrast, uniformity, X-axis gradient, Y-axis gradient and other features are extracted from the cropped area and spliced ​​into a multi-dimensional feature vector.

[0072] Assume that the image of the suspected bullet hole expansion area is represented by f(x,y),[x∈(0,M-1);y∈(0,N-1)], where M is the pixel width of the area and N is the pixel height of the area; the gray level co-occurrence matrix of the area is represented by P(i,j,d,θ), where i represents the x-axis subscript of the gray level co-occurrence matrix, j represents the y-axis subscript of the gray level co-occurrence matrix, d is the distance between two pixels (x1,y1) and (x2,y2), and θ is the angle between the two pixels. Then the second-order texture angle moment is expressed as Texture entropy is expressed as The contrast ratio is expressed as Homogeneity using the inverse difference matrix The X-axis gradient of the region is represented by f5, and the Y-axis gradient is represented by f6. Finally, the texture feature matrix of the region is obtained by flattening the above six matrices into one dimension and concatenating them. The formula is f = [f1, f2, f3, f4, f5, f6].

[0073] Feature fusion and optimization modules such as Figure 5 As shown in the lower part, the texture features are concatenated with the convolutional features extracted by the network, and the channel attention mechanism of AvgPooling+SoftMax is used for feature optimization. Finally, a fully connected layer FC is used for binary classification, where the classification result 0 represents a bullet hole and the classification result 1 represents a non-bullet hole.

[0074] The model training uses GIOU Loss as the loss function of the target box, and the cross entropy loss function L is used to evaluate the classification of bullet holes. The calculation formulas are:

[0075]

[0076]

[0077]

[0078] Among them, A and B represent the predicted target box and the real target box respectively, C is the smallest rectangle that can enclose A and B; N represents the number of samples, y i represents the classification category of the i-th sample, p i It represents the correct prediction probability of the i-th sample. The final total loss function is the sum of the two loss functions.

[0079] S3: Perform inference based on the detection model trained in S2 to obtain the bullet hole detection result of a single frame image;

[0080] The bullet hole detection result is obtained based on the classification detection model inference. The detection result of the k-th frame image is represented as g k (i)=(x i ,y i ); {0≤i<Nk}, where x i With y i Respectively represent the pixel coordinates of the center point of the i-th bullet hole detected in the image, N k Represents the total number of bullet holes detected in the kth frame.

[0081] S4: Based on the S3 detection results of multiple frames in the video, a bullet hole integral map is constructed, such as Figure 7 As shown;

[0082] The bullet hole integral map is obtained by quantizing and accumulating the results of the current frame and the previous frames. When the algorithm is initialized, a fixed-size grid map is constructed based on the input image. Taking the resolution of 960*960 as an example, the image is gridded into 96*96, and the step size of each grid is 10*10. The initial value of each network is 0. The coordinates of the center point of the bullet hole detected in each frame fall within the corresponding grid after quantization, and the grid value is increased by one. The grid integral map is the cumulative sum of the results of the current frame and the previous 19 frames. If the current frame number is less than 19, the integral map is calculated from the first frame to the current frame. The cumulative sum is multiplied by a fixed grayscale coefficient to obtain a gridded grayscale integral map. The integral map formula of the kth frame is expressed as:

[0083]

[0084] where g m (i) represents the number of bullet holes in the (a, b) grid in the mth frame detection result, a represents the subscript of the x-axis in the grid, and b represents the subscript of the y-axis in the grid. The integral graph method can effectively prevent the situation where the total number of bullet holes in the kth frame is less than the total number of bullet holes in the k-1th frame due to missed detection, thereby avoiding recognition errors caused by missed detections in the detection network.

[0085] S5: Based on the bullet hole integral map of the current frame obtained in S4, the bullet hole integral map of the previous frame is registered and frame difference is performed, and finally the newly added bullet holes of the current frame are obtained.

[0086] The commonly used Sift (Scale-invariant feature transform) is used to extract features for the front and back frame matching of the integral image, and then the registration transposed matrix T(a, b) of the two images is calculated to perform image matching. Since the target surface will shake during the shooting process due to factors such as shooting or longitudinal wind, the integral images of the front and back frames do not correspond to the same position in the original image in the same grid. Therefore, registering the integral images of the front and back frames can effectively suppress false detection caused by image jitter and increase the robustness of the algorithm. In the experiment, it was found that due to the short time interval between the front and back frames, the integral images of the two frames were not distorted. The obtained registration matrix T(a, b) is the grid offset of the k frame based on the k-1 frame on the X-axis and Y-axis. Therefore, the integral image of the k frame after registration with the k-1 frame as the reference frame is expressed as

[0087] Integral image frame difference, the k-1 frame integral image is subtracted from the registered k-frame integral image to obtain the difference image. Then, the difference image is binarized using a fixed threshold δ. The selection of the threshold is related to the cumulative number of frames of the integral image. The binarization image formula can be expressed as Among them, δ is the threshold of binarization, the grid (a, b) with a value of 1 is the grid position of the newly added bullet hole, and then the grid position is converted to the original image, and finally the coordinates of the center point of the newly added bullet hole are obtained.

[0088] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting bullet holes based on vision, characterized in that The steps include: S1, obtain a target shooting bullet hole image dataset and mark the bullet holes; S2, constructing a deep fusion network of convolutional features and texture features and training it, including the following steps: S21, performing multi-layer slicing on the image tensor of the target shooting image, and then recombining; S22, inputting the reassembled image tensor into a residual-like structure to extract high-level semantic information; S23, using the single-scale target regression sub-network, obtains the suspected bullet hole area through the coordinates obtained by regression, extracts texture features from the suspected bullet hole area, and fuses them with the convolutional features after input into the single-scale target regression sub-network, performs feature optimization through the channel attention mechanism, and classifies the optimized features into bullet holes and non-bullet holes. The deep fusion network is trained based on the binary classification results and the labeled bullet holes; S3, performs inference based on the trained deep fusion network to obtain the bullet hole detection results of a single frame image; S4, constructing a bullet hole integral image of the current frame image based on the detection results of the multiple frame images; S5, based on the bullet hole integral map of the current frame image and the bullet hole integral map of the previous frame image, register and perform frame difference, and finally obtain the newly added bullet holes of the current frame.

2. A method for detecting bullet holes based on vision according to claim 1, characterized in that In S1, a rectangular frame is used to mark the bullet holes in the target shooting image; the single-scale target regression subnetwork in S23 determines the suspected bullet hole target frame (X, Y, W, H) through the coordinates obtained by regression, where X and Y respectively represent the center coordinates of the suspected bullet hole target frame, and W and H respectively represent the width and height of the suspected bullet hole target frame; and texture features are extracted from the suspected bullet hole target frame; The optimized features are classified into two categories to obtain the binary classification confidence of bullet holes and non-bullet holes. The predicted bullet hole target box and the labeled bullet hole rectangular box obtained by the binary classification results are used to train the deep fusion network.

3. The method for detecting bullet holes based on vision according to claim 1, characterized in that In the S23, the obtained suspected bullet hole area is expanded and cropped, and texture features are extracted from the cropped suspected bullet hole area. The image of the cropped suspected bullet hole area is represented by f(x,y), [x∈(0,M-1); y∈(0,N-1)], where M is the pixel width of the area, N is the pixel height of the area, and the gray level co-occurrence matrix of the area is represented by P(i,j,d,θ), where i represents the x-axis subscript of the gray level co-occurrence matrix, j represents the y-axis subscript of the gray level co-occurrence matrix, d represents the distance between two pixels, and θ represents the angle between two pixels; extracting texture features includes: extracting texture angular second-order moment, texture entropy, contrast, uniformity, X-axis gradient, and Y-axis gradient; The second-order moment of texture angle is expressed as Texture entropy is expressed as The contrast ratio is expressed as Homogeneity using the inverse difference matrix express; The X-axis gradient of this region is represented by f5, and the Y-axis gradient is represented by f6; The extracted texture features are flattened into one dimension and concatenated to obtain the texture feature matrix f = [f1, f2, f3, f4, f5, f6] of the area.

4. The method for detecting bullet holes based on vision according to claim 1, characterized in that In S23, GIOULoss is used as the loss function of the bullet hole area, and the cross entropy loss function L is used to evaluate the classification of bullet holes: Among them, A and B represent the predicted bullet hole area and the real bullet hole area respectively, C represents the smallest rectangle that can enclose A and B, N represents the number of samples, and y i represents the classification category of the i-th sample, p i It represents the correct prediction probability of the i-th sample. The final total loss function is the sum of the two loss functions.

5. The method for detecting bullet holes based on vision according to claim 1, characterized in that In S2, the target image is scaled to a fixed size and three-channel grayscale normalization is performed, and then the image tensor of the target image is converted from RBG space to RGBP space, and the converted image tensor is used as the input of S21.

6. The method for detecting bullet holes based on vision according to claim 1, characterized in that In S4, a bullet hole integral map is constructed for the detection results of multiple frames in the video. The bullet hole integral map is obtained by quantizing and accumulating the results of the current frame and the previous frames. During initialization, a grid map is constructed based on the input image. The initial value of each network is 0. The coordinates of the center points of the bullet holes detected in each frame fall within the corresponding grid after quantization, and the grid value is increased by 1. The grid integral map is the cumulative sum of the results of the current frame and the previous set of frames. If the current frame number is less than the number of frames of the previous 1 frame, the integral map is calculated by accumulating from the first frame to the current frame, and the cumulative sum is multiplied by the grayscale coefficient to obtain a gridded grayscale integral map: where g m (i) represents the number of bullet holes in the (a, b) grid in the mth frame detection result, a represents the subscript of the x-axis in the grid, and b represents the subscript of the y-axis in the grid.

7. The method for detecting bullet holes based on vision according to claim 1, characterized in that In S5, the bullet hole integral map of the current frame is matched with the bullet hole integral map of the previous frame, and the frame difference is performed to finally obtain the newly added bullet holes of the current frame; The front and back frames of the bullet hole integral image are matched. The commonly used scale-invariant feature transform Sift method is used to extract features. Then the registration transposed matrix T(a, b) of the two images is calculated to perform image matching. The obtained registration transposed matrix T(a, b) is the grid offset of the k frame based on the k-1 frame on the X-axis and Y-axis. With the k-1 frame as the reference frame, the k-frame integral image after registration is expressed as The frame difference of the bullet hole integral image is obtained by subtracting the k-1 frame integral image from the registered k frame integral image, and the difference image is binarized using the threshold δ: Among them, δ is the threshold of binarization. The selection of the threshold is related to the cumulative number of frames of the integral image. The grid (a, b) with a grid value of 1 is used as the grid position of the newly added bullet hole, and then the grid position is converted to the original image, and finally the coordinates of the center point of the newly added bullet hole are obtained.

8. A bullet hole detection system based on vision, including a deep fusion network of convolutional features and texture features and a bullet hole integral map post-processing module, characterized in that The fusion network is a lightweight network, including a focusing layer, a nested bottleneck layer, a convolutional layer, a feature fusion and a single-scale target regression module; The focusing layer performs multi-layer slicing on the image tensor of the target shooting image and then recombines the image; The nested bottleneck layer is a residual-like structure, which is used to extract high-level semantic information from the reassembled image tensor; The feature fusion and single-scale target regression module includes a single-scale target regression subnetwork, a texture feature extraction unit, and a feature fusion and optimization module; The single-scale target regression subnetwork includes a two-dimensional convolution layer and an activation layer. The two-dimensional convolution layer is connected to the nested bottleneck layer, and the convolution feature is output to the activation layer. The coordinates output by the activation layer are used to obtain the suspected bullet hole area. The texture feature extraction unit extracts texture features from the suspected bullet hole area; The feature fusion and optimization module includes a fusion module, a pooling layer, a normalization layer and a fully connected layer. The extracted texture features are fused with the convolution features output by the two-dimensional convolution layer through the fusion module. The features are optimized through the channel attention mechanism composed of the pooling layer and the normalization layer. The optimized features are passed through the fully connected layer to perform binary classification of bullet holes and non-bullet holes. The deep fusion network is trained based on the binary classification results and the bullet holes annotated in the training set. Inference is performed based on the trained deep fusion network to obtain bullet hole detection results. The bullet hole integral image post-processing module comprises a bullet hole integral image construction module, an inter-frame integral image registration module and an integral image frame difference module; The bullet hole integral map construction module constructs an integral map according to the bullet hole detection results of the current frame and the previous frames; The inter-frame integral image registration module registers the integral images of two adjacent frames; The integral image frame difference module performs frame difference on the registered integral image to finally obtain the coordinates of the newly added bullet holes in the current frame.

9. A visual-based bullet hole detection system according to claim 8, characterized in that The focusing layer includes a slicing unit, a combining unit, a two-dimensional convolution layer, a BN layer and an activation layer connected in sequence. The slicing unit performs multi-layer slicing on the image tensor of the target image, the combining unit combines the multi-layer slicing, and the activation layer Where x represents the input image tensor, max(0,x+n) means obtaining the part of the image tensor x+n that is greater than 0, and the part not greater than 0 is equal to 0, min(m,*) means obtaining the part of the image tensor that is less than m, and the part greater than or equal to m is directly equal to m, where m and n are constants obtained based on experience, and m>n.

10. A visual-based bullet hole detection system according to claim 8, characterized in that The nested bottleneck layer includes a bottleneck module, a convolution layer, a splicing unit, a BN layer and an activation layer. After the image tensor passes through the convolution layer and a set of nested bottleneck modules, it is spliced ​​with the image tensor that only passes through the convolution layer through the splicing unit, and then output through the BN layer, the activation layer and the convolution layer. The structure of the bottleneck module is that the input passes through two one-dimensional convolution layers and is added to the original input.