Target detection system, training method, target detection method and electronic equipment

Abstract

The invention discloses a target detection system, a training method, a target detection method and electronic equipment, and belongs to the technical field of computer vision. On the basis that a target detection module is adopted to perform target coarse detection on a to-be-detected image, a region supplement module is further adopted to determine a candidate target region based on a centrality graph of the to-be-detected image, the candidate target region is cut out from the to-be-detected image, and after scale scaling is performed through a corresponding scaling coefficient, the candidate target region is obtained. Splicing into a spliced image, and carrying out target detection again to obtain a target fine detection result so as to supplement the deficiency of the target coarse detection result; according to the method, the distribution condition of the center position of the target in the to-be-detected image is dynamically learned through the centrality map prediction module, and the possible region of the target in the to-be-detected image is further mined through the region supplement module based on the centrality map information, so that the region can be covered more completely; and the uncertainty caused by insufficient target rough detection precision is eliminated, and the accuracy of target detection is relatively high.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and more specifically, relates to an object detection system, a training method, an object detection method, and an electronic device. Background Art

[0002] Images are an important source for humans to understand the world. With the rapid development of computer vision technology, object detection has become increasingly important in the fields of image processing and artificial intelligence and has been widely applied to various scenarios, such as autonomous driving, drones, and other scenarios. Therefore, it is of great significance to study an object detection method.

[0003] Existing object detection methods often rely on object detection models constructed based on deep learning technology. The quality and quantity of data are important factors determining the generalization performance of object detection models. When using object detection models to solve image object detection tasks, dense objects and large-scale variations are common problems. On the one hand, dense objects are difficult to distinguish in terms of features and require special processing; on the other hand, large-scale variations require effective separate processing of multi-scale objects existing in images.

[0004] To solve the above problems, in existing object detection methods, the commonly used image data processing operations mainly include color transformation, clustering, attention mechanisms, etc. These operations simulate the transformations in aspects such as perspective, light, and scale in the natural environment to a certain extent and also make some targeted processing for dense small object regions. Therefore, the generalization performance of object detection models can be improved to a certain extent. However, conventional operations such as color transformation are usually used as one of the steps for preprocessing input image data, independent of the training process of object detection models, directly processing all data uniformly, which is a "blind" operation with poor adaptability to different images, resulting in low accuracy of object detection models. For other operations such as clustering and attention mechanisms, the improvement of detection effects is limited, and the detection accuracy for dense objects cannot be well improved. Summary of the Invention

[0005] Aiming at the above defects or improvement requirements of the prior art, the present invention provides an object detection system, a training method, an object detection method, and an electronic device to solve the technical problem of low accuracy of object detection in the prior art.

[0006] To achieve the above object, in a first aspect, the present invention provides an object detection system, including:

[0007] An object detection module, configured to perform object detection on a to-be-detected image: extract multi-scale features of the to-be-detected image, and based on the multi-scale features, extract preliminary detection frames of different scales, and merge the detection frames of different scales to obtain a rough object detection result;

[0008] The centrality map prediction module is used to predict the centrality map of the image to be detected based on multi-scale features;

[0009] The region supplement module is used to binarize the centrality map to obtain a corresponding binary map; perform connected component analysis on the binary map to obtain several sub-region reference positions; crop the image to be detected based on the obtained sub-region reference positions to obtain several cropped images; scale each cropped image with a corresponding scaling factor and then splice them to obtain a spliced image;

[0010] The object detection module is also used to perform object detection on the spliced image to obtain a fine object detection result;

[0011] The merging module is used to merge the rough object detection result and the fine object detection result to obtain the final object detection result.

[0012] Further preferably, the region supplement module performs connected component analysis on the binary map in the following manner:

[0013] Perform grid division on the binary map to obtain a grid of M net rows and N net columns, and then construct a grid map of size M net ×N net ; each pixel value in the grid map is the number of pixels with a value of 1 in the corresponding grid in the binary map; both M net and N net are positive integers;

[0014] Set the pixel values in the grid map that are less than a preset threshold to 0, so as to obtain several connected components as several sub-region reference positions.

[0015] In a second aspect, the present invention provides a training method for the above object detection system; wherein, the scaling factors adopted in the region supplement module of the object detection system are derived from a set of scaling factors; each scaling factor in the set of scaling factors corresponds to a scale range, and the scale ranges corresponding to the respective scaling factors are all different;

[0016] The above training method includes:

[0017] S1. Initialize the set of scaling factors, the current iteration number t, the object detection loss of each image sample in the first training set at the iteration number t The joint loss at the iteration number t where m is the number of image samples in the first training set;

[0018] S2. Let \(t = t + 1\); input each image sample in the first training set into the object detection system to obtain the corresponding target fine detection results; calculate the difference loss between the target fine detection results and the corresponding true detection results of each image sample in the first training set as the object detection loss of each image sample in the first training set at the current iteration \(t\).

[0019] S3. With the goal of minimizing the sum of the object detection losses of each image sample in the first training set, adjust the parameters of the object detection module in the object detection system;

[0020] Calculate the joint loss at the current iteration \(t\); And based on Adjust the scaling coefficients in the scaling coefficient set; where the coefficients \(\alpha,\beta\) satisfy \(\alpha+\beta = 1,0\leqslant\alpha,\beta\leqslant1\); \(c_j\) j Is the \(j\)-th scaling coefficient in the scaling coefficient set; \(N\) is the number of scaling coefficients in the scaling coefficient set;

[0021] S4. Determine whether the current iteration \(t\) is less than the preset number of iterations. If so, go to S2; otherwise, complete the training of the current round.

[0022] Further preferably, the above based on Adjusting the scaling coefficients in the scaling coefficient set includes:

[0023] When The scaling coefficients in the scaling coefficient set remain unchanged;

[0024] When The scaling coefficients in the scaling coefficient set are adjusted.

[0025] Further preferably, the above training method further includes the following operations performed before S1: pre-train the above object detection module, including:

[0026] Input each image sample in the second training set into the object detection module to obtain the corresponding target rough detection results; calculate the difference loss between the target rough detection results and the corresponding true detection results of each image sample in the second training set as the object detection loss of each image sample in the second training set;

[0027] With the goal of minimizing the sum of the object detection losses of each image sample in the second training set, adjust the parameters of the object detection module, thereby realizing the pre-training of the object detection module.

[0028] Further preferably, the above training method further includes the following operations performed before S1: pre-train the centrality map prediction module in the object detection system, including:

[0029] Input each image sample in the third training set into the centrality map prediction module to obtain the corresponding centrality map prediction result; calculate the difference loss between the centrality map prediction result of each image sample in the third training set and the corresponding ground truth result as the prediction loss of each image sample in the third training set.

[0030] Aim to minimize the sum of the prediction losses of each image sample in the third training set, and adjust the parameters of the centrality map prediction module, thereby realizing the pre-training of the centrality map prediction module.

[0031] In a third aspect, the present invention provides an object detection method, including: inputting the image to be detected into the object detection system provided in the first aspect of the present invention to obtain the final object detection result.

[0032] In a fourth aspect, the present invention provides an electronic device, including: a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it executes the method provided in the second or third aspect of the present invention.

[0033] In a fifth aspect, the present invention further provides a computer-readable storage medium, the computer-readable storage medium includes a stored computer program, wherein when the computer program is run by a processor, it controls the device where the storage medium is located to execute the method provided in the second or third aspect of the present invention.

[0034] In a sixth aspect, the invention further provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, they implement the method provided in the second or third aspect of the present invention.

[0035] Generally speaking, through the above technical solutions conceived by the present invention, the following beneficial effects can be achieved:

[0036] 1. The present invention provides an object detection system. On the basis of performing rough object detection on the image to be detected by using the object detection module, the region supplement module further determines the candidate target regions based on the centrality map of the image to be detected, crops them out from the image to be detected, scales them through the corresponding scaling factor, splices them into a spliced image and performs object detection again to obtain the object fine detection result to supplement the deficiency of the object rough detection result; the present invention dynamically learns the distribution of the target center positions in the image to be detected through the centrality map prediction module, and further mines the regions where the target may exist in the image to be detected based on the centrality map information through the region supplement module, which can cover the regions more completely and eliminate the uncertainty caused by the insufficient accuracy of the object rough detection, and the accuracy of object detection is relatively high.

[0037] 2. Further, in a target detection system provided by the present invention, the region supplementation module performs grid division on the binary image, counts the number of pixel points with a value of 1 within the grid, and filters out regions with a relatively small number of pixel points with a value of 1, thereby obtaining several connected regions. Through the aggregation method, the detection efficiency can be further improved, and the reference positions of each sub-region can be accurately obtained.

[0038] 3. The present invention provides a training method for a target detection system, which uses reward and punishment measures for joint optimization. During the training process of the detection network, the scaling coefficients adopted in the region supplementation module of the target detection system are simultaneously optimized. The non-differentiable scaling coefficient parameter optimization operation is combined with the training of the target detection module, enabling the region supplementation module to track the dynamic training process of the target detection module, adaptively adjust the distribution of the scaling coefficients according to the specific characteristics of the image data and the target, and perform region supplementation adaptively, further enhancing the generalization ability suitable for the target detection module and greatly improving the detection accuracy of the target detection system.

[0039] 4. Further, in the training method of the target detection system provided by the present invention, when the joint loss in the current iteration is greater than or equal to the joint loss in the previous iteration, each scaling coefficient in the scaling coefficient set remains unchanged; when the joint loss in the current iteration is less than the joint loss in the previous iteration, each scaling coefficient in the scaling coefficient set is adjusted; the scale factor can be jointly optimized from two aspects of the detection task itself and the scale factor value in the case of no labels, with strong adaptability to detection tasks in different situations and good optimization effects, further improving the detection accuracy of the target detection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the working process of the target detection system provided by the embodiment of the present invention;

[0041] Figure 2 It is a schematic diagram of the working process of the region supplementation module provided by the embodiment of the present invention;

[0042] Figure 3 It is a schematic diagram of the structure of the target detection module provided by the embodiment of the present invention;

[0043] Figure 4 It is a flowchart of the joint optimization training provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0045] To achieve the above objectives, in a first aspect, the present invention provides a target detection system, including:

[0046] A target detection module for performing target detection on an image to be detected: extracting multi-scale features of the image to be detected, and based on the multi-scale features, extracting preliminary detection frames of different scales, and merging the detection frames of different scales to obtain a rough target detection result;

[0047] A centrality map prediction module for predicting the centrality map of the image to be detected based on the multi-scale features;

[0048] A region supplementation module for performing binarization processing on the centrality map to obtain a corresponding binary map; performing connected component analysis on the binary map to obtain several reference positions of sub-regions; cropping the image to be detected based on the obtained reference positions of sub-regions to obtain several cropped images; scaling each cropped image with a corresponding scaling factor and then stitching them to obtain a stitched image;

[0049] The target detection module is further used for performing target detection on the stitched image to obtain a fine target detection result;

[0050] A merging module for merging the rough target detection result and the fine target detection result to obtain a final target detection result.

[0051] It should be noted that there are various target detection modules that can be used, such as RCNN, YOLO, DETR, FCOS, RetinaNet, etc., which are not limited here. There are various centrality map prediction modules that can be used, such as artificial neural networks, deep learning networks, Transformer modules, etc., which are not limited here.

[0052] It should be noted that there are various methods for performing connected component analysis, such as depth-first search, breadth-first search, A* algorithm, dynamic programming, etc., which are not limited here. Preferably, in an alternative embodiment, the region supplementation module performs connected component analysis on the binary map in the following manner:

[0053] Performing grid division on the binary map to obtain a grid of M net rows and N net columns, and then constructing an M net ×N netGrid graph of size; each pixel value in the grid graph is the number of pixels with a value of 1 in the corresponding grid in the binary graph; M net and N net are both positive integers; in an alternative embodiment, M net and N net both have a value range of [10, 40]; preferably, M net and N net take values of 20 and 20 respectively;

[0054] Set the pixel values less than the preset threshold in the grid graph to 0, so as to obtain several connected regions as the reference positions of several sub-regions.

[0055] In an alternative embodiment, the method for stitching each scaled and cropped image includes: stitching each scaled and cropped image onto an image of a preset size, and the stitching process only needs to meet the width constraint and the height constraint.

[0056] It should be noted that the target detection result mainly includes the detection box where the target is located; there are various methods for the merging module to merge the target rough detection result and the target fine detection result, and methods such as non-maximum suppression, one-to-many sample assignment, and adaptive merging can be used for merging.

[0057] It should be noted that the training method of the above target detection system can adopt the conventional training method, such as the end-to-end training method. Based on this, preferably, in the second aspect, the present invention provides the training method of the above target detection system; wherein, the scaling coefficient adopted in the region supplement module of the target detection system comes from the scaling coefficient set; each scaling coefficient in the scaling coefficient set corresponds to a scale range, and the scale ranges corresponding to each scaling coefficient are all different;

[0058] The above training method includes:

[0059] S1. Initialize the scaling coefficient set, the current iteration number t, and the target detection loss of each image sample in the first training set at the iteration number t The joint loss at the iteration number t where m is the number of image samples in the first training set;

[0060] S2. Let t = t + 1; input each image sample in the first training set into the target detection system to obtain the corresponding target fine detection result; calculate the difference loss between the target fine detection result of each image sample in the first training set and the corresponding true detection result as the target detection loss of each image sample in the first training set at the current iteration number t where the image samples in the first training set all carry the corresponding true target box labels;

[0061] S3. Adjust the parameters of the object detection module in the object detection system with the goal of minimizing the sum of the object detection losses of each image sample in the first training set;

[0062] Calculate the combined loss at the current iteration number t and based on adjust the scaling factors in the scaling factor set; where the coefficients α, β satisfy α + β = 1, 0 ≤ α, β ≤ 1; c j is the j-th scaling factor in the scaling factor set; N is the number of scaling factors in the scaling factor set; preferably, a typical value is 4, and the initial values of the parameters of the scaling factor set are {1, 2, 4, 6}.

[0063] S4. Determine whether the current iteration number t is less than the preset iteration number. If so, go to S2; otherwise, complete the training of the current round.

[0064] The related technical solutions are the same as the object detection system provided in the first aspect of the present invention, and will not be elaborated here.

[0065] In an alternative embodiment, the above-mentioned adjustment of the scaling factors in the scaling factor set based on includes:

[0066] When keep each scaling factor in the scaling factor set unchanged;

[0067] When adjust the scaling factors in the scaling factor set; preferably, a typical adjustment method is random perturbation; preferably, a typical perturbation range is a random value between [-0.5, 0.5].

[0068] In an alternative embodiment, the above-mentioned training method further includes the following operations performed before S1: pre-train the above-mentioned object detection module, including:

[0069] Input each image sample in the second training set into the object detection module to obtain the corresponding rough object detection result; calculate the difference loss between the rough object detection result of each image sample in the second training set and the corresponding true detection result as the object detection loss of each image sample in the second training set; where the image samples in the second training set all carry the corresponding true object box labels;

[0070] Adjust the parameters of the object detection module with the goal of minimizing the sum of the object detection losses of each image sample in the second training set, so as to realize the pre-training of the object detection module.

[0071] In an alternative embodiment, the above training method further includes the following operations performed before S1: pre-training the centrality map prediction module in the object detection system, including:

[0072] Inputting each image sample in the third training set into the centrality map prediction module to obtain the corresponding centrality map prediction result; calculating the difference loss between the centrality map prediction result of each image sample in the third training set and the corresponding ground truth result as the prediction loss of each image sample in the third training set; wherein, each image sample in the third training set carries the corresponding ground truth centrality map.

[0073] Taking the minimization of the sum of the prediction losses of each image sample in the third training set as the objective, adjusting the parameters of the centrality map prediction module, thereby realizing the pre-training of the centrality map prediction module.

[0074] It should be noted that the above first training set, second training set, and third training set may be the same or different, and are not limited here.

[0075] In summary, different from the conventional practice of general object detection tasks, the present invention combines object detection with an adaptive region supplementation algorithm and introduces an adaptive strategy in the processing of both, enabling image data and features to be dynamically adjusted following the change process of the detection stage, adaptively performing data augmentation and region supplementation according to the characteristics of the specific dataset and application tasks, generating more effective training samples for optimizing the object detection module, improving the training efficiency of the object detection module, and effectively enhancing the generalization ability of the object detection module; the present invention takes the object detection module as the main module, uses the region supplementation module for region supplementation, and realizes joint optimization through the synergistic effect of the two, ultimately achieving better performance of the object detection system for executing computer vision tasks.

[0076] To further illustrate the object detection system and its training method provided by the present invention, a specific embodiment is described in detail below:

[0077] This embodiment provides an object detection system that does not require additional data samples and does not change the structure of the object detection model under a specific detection task. However, through the joint optimization of the region supplementation module and the object detection module, "adaptive" data processing is performed on the original dataset, which can generate more perfect training images, improve the detection coverage, thereby more effectively improving the detection accuracy in the training process of the object detection system and enhancing the generalization performance of the object detection system. In this embodiment, a convolutional neural network is used as the object detection module.

[0078] The region supplementation module provided by the present invention is an adaptive module, and the overall object detection flowchart is as Figure 1As shown, different from conventional image - processing - based operations, the region supplementation module provided by the present invention is a feature - processing and region - supplementation method, including: a scale - selection process and a region - supplementation process. The specific strategy of the region supplementation module is as Figure 2 shown; the input of the region supplementation module is the feature maps of different scales extracted by the target detection module (a convolutional neural network for a specific target detection task in this embodiment). After aggregating and strengthening the features of the feature maps of different scales, they are then fused. Its advantage is that it can improve the efficiency of joint optimization by using the spatial constraints contained in the feature maps of different scales, and at the same time strengthen the features of dense small objects; the output of the region supplementation module is a new image formed by assembling the supplemented regression results under the current scale parameters. Among them, for the input target rough detection result, the process of adaptive processing is as follows: after obtaining the centrality map of the image to be detected and the target rough detection result, the centrality map is binarized by double - threshold processing to obtain a binary map; then region grid selection and merging are performed to obtain a supplemented region and supplement the regression result; finally, the supplemented regression result is scaled by the scaling factor corresponding to the current scale and cropped and assembled from the input image to be detected for output. Specifically, the process of binarizing the centrality map by double - threshold processing is as follows:

[0079] For the centrality map C of each image, first, a binary map C t is obtained by double - threshold processing. The specific method is:

[0080]

[0081] where, represents the element value of the i - th row and j - th column in the binary map C t ; T h is the preset high threshold, and T l is the preset low threshold.

[0082] For the obtained binary map C t , first, grid processing is performed, and then the corresponding regions where the number of pixels with a value of 1 in the grid is greater than the screening threshold T are selected. After neighboring region merging, several sub - region reference positions are obtained; the image to be detected is cropped based on the obtained sub - region reference positions to obtain several cropped images; after each cropped image is scaled by the corresponding scaling coefficient, they are stitched to obtain a stitched image; the stitched image is input into the target detection module to obtain the target fine detection result as a supplement to the target rough detection result.

[0083] The fitting target detection module can sample according to the characteristics of objects in the image data and implement classical data augmentation operations such as scaling, object region clustering, super-resolution, etc. Compared with the data augmentation method using the input image data, in this embodiment, the objects are adjusted adaptively according to the actual detection situation. On the one hand, it can accelerate the convergence speed during the training of the target detection module, and on the other hand, it takes into account the detection conditions required for different scale targets, which can enhance the adaptability of the target detection module to multi-scale objects.

[0084] Before joint optimization, the target detection module and the centrality map prediction module are pre-trained respectively to obtain the pre-trained models of the target detection module and the centrality map prediction module suitable for specific computer vision tasks; in the joint optimization stage, a set of scaling coefficients {c1, c2, …, c N} are initialized, and each scaling coefficient corresponds to a scale range. The centrality map of the image samples in the training set is binarized to obtain the corresponding binary map; and the connected component analysis is performed on the binary map to obtain the reference positions of several sub-regions; the image to be detected is cropped based on the obtained reference positions of the sub-regions to obtain several cropped images (i.e., the object targets of different scales in the image to be detected); the cropped images are scaled according to the scaling coefficients corresponding to their scales and then stitched together, and the stitched image is input into the target detection module for detection again to obtain the fine target detection result. Record the detection losses of the fine target detection results and the corresponding true detection results of each image sample in the training set as l1', l2', …, l m ', and optimize the joint loss parameter with the following formula:

[0085]

[0086] where m is the number of image samples in the training set; is the detection loss in the previous iteration during the joint optimization process, and c j is the j-th scale parameter. Both α and β are hyperparameters (α + β = 1, 0 ≤ α, β ≤ 1), and the specific values can be selected according to different tasks (a typical value is α = β = 0.5). In this embodiment, the simulated annealing algorithm and the joint loss S total are used as the standard for joint optimization scale parameter selection.

[0087] The joint optimization of the region supplementation module and the target detection module is based on a coarse-to-fine detection structure. Among them, the feature adjustment and image data processing are the region supplementation module, and the target detection module is a convolutional neural network suitable for the target detection task. The region supplementation module enhances the learning ability from the rough target detection results, while the target detection module also continuously adjusts according to the output of the region supplementation module, and tries to generate better parameter selections for detection. Through the "collaboration" between the region supplementation module and the target detection module, the common optimization of the region supplementation module and the target detection module is achieved, and finally a better performance of the target detection module is realized.

[0088] Specifically, a reward and punishment measure is adopted for the joint optimization. During the training process of the target detection module, the scale parameters output by the data augmentation module are optimized simultaneously, and the non-differentiable parameter optimization operation is combined with the training of the target detection module, so that the region supplementation module can track the training process of the dynamic target detection module, adaptively adjust the parameter distribution according to the characteristics of the specific dataset and the target, and perform region supplementation adaptively, further enhancing the generalization ability suitable for the target detection module.

[0089] The reward and punishment measure is specifically as follows: The rough target detection results and the centrality map are processed by the region supplementation module. The region supplementation module gives corresponding scale parameters for the centrality map and supplements the region. The image is cropped and assembled from the input image according to the supplemented region and the rough target detection results for fine detection. If the combined loss of the current detection loss and the change in the scaling factor increases, it indicates that the current scale parameter adjustment has no obvious improvement on the detection, and the group of scaling factors is rejected and the scaling factor remains unchanged. If the combined loss decreases, it indicates that the current scale adjustment is beneficial to the detection, and the scaling factor is updated to the current scaling factor with a high probability. The detection loss and the scale adjustment loss act jointly to optimize the region supplementation module and the target detection module.

[0090] During the joint optimization process, the loss function corresponding to the target detection module is related to the specific computer vision task, and the loss used by the region supplementation module is the combined loss, which is specifically as follows:

[0091]

[0092] Among them, α and β are proportional parameters used to adjust the ratio of the two losses (α + β = 1, 0 ≤ α, β ≤ 1). S total is the combined loss.

[0093] In this embodiment, taking the target detection task of UAV aerial images as an example, the above method is described in detail, which specifically includes:

[0094] Pre-train the target detection module to obtain a pre-trained target detection module. Among them, the target detection module selects the classic YOLO structure, which includes a backbone feature extraction network, a neck feature fusion network, and a detection head. Its specific structure is as Figure 3 shown;

[0095] Pre-train the centrality map prediction module to obtain a pre-trained centrality map prediction module;

[0096] Based on the rough target detection results output by the target detection module and the centrality map output by the centrality map prediction module, jointly optimize the target detection module and the region supplementation module;

[0097] Use the obtained target detection system to detect the targets in the UAV aerial images.

[0098] It should be noted that the training sets used in the pre-training stage of the target detection module, the pre-training stage of the centrality map prediction module, and the joint optimization stage can be the same or different. In this embodiment, it is taken as an example that the three training sets are the same.

[0099] The training set is obtained through the following methods:

[0100] Data preparation: Collect multiple image samples to form a training set; obtain the bounding box coordinates, categories of each object target in each image sample in the training set, and calculate the center point coordinates. Calculate the centrality heat map of each image sample according to the center point and bounding box information. Specifically, for each object target in the image sample, construct a corresponding two-dimensional Gaussian distribution with the center point coordinates of the object target as the mean and the width and height of the bounding box as the variance; superimpose the two-dimensional Gaussian distributions of each object target to obtain the centrality heat map of the image sample. Among them, when there is an overlap in the superimposing method, take the maximum value of the overlapping part.

[0101] Data processing: Crop the image samples in the data set and reshape the data size to make the image samples of the same size;

[0102] Data augmentation: Perform data augmentation operations such as geometric transformation on the obtained image samples to obtain the final training set.

[0103] As Figure 4 shown, the above joint optimization process includes:

[0104] Input the image samples in the training set into the target detection system. Use the rough target detection results obtained by passing through the target detection module for the first time and the centrality map obtained by passing through the centrality map prediction module as the input of the region supplementation module. Perform region supplementation in the region supplementation module according to the centrality map to obtain a spliced image; input the spliced image into the target detection module for the second target detection to obtain the fine target detection results.

[0105] If the combined loss of the current detection loss and the change in the scaling factor increases, it indicates that the current scaling factor does not significantly improve the detection. Reject this set of scaling factors and continue to optimize during the training process. At the same time, the object detection module is supervised and trained by its own loss function according to the accuracy of the current detection. If the combined loss decreases, it indicates that the adjustment of the current scaling factor is beneficial to the detection, and there is no situation of excessive adjustment of the scaling factor. At this time, update the scaling factor to the currently selected scaling factor with a high probability. The detection loss and the scale adjustment loss act jointly to optimize the region supplement module and the object detection module.

[0106] After the above joint optimization is completed, an object detection system is obtained. Through the forward prediction of the object detection system, the detection of object targets in the UAV aerial image is realized, and the detection results and evaluation indicators are output.

[0107] In summary, the present invention can solve the technical problems existing in the prior art that the conventional image data processing is "blind", cannot make full use of the intermediate results of detection, nor can it adaptively process the characteristics of specific data sets and target features, resulting in low training efficiency and limited effects. Specifically, it is reflected in the following aspects:

[0108] Different from the conventional practice of taking data processing as an independent data preprocessing step, the present invention introduces an adaptive strategy, combines the two independent processes of data processing in the region supplement module and the training of the object detection module, enables the data processing process to change dynamically following the training process of the object detection module, and adaptively generates more effective training samples for optimizing the object detection module according to the characteristics of specific targets and the intermediate results of the object detection process, improves the training efficiency of the object detection module, and at the same time more effectively enhances the generalization performance of the object detection module.

[0109] The method of the present invention uses the region supplement module as the intermediate layer for two-stage detection of coarse and fine, for feature adjustment and image data processing; improves the learning ability from the rough object detection results generated by the object detection module, and the object detection module also continuously adjusts according to the output of the region supplement module to generate better parameter selections for detection as much as possible. Through the "collaboration" between the region supplement module and the object detection module, the common optimization of the scaling coefficients in the object detection module and the region supplement module is realized, and finally a better performance of the image detection task is achieved.

[0110] The method of the present invention uses reward and punishment measures for joint optimization. Different from the methods of network optimization in conventional object detection networks, the present invention uses reward and punishment measures for joint optimization, simultaneously optimizing the scaling factor during the training process of the object detection system, combining the non-differentiable parameter optimization operation with the training of the object detection module, enabling the region supplement module to track the training process of the dynamic object detection module, adaptively adjusting the parameter distribution according to the characteristics of the specific dataset and the object characteristics, and adaptively performing region supplementation, further enhancing the generalization ability of the object detection module suitable for image detection tasks.

[0111] The region supplement module proposed by the present invention generates the regions to be supplemented and a set of scaling factors, rather than directly using the rough object detection results of the rough object detector. The coverage of the regions is more complete, and the uncertainty caused by the insufficient accuracy of the rough detector is eliminated, which can further increase the image data and more effectively prevent the missed detection phenomenon that occurs during the training of the object detection module suitable for specific image detection.

[0112] In a third aspect, the present invention provides an object detection method, including: inputting the image to be detected into the object detection system provided in the first aspect of the present invention to obtain the final object detection result.

[0113] The related technical solutions are the same as those of the object detection system provided in the first aspect of the present invention, and will not be elaborated here.

[0114] In a fourth aspect, the present invention provides an electronic device, including: a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it executes the method provided in the second aspect or the third aspect of the present invention.

[0115] The related technical solutions are the same as those of the methods provided in the second aspect and the third aspect of the present invention, and will not be elaborated here.

[0116] In a fifth aspect, the present invention further provides a computer-readable storage medium, the computer-readable storage medium includes a stored computer program, wherein when the computer program is run by a processor, it controls the device where the storage medium is located to execute the method provided in the second aspect or the third aspect of the present invention.

[0117] The related technical solutions are the same as those of the methods provided in the second aspect and the third aspect of the present invention, and will not be elaborated here.

[0118] In a sixth aspect, the invention further provides a computer program product, including computer program / instructions, and when the computer program / instructions are executed by a processor, they implement the method provided in the second aspect or the third aspect of the present invention.

[0119] The related technical solutions are the same as the methods provided in the second and third aspects of the present invention, and will not be elaborated here.

[0120] Those skilled in the art can easily understand that the above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A target detection system, characterized in that, Including: A target detection module for performing target detection on an image to be detected: extracting multi-scale features of the image to be detected, extracting preliminary detection frames of different scales based on the multi-scale features, merging the detection frames of different scales, and obtaining a rough target detection result; A centrality map prediction module for predicting a centrality map of the image to be detected based on the multi-scale features; A region supplementation module for performing binarization processing on the centrality map to obtain a corresponding binary map; performing connected component analysis on the binary map to obtain several sub-region reference positions; cropping the image to be detected based on the obtained sub-region reference positions to obtain several cropped images; scaling each cropped image with a corresponding scaling factor and then splicing them to obtain a spliced image; The target detection module is further configured to perform target detection on the spliced image to obtain a fine target detection result; A merging module for merging the rough target detection result and the fine target detection result to obtain a final target detection result.

2. The object detection system according to claim 1, characterized in that, The region supplementation module performs connected component analysis on the binary map in the following manner: Perform grid division on the binary image to obtain M net rows and N net columns of grids, and then construct a grid image of size M net ×N net ; Each pixel value in the grid image is the number of pixels with a value of 1 in the corresponding grid in the binary image; M net and N net are both positive integers; Setting the pixel values less than a preset threshold in the grid map to 0, thereby obtaining several connected components as several sub-region reference positions.

3. The training method of the object detection system according to claim 1 or 2, characterized in that, The scaling factors adopted in the region supplementation module of the target detection system are from a set of scaling factors; each scaling factor in the set of scaling factors corresponds to a scale range, and the scale ranges corresponding to the scaling factors are all different; The training method includes: S1. Initialize the scaling factor set, the current iteration number t, and the object detection losses of each image sample in the first training set at the iteration number t. The combined loss at the iteration number t m is the number of image samples in the first training set; S2. Let \(t = t + 1\); input each image sample in the first training set into the target detection system to obtain the corresponding target fine detection result; calculate the difference loss between the target fine detection result and the corresponding true detection result of each image sample in the first training set as the target detection loss of each image sample at the current iteration number \(t\). S3. Taking the minimization of the sum of the target detection losses of each image sample in the first training set as the goal, adjusting the parameters of the target detection module in the target detection system; Calculate the combined loss at the current iteration number t and based on adjust the scaling factors in the set of scaling factors; wherein, the coefficients α, β satisfy α + β = 1, 0 ≤ α, β ≤ 1; c j is the j-th scaling factor in the set of scaling factors; N is the number of scaling factors in the set of scaling factors; S4. Judging whether the current iteration number t is less than a preset iteration number. If so, go to S2; otherwise, complete the training of the current round.

4. The training method according to claim 3, wherein The said based on Adjusting the scaling factors in the said set of scaling factors includes: When occurs, each scaling factor in the set of scaling factors remains unchanged; When each scaling factor in the set of scaling factors is adjusted.

5. The training method according to claim 3 or 4, characterized in that It further includes: The following operations performed before S1: pre-training the target detection module, including: Inputting each image sample in the second training set into the target detection module to obtain a corresponding rough target detection result; calculating the difference loss between the rough target detection result of each image sample in the second training set and the corresponding true detection result as the target detection loss of each image sample in the second training set; Taking the minimization of the sum of the target detection losses of each image sample in the second training set as the goal, adjusting the parameters of the target detection module, thereby realizing the pre-training of the target detection module.

6. The training method according to claim 3 or 4, characterized in that It further includes: The following operations performed before S1: pre-training the centrality map prediction module in the target detection system, including: Inputting each image sample in the third training set into the centrality map prediction module to obtain a corresponding centrality map prediction result; Calculating the difference loss between the centrality map prediction result of each image sample in the third training set and the corresponding true result as the prediction loss of each image sample in the third training set; Taking the minimization of the sum of the prediction losses of each image sample in the third training set as the goal, adjusting the parameters of the centrality map prediction module, thereby realizing the pre-training of the centrality map prediction module.

7. A target detection method, characterized in that, Including: Input the image to be detected into the target detection system described in claim 1 or 2 to obtain the final target detection result.

8. An electronic device, characterized in that, Comprising: A memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it executes the method described in any one of claims 3-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the computer program is run by a processor, it controls the device where the storage medium is located to execute the method described in any one of claims 3-7.

10. A computer program product, characterized in that, Comprising a computer program / instructions, when the computer program / instructions are executed by a processor, the method described in any one of claims 3-7 is implemented.

Images