Target detection method and related equipment

Abstract

The embodiment of the present invention discloses a target detection method and related equipment. When performing target detection, initial prediction information of an image to be detected can be obtained, and the initial prediction information includes not only an initial prediction border and a corresponding initial prediction category confidence, but also an initial prediction offset reflecting the offset degree between the initial prediction border and the target border; then the initial prediction information is screened according to the initial prediction category confidence and the initial prediction offset, and part of the initial prediction information is filtered to obtain final prediction information of the target, and the prediction information includes a prediction border and a corresponding prediction category confidence; because the initial prediction information is screened according to both the initial prediction category confidence and the initial prediction offset, not only can the amount of prediction information of the target generated during target detection be reduced, but also the filtering accuracy of invalid prediction information can be effectively improved, and valid prediction information about the target is retained, which is helpful to improve the detection accuracy of target detection.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, specifically to the field of target detection technology, and in particular to a target detection method, a target detection device, a terminal device and a computer storage medium. Background Art

[0002] Object detection technology integrates cutting-edge technologies in many fields such as image processing, pattern recognition, artificial intelligence, and automatic control. It has been widely used in many fields such as face recognition, intelligent transportation systems, intelligent monitoring systems, and industrial inspection.

[0003] The basic idea of ​​existing target detection methods is to first process the image to obtain candidate regions (candidate frames), and then classify and regress the candidate regions to achieve target detection and recognition of the image. However, due to the huge number of candidate frames, in order to eliminate redundant candidate frames, find the best object detection position, and speed up target detection and recognition, methods such as NMS (Non-maximum suppression) algorithms are generally used to remove redundant candidate frames.

[0004] However, when the NMS algorithm filters the candidate boxes according to the order of their category confidence, since the category classifier for obtaining the category confidence of the candidate boxes is robust to occlusion and translation, the first candidate box of the real object in the complete frame, and the second candidate box with an offset position and only part of the real object in the frame (for example, 80% of the real object in the frame), the category confidence of the first candidate box is lower than the category confidence of the second candidate box. Assuming that the IOU threshold of the NMS algorithm is set to 0.5, the first candidate box will be filtered. It can be seen that relying solely on the category confidence cannot accurately reflect the accuracy of the candidate box, which may cause the accurate candidate box to be filtered, resulting in too high a filtering error rate of the NMS algorithm, resulting in low accuracy of the target detection algorithm. Summary of the invention

[0005] The embodiments of the present invention provide a target detection method and related devices, which can reduce the amount of target prediction information generated during target detection and better improve the processing efficiency of target detection.

[0006] On the one hand, an embodiment of the present invention provides a target detection method, including:

[0007] Acquire the image to be detected;

[0008] Acquire initial prediction information of the target according to the image to be detected, wherein the initial prediction information includes an initial prediction border, an initial prediction category confidence corresponding to the initial prediction border, and an initial prediction offset, and the initial prediction offset is used to reflect the degree of offset between the initial prediction border and the target border;

[0009] Initial prediction information is screened according to the initial prediction category confidence and the initial prediction offset to obtain prediction information of the target, wherein the prediction information includes a prediction bounding box and a corresponding prediction category confidence.

[0010] The initial prediction information is screened according to the initial prediction category confidence and the initial prediction deviation to obtain the prediction information of the target, wherein the prediction information includes a prediction bounding box and a corresponding prediction category confidence, including:

[0011] Constructing a two-dimensional Gaussian distribution function corresponding to the initial predicted border of the target according to the position information of the initial predicted border;

[0012] Constructing a first mixed Gaussian distribution function based on the two-dimensional Gaussian distribution function and the corresponding weight, wherein the weight of the two-dimensional Gaussian distribution function is the ratio of the bias product to the total bias product, wherein the bias product is the product of the initial prediction category confidence and the initial prediction offset of the initial prediction bounding box corresponding to the two-dimensional Gaussian distribution function, and the total bias product is the sum of all the bias products in the image to be detected;

[0013] Processing the first mixed Gaussian distribution function according to an estimation algorithm to obtain a second mixed Gaussian distribution function, wherein the estimation algorithm is used to find a set of two-dimensional Gaussian distribution functions and corresponding weights from the first mixed Gaussian distribution function to form the second mixed Gaussian distribution function, so that the second mixed Gaussian distribution function represents the first mixed Gaussian distribution function, and the number of items of the second mixed Gaussian distribution function is less than the number of items of the first mixed Gaussian distribution function;

[0014] Filter each weight in the second mixed Gaussian distribution function according to a preset weight range, and obtain a weight within the preset weight range as a first weight;

[0015] According to the two-dimensional Gaussian distribution function corresponding to the first weight, the corresponding initial prediction bounding box and the initial prediction category confidence are obtained, and the obtained initial prediction bounding box and the initial prediction category confidence are used as the prediction bounding box and the prediction category confidence of the target.

[0016] The estimation algorithm is an estimation algorithm that performs density estimation on a mixed Gaussian distribution function through iteration.

[0017] Wherein, the initial prediction information of the target is obtained according to the image to be detected, including:

[0018] Initial prediction information of the target is obtained according to the image to be detected and the detection model, wherein the detection model is trained using a training image containing annotation information based on a loss function that reflects the difference between position information of an initial prediction bounding box obtained by the detection model and position information of a target bounding box, the difference between an initial prediction category confidence obtained by the detection model and target category information, and the difference between an initial prediction offset degree obtained by the detection model and a target offset degree, wherein the annotation information includes position information of a target bounding box of the training image, target category information and target offset degree.

[0019] Wherein, the method further comprises:

[0020] Dividing the image to be detected into a plurality of grids at intervals of a first preset length, wherein the grids contain at least one complete target, and the image to be detected has initial prediction information or prediction information of the target;

[0021] Perform the following processing on each mesh separately:

[0022] Non-maximum suppression processing is performed on the initial prediction border or the prediction border located in the grid to eliminate part of the initial prediction border or part of the prediction border in the grid.

[0023] Wherein, the method further comprises:

[0024] Dividing the feature map into a plurality of grids at intervals of a second preset length, wherein the feature map is obtained by extracting features from the image to be detected, the grids contain at least one complete target after feature extraction, and the image to be detected has initial prediction information or prediction information of the target;

[0025] Perform the following processing on each mesh separately:

[0026] Non-maximum suppression processing is performed on the initial prediction border or the prediction border mapped in the grid to eliminate a portion of the initial prediction border or the prediction border mapped in the grid.

[0027] Wherein, the method further comprises:

[0028] Clustering is performed on the training image containing the position information of the target frame to obtain multiple clustering frame widths and multiple clustering frame heights;

[0029] Calculate the average box width of multiple cluster boxes and the average box height of multiple cluster boxes;

[0030] The larger value of the average frame width and the average frame height is obtained, and the product of a preset multiple and the larger value is used as the first preset length.

[0031] Wherein, the initial prediction information of the target is obtained according to the image to be detected, including:

[0032] Performing feature extraction on the image to be detected to obtain a feature map;

[0033] Setting a first anchor point at the center of each cell of the feature map, and setting a plurality of first anchor frames with the same anchor frame area and different length-to-width ratios around the first anchor point, wherein the areas of the first anchor frames in different cells are different;

[0034] Obtaining intersection-and-union ratios between the first anchor frame and the target frame mapped on the feature map respectively, and obtaining a maximum value of multiple intersection-and-union ratios as a maximum intersection-and-union ratio;

[0035] Taking the length of the first anchor frame corresponding to the maximum intersection-over-union ratio as a step length, setting a plurality of second anchor points on the feature map, and generating a plurality of second anchor frames at the positions of the second anchor points according to the anchor frame area of ​​the first anchor point corresponding to the maximum intersection-over-union ratio and the length-to-width ratio of the anchor frame;

[0036] The initial predicted bounding box of the target is generated according to the second anchor box.

[0037] On the other hand, an embodiment of the present invention provides a target detection device, including:

[0038] An image acquisition unit, used for acquiring an image to be detected;

[0039] An initial prediction information acquisition unit, used to acquire initial prediction information of the target according to the image to be detected, wherein the initial prediction information includes an initial prediction border, an initial prediction category confidence corresponding to the initial prediction border, and an initial prediction deviation, and the initial prediction deviation is used to reflect the degree of deviation between the initial prediction border and the target border;

[0040] The target prediction information acquisition unit is used to perform initial prediction information screening according to the initial prediction category confidence and the initial prediction deviation to obtain the prediction information of the target, wherein the prediction information includes a prediction bounding box and a corresponding prediction category confidence.

[0041] Wherein, the device further comprises:

[0042] A first division unit is used to divide the image to be detected into a plurality of grids at intervals of a first preset length, wherein the grids contain at least one complete target, and initial prediction information or prediction information of the target is displayed on the image to be detected;

[0043] The first culling unit is used to perform the following processing on each mesh separately:

[0044] Non-maximum suppression processing is performed on the initial prediction border or the prediction border located in the grid to eliminate part of the initial prediction border or part of the prediction border in the grid.

[0045] Wherein, the device further comprises:

[0046] a second division unit, configured to divide the feature map into a plurality of grids at intervals of a second preset length, wherein the feature map is obtained by extracting features from the image to be detected, the grids contain at least one complete target after feature extraction, and initial prediction information or prediction information of the target is displayed on the image to be detected;

[0047] The second culling unit is used to perform the following processing on each mesh separately:

[0048] Non-maximum suppression processing is performed on the initial prediction border or the prediction border mapped in the grid to eliminate a portion of the initial prediction border or the prediction border mapped in the grid.

[0049] Wherein, the initial prediction information acquisition unit includes:

[0050] A feature extraction module, used for extracting features from the image to be detected to obtain a feature map;

[0051] A first anchor frame setting module, configured to respectively set a first anchor point at the center of each cell of the feature map, and to set a plurality of first anchor frames with the same anchor frame area and different length-to-width ratios around the first anchor point, wherein the areas of the first anchor frames in different cells are different;

[0052] an IoU acquisition module, used to respectively acquire IoU ratios between the first anchor frame and a target frame mapped on the feature map, and acquire a maximum value among a plurality of IoU ratios as a maximum IoU ratio;

[0053] A second anchor frame setting module, configured to set a plurality of second anchor points on the feature map with a length of the first anchor frame corresponding to the maximum intersection-and-union ratio as a step length, and generate a plurality of second anchor frames at positions of the second anchor points with an anchor frame area of ​​the first anchor point corresponding to the maximum intersection-and-union ratio and a length-to-width ratio of the anchor frame;

[0054] An initial prediction box generation module is used to generate the initial prediction box of the target according to the second anchor box.

[0055] On the other hand, an embodiment of the present invention provides a terminal device, including: a processor and a memory;

[0056] The processor is connected to a memory, wherein the memory is used to store program codes, and the processor is used to call the program codes to execute the target detection method.

[0057] On the other hand, an embodiment of the present invention provides a computer storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the target detection method is executed.

[0058] When performing target detection, the embodiment of the present invention can obtain initial prediction information of the image to be detected, wherein the initial prediction information not only includes an initial prediction border and a corresponding initial prediction category confidence, but also includes an initial prediction offset reflecting the degree of offset between the initial prediction border and the target border; then the initial prediction information is screened according to the initial prediction category confidence and the initial prediction offset, and part of the initial prediction information is filtered to obtain final prediction information of the target, wherein the prediction information includes a prediction border and a corresponding prediction category confidence; since the initial prediction information is screened based on both the initial prediction category confidence and the initial prediction offset, not only can the amount of prediction information of the target generated during target detection be reduced, but also the filtering accuracy of invalid prediction information can be effectively improved, and valid prediction information about the target is retained, which helps to improve the detection accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0060] Figure 1 is a scene schematic diagram of a target detection method provided by an embodiment of the present invention;

[0061] Figure 2 is a schematic diagram of a scenario of another target detection method provided by an embodiment of the present invention;

[0062] Figure 3 is a flow chart of a target detection method provided by an embodiment of the present invention;

[0063] Figure 4 It is a schematic diagram of a process of screening initial prediction information in a target detection method provided by an embodiment of the present invention;

[0064] Figure 5 It is a visualization diagram of the two-dimensional Gaussian distribution function;

[0065] Figure 6 It is a schematic diagram of a process of filtering predicted borders in a target detection method provided by an embodiment of the present invention;

[0066] Figure 7 It is a schematic diagram of gridding an image to be detected in a target detection method provided by an embodiment of the present invention;

[0067] Figure 8 It is a schematic diagram of a process of filtering predicted borders in another target detection method provided by an embodiment of the present invention;

[0068] Fig. 9 is a schematic diagram of gridding a feature map in a target detection method provided by an embodiment of the present invention;

[0069] Fig.10 It is a schematic diagram of a process of obtaining initial prediction information of a target in a target detection method provided by an embodiment of the present invention;

[0070] Fig.11 is a schematic diagram of a feature map in a target detection method provided by an embodiment of the present invention;

[0071] Fig.12 is a schematic structural diagram of a target detection device provided by an embodiment of the present invention;

[0072] Fig.13 is a schematic structural diagram of another target detection device provided by an embodiment of the present invention;

[0073] Fig.14 is a schematic structural diagram of another target detection device provided by an embodiment of the present invention;

[0074] Fig.15 is a structural schematic diagram of an initial prediction information acquisition unit of a target detection device provided by an embodiment of the present invention;

[0075] Fig.16 It is a structural diagram of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0077] It should be understood that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0078] See also Figure 1 , Figure 1 is a scene schematic diagram of a target detection method provided by an embodiment of the present invention; Figure 1 In the description of the target detection method, a shelf image is taken as an example. The user can use a terminal such as a mobile phone 11 to enter the WeChat mini program section, and then enter the corresponding mini program A. After the user fills in relevant information (such as the supermarket where the shelf is located) in the mini program A, he starts to take pictures of the goods on the shelf, and a shelf product image 12 can be obtained; the mini program A is a product recognition application created based on the target detection method of the embodiment of the present invention. The application is installed in the cloud server 13. Therefore, after uploading the shelf product image 12 to the cloud server 13 through the mini program A, the shelf product is identified in the cloud server 13. The image 12 is used to detect and identify goods, and a specific product identification result can be obtained. At the same time, a result statistical table 15 will be generated according to the product identification result. For large chain supermarkets, a standard-reaching statistical table 14 will also be generated. After the standard-reaching standard is set in advance, the cloud server 13 determines whether the shelf product image 12 meets the standard according to the preset standard-reaching standard and the product identification result, that is, it can determine whether the product on the shelf of the supermarket where the shelf is located is qualified. Finally, the standard-reaching statistical table 14 and the result statistical table 15 are returned to the mobile phone 11, and the user can quickly determine the standard-reaching status of the supermarket according to the returned table. Among them, the product identification result includes the product category (such as beer, cola, orange juice, etc.), the corresponding quantity of the product, etc.

[0079] refer to Figure 2 , Figure 2 is a schematic diagram of another target detection method provided by an embodiment of the present invention; Figure 1The solution can also be applied to the monitoring camera 21, which is installed in the store to automatically obtain the shelf commodity image in real time, and transmit the shelf commodity image to the cloud server 22 by wired or wireless means, and perform commodity detection and identification on the shelf commodity image in the cloud server 22, and generate commodity identification results. Similarly, the standard-reaching statistical table and the result statistical table are generated according to the commodity identification results; in addition, the cloud server 22 will also generate the total standard-reaching statistical table of all stores. When the store staff needs it, they can send the corresponding result statistical table and standard-reaching statistical table to the cloud server 22 through the mobile phone 23 and other terminals; similarly, when the management center of the chain supermarket, that is, the staff of the headquarters, wants to understand the situation of each store, they can send the total standard-reaching statistical table or the statistical table of a certain store to the cloud server 22 through the computer 24 of the headquarters, so that the staff of the headquarters can control the execution of all stores in real time. It is realized that everyone from individuals to the headquarters can control the changes in shelves in real time, ensure the smooth flow of sales channels, control the execution of store activities, maximize sales efficiency, and greatly reduce the time of field sales staff in the store, and improve operational efficiency.

[0080] The specific process of commodity detection on shelf commodity images can be found in the following Figures 3 to 11 Of course, the target detection method described in the embodiment is not limited to the recognition and detection of shelf goods, but can also be the recognition and detection of various other targets, such as vehicle statistics detection on the road (vehicle type and number, etc.), or pedestrian statistics detection (total number of pedestrians, etc.), or face recognition detection, or urban building statistics detection, or animal statistics detection, etc., all of which are applicable.

[0081] See also Figure 3 , is a flow chart of a target detection method provided by an embodiment of the present invention; the target detection method comprises:

[0082] Step S301, obtaining an image to be detected;

[0083] Specifically, an image to be detected containing the target is obtained using an image acquisition device. The image acquisition device, for example, a camera set on an intelligent terminal (such as a mobile phone, etc.) or a special monitoring camera, etc., can obtain the image to be detected containing the target.

[0084] Step S302, obtaining initial prediction information of the target according to the image to be detected, wherein the initial prediction information includes an initial prediction bounding box (i.e., an initial candidate bounding box), an initial prediction category confidence corresponding to the initial prediction bounding box, and an initial prediction offset, wherein the initial prediction offset is used to reflect the offset degree between the initial prediction bounding box and the target bounding box;

[0085] Specifically, a pre-trained detection model is used to process the image to be detected to obtain initial prediction information of the target, wherein the initial prediction offset is used to reflect the degree of offset between the position information of the initial prediction frame and the position information of the target frame (referring to the real frame of the target), and the position information of the frame refers to information indicating a frame, which can be represented by the coordinates of the center point of the frame and the width and height of the frame (taking the length and width of a rectangle as an example, the width of the frame corresponds to the length in the usual sense, and the height of the frame corresponds to the width in the usual sense), or by two corner points of the frame, such as the specific coordinates of the upper left corner point and the lower right corner point; or by multiple sides (such as two sides, with the extreme points of the two sides given) to represent a frame. The algorithm for obtaining the initial predicted bounding box and the corresponding initial category confidence can be implemented by using the target detection algorithm in the prior art, for example, the anchor-based target detection algorithm, R-CNN algorithm, Faster R-CNN algorithm, RPN algorithm, FPN algorithm and SSD algorithm, etc., or the traditional algorithm, which uses a pyramid multi-scale and traversal sliding window method to determine whether there is a recognized target at this scale and position by scale and position; or the YOLO algorithm, etc.

[0086] Among them, the YOLO algorithm is a one-stage fast target detection algorithm. YOLO divides the entire image into S×S grids, each grid is responsible for detecting the target centered in the grid, and solves the problem at one time by predicting the bbox, confidence, and all category probability vectors of the targets contained in all grids at one time.

[0087] The FPN algorithm proposes a method that can effectively generate multi-dimensional feature expressions of images of the same scale in a single image view by utilizing the feature expression structure of each layer from bottom to top in the conventional convolutional neural network model. It can effectively empower the conventional convolutional neural network model, thereby generating feature maps with stronger expressive power.

[0088] The SSD algorithm is a one-stage target detection algorithm that uses a convolutional neural network to perform detection directly, rather than performing detection after the fully connected layer like Yolo.

[0089] Step S303 , performing initial prediction information screening according to the initial prediction category confidence and the initial prediction deviation to obtain prediction information of the target, wherein the prediction information includes a prediction bounding box (ie, a candidate box) and a corresponding prediction category confidence.

[0090] Specifically, the initial prediction information is screened according to the obtained initial prediction category confidence and initial prediction offset of the target, some invalid and erroneous initial prediction information is removed, and the initial prediction bounding box and the corresponding initial prediction category confidence obtained after filtering (i.e. the remaining) are used as the prediction bounding box and prediction category confidence of the target respectively, so that the amount of prediction information of the target finally obtained is greatly reduced. Since the initial prediction information is screened by combining the initial prediction category confidence and the initial prediction offset at the same time, it is avoided that in the prior art, due to the robustness of the category classifier to occlusion and translation, a single category confidence cannot accurately reflect the accuracy of the candidate box, which in turn causes the accurate prediction bounding box to be filtered, resulting in low target detection accuracy; the filtering accuracy of invalid initial prediction information is effectively improved, and the accuracy of target detection is also improved.

[0091] When performing target detection, the embodiment of the present invention can obtain initial prediction information of the image to be detected, wherein the initial prediction information not only includes an initial prediction border and a corresponding initial prediction category confidence, but also includes an initial prediction offset reflecting the degree of offset between the initial prediction border and the target border; then the initial prediction information is screened according to the initial prediction category confidence and the initial prediction offset, and part of the initial prediction information is filtered to obtain final prediction information of the target, wherein the prediction information includes a prediction border and a corresponding prediction category confidence; since the initial prediction information is screened based on both the initial prediction category confidence and the initial prediction offset, not only can the amount of prediction information of the target generated during target detection be reduced, but also the filtering accuracy of invalid prediction information can be effectively improved, and valid prediction information about the target is retained, which helps to improve the detection accuracy of target detection.

[0092] In particular, when two prediction boxes, one large and one small, are generated around the target during target detection, the category confidence of the large prediction box is higher than that of the small prediction box, and the large prediction box is closer to the real border of the target. The small prediction box is about one tenth of the large prediction box. When the existing NMS algorithm is used to filter the prediction boxes, since the intersection-over-union ratio between the large and small prediction boxes is less than a preset threshold (assuming that the threshold is set to 0.5), the small prediction box (invalid prediction box) cannot be removed at this time, resulting in subsequent redundant calculations. However, by using the method of the embodiment of the present invention, the initial prediction box is filtered in combination with the category confidence and offset of the initial prediction box, so that the above-mentioned small prediction box can be filtered, thereby improving the filtering accuracy of the invalid border.

[0093] In addition, the prediction information of the target also includes a prediction offset, which is used to reflect the degree of offset between the prediction frame (position information) and the target frame (position information). Specifically, the initial prediction offset obtained after filtering is used as the prediction offset of the target. Optionally, the method of the embodiment of the present invention also includes outputting the prediction information of the target. When the method of the embodiment of the present invention is applied to a computer or a mobile phone, the prediction information of the target can be output on a display screen, so that the user can know the recognition and detection results of the target.

[0094] Further, refer to Figure 3 , the target detection method of the embodiment of the present invention further includes:

[0095] Step S304, post-processing the prediction information of the target, the post-processing includes removing redundant prediction borders, etc., and can be to use the existing NMS algorithm (including the standard NMS algorithm or the improved NMS algorithm) to continue filtering redundant prediction borders through intersection-over-union calculation to reduce the number of prediction borders.

[0096] Optionally, in step S302, obtaining initial prediction information of the target according to the image to be detected includes:

[0097] Initial prediction information of the target is obtained according to the image to be detected and the detection model, wherein the detection model is trained using a training image containing annotation information based on a loss function that reflects the difference between position information of an initial prediction bounding box obtained by the detection model and position information of a target bounding box, the difference between an initial prediction category confidence obtained by the detection model and target category information, and the difference between an initial prediction offset degree obtained by the detection model and a target offset degree, wherein the annotation information includes position information of a target bounding box of the training image, target category information and target offset degree.

[0098] Specifically, multiple training images containing annotation information are obtained. Taking shelf commodity images as an example, a large number of shelf commodity images need to be obtained to cover all commodities as much as possible; the bounding boxes and categories of the targets in the training images are annotated by manual annotation, which may be to frame the real bounding box of the target (i.e., the target bounding box), and a bounding box representation method is selected to represent the target bounding box. For example, the target bounding box is represented by the center point of the bounding box and the width and height of the bounding box, so that the position information of the target bounding box (the coordinates of the center point of the bounding box and the width and height of the bounding box) and the target category information (i.e., the real category information of the target) can be obtained; and the target offset is the offset between the initial predicted bounding box of the detection model and the target bounding box. Since the position information of the target bounding box is known, the detection model can also obtain the position information of the initial predicted bounding box (the same as the target bounding box, represented by the center point of the bounding box and the width and height of the bounding box), and the target offset (i.e., the real offset of the target) can be obtained according to the position information of the initial predicted bounding box and the position information of the target bounding box. In addition, the loss functions used in model training include a first loss function for the initial predicted border, a second loss function for the initial category confidence, and a third loss function for the initial offset. One or more of the first loss function, the second loss function, and the third loss function can adopt a cross-entropy loss function. The cross-entropy loss function is used to narrow the difference between the real labeled data and the predicted data and improve the prediction accuracy of the detection model.

[0099] Then start training the detection model. Each time a training image is input, a set of output data can be obtained, including the initial predicted bounding box, the initial predicted category confidence and the initial predicted offset. Then compare it with the real labeled data of the corresponding training image. When the data output by the model is significantly different from the real labeled data, adjust the parameters of the model; otherwise, input the next training image and repeat the above process. Repeat the model parameters of the detection model to minimize the difference between the real labeled data and the output data of the model. Then the detection model training is completed.

[0100] Optional, reference Figure 4 , is a schematic diagram of a process of screening initial prediction information in a target detection method provided by an embodiment of the present invention, wherein step S303 includes:

[0101] Step S401, constructing a two-dimensional Gaussian distribution function corresponding to the initial predicted border of the target according to the position information of the initial predicted border;

[0102] Specifically, the corresponding two-dimensional Gaussian distribution function is constructed according to the position information of the initial predicted border. Taking the center point of the border and the width and height of the border as examples, according to the coordinates of the center point of the initial predicted border and the width and height of the initial predicted border As a parameter fitting two-dimensional Gaussian distribution, the two-dimensional Gaussian distribution function corresponding to the initial predicted border can be obtained. N initial predicted borders can obtain N two-dimensional Gaussian distribution functions. The two-dimensional Gaussian distribution function is shown in the following formula:

[0103]

[0104] in, p Represents a two-dimensional Gaussian distribution function, see Figure 5 , is a visualization diagram of the two-dimensional Gaussian distribution function, namely the Gaussian heat map, Figure 5 Visualization of the two-dimensional Gaussian distribution function of 5 targets. p That is Figure 5 The set of all two-dimensional coordinate points in a single ellipse 51 in .

[0105] Step S402, constructing a first mixed Gaussian distribution function based on the two-dimensional Gaussian distribution function and the corresponding weight, wherein the weight of the two-dimensional Gaussian distribution function is the ratio of the bias product to the total bias product, wherein the bias product is the product of the initial prediction category confidence and the initial prediction offset of the initial prediction bounding box corresponding to the two-dimensional Gaussian distribution function, and the total bias product is the sum of all the bias products in the image to be detected;

[0106] Specifically, a weighted operation is performed according to each two-dimensional Gaussian distribution function and the corresponding weight to construct a first mixed Gaussian distribution function, wherein the weight of the two-dimensional Gaussian distribution function is the partial product c The ratio of the total bias product, bias product c is the product of the initial prediction category confidence and the initial prediction offset of the initial prediction bounding box corresponding to the two-dimensional Gaussian distribution function. Therefore, the weight of a two-dimensional Gaussian distribution function is As follows:

[0107] ,

[0108] Then the first mixed Gaussian distribution function can be expressed as:

[0109] ,

[0110] in, Indicates i The first mixed Gaussian distribution function is equivalent to superimposing N two-dimensional Gaussian distribution functions according to different weights.

[0111] Step S403, processing the first mixed Gaussian distribution function according to an estimation algorithm to obtain a second mixed Gaussian distribution function, wherein the estimation algorithm is used to find a set of two-dimensional Gaussian distribution functions and corresponding weights from the first mixed Gaussian distribution function to form the second mixed Gaussian distribution function, so that the second mixed Gaussian distribution function represents the first mixed Gaussian distribution function, and the number of items of the second mixed Gaussian distribution function is less than the number of items of the first mixed Gaussian distribution function;

[0112] Specifically, an estimation algorithm is used to find a set (more than one) of two-dimensional Gaussian distribution functions and corresponding weights from the first mixed Gaussian distribution function. The Gaussian heat map corresponding to this set of data can represent the Gaussian heat map of the first mixed Gaussian distribution function. According to this set of data, a second mixed Gaussian distribution function can be obtained. The second mixed Gaussian distribution function can represent the first mixed Gaussian distribution function, and the number of items of the second mixed Gaussian distribution function is much smaller than the number of items of the first mixed Gaussian distribution function, that is, this process can reduce the number of two-dimensional Gaussian distribution functions of the image to be detected, which is equivalent to removing some invalid initial prediction borders of the image to be detected and suppressing some invalid initial prediction borders. In fact, in the process of obtaining the second mixed Gaussian distribution function from the first mixed Gaussian distribution function, for the unselected two-dimensional Gaussian distribution functions (that is, the remaining items of the first mixed Gaussian distribution function except for the items in the second mixed Gaussian distribution function), the weights corresponding to these items are set to preset values ​​to achieve filtering to obtain the second mixed Gaussian distribution function. Among them, the preset value is any value less than or equal to 0, or greater than or equal to 1. When the preset value is set to 0, it is equivalent to directly eliminating the item with a weight of 0. At this time, the first mixed Gaussian distribution function is the second mixed Gaussian distribution function; and when the preset value is set to a value less than 0, or greater than or equal to 1, the items with weights between 0 and 1 in the first mixed Gaussian distribution function are taken out through weight comparison and judgment to form the second mixed Gaussian distribution function.

[0113] Optionally, the estimation algorithm is an estimation algorithm for density estimation of a mixed Gaussian distribution function by iteration, that is, using the estimation algorithm, the density estimation of the first mixed Gaussian distribution function is performed to obtain a set of data that best represents the first mixed Gaussian distribution function, and this set of data constitutes the second mixed Gaussian distribution function. The estimation algorithm can be an algorithm based on density estimation of a mixed Gaussian distribution function by iteration. Specifically, the estimation algorithm includes an EM algorithm, a density estimation algorithm or a clustering algorithm, wherein the idea of ​​EM is that the first step is to guess the implicit category variable, and the second step is to update other parameters to obtain the maximum maximum likelihood estimate. In the embodiment of the present invention, that is, by continuously iteratively updating the value of the weight of each item in the first mixed Gaussian distribution function, in order to improve the calculation processing speed, here, taking the value of the weight as 0 as an example, directly eliminating the corresponding item to find the maximum maximum likelihood estimate, when the maximum maximum likelihood estimate is found, the first mixed Gaussian distribution function at this time is the second mixed Gaussian distribution function.

[0114] Step S404, screening each weight in the second mixed Gaussian distribution function according to a preset weight range, and obtaining a weight within the preset weight range as a first weight;

[0115] Specifically, in order to further improve the accuracy of the predicted border of the target, a preset weight range can be set to screen the weights of the second mixed Gaussian distribution function. The preset weight range can be set to any value between 0 and 1 (including 0) according to specific needs, such as 0.2-0.99. Users can adjust the preset weight range as needed, remove items with lower weight values ​​to further reduce the invalid predicted border of the target, and improve the detection accuracy of the target.

[0116] Step S405, according to the two-dimensional Gaussian distribution function corresponding to the first weight, obtain the corresponding initial prediction bounding box and the initial prediction category confidence, and use the obtained initial prediction bounding box and the initial prediction category confidence as the prediction bounding box and the prediction category confidence of the target.

[0117] Specifically, under ideal conditions, the position information of the initial prediction bounding box used to construct the function can be found according to the two-dimensional Gaussian distribution function corresponding to the first weight. According to the position information, the corresponding initial prediction bounding box, initial prediction category confidence and initial prediction offset can be found in all the initial prediction information of the target. At this time, the found initial prediction bounding box can be used as the prediction bounding box of the target, and the initial prediction category confidence is the prediction category confidence of the target; similarly, the initial prediction offset is the prediction offset of the target.

[0118] Furthermore, the embodiment of the present invention also provides two methods for post-processing the prediction information of the target. In the embodiment of the present invention, two methods for filtering the prediction border of the target are provided, which is equivalent to the method for filtering the prediction information of the target. The first method, in order to reduce the number of the initial prediction border or the prediction border of the target and improve the border filtering speed, refers to Figure 6 , Figure 6 : is a flow chart of filtering predicted borders in a target detection method provided by an embodiment of the present invention; the target detection method further includes:

[0119] Step S601, dividing the image to be detected into a plurality of grids at intervals of a first preset length, wherein the grids contain at least one complete target, and the image to be detected has initial prediction information or prediction information of the target;

[0120] Specifically, it is necessary to obtain multiple training images containing targets in advance, where the targets can be one or more different targets. For example, taking shelf goods as an example, the shelf goods image can include beverages such as cola, beer, orange juice, and cans of orange juice, cola, etc. The obtained training images try to cover all target types that are intended for subsequent target detection; then the targets in the obtained training images are marked with borders to obtain target borders, and then clustering is performed based on the training images containing the position information of the target borders to obtain multiple cluster frame widths and multiple cluster frame heights; then the average frame width of the multiple cluster frame widths and the average frame height of the multiple cluster frame heights are calculated; finally, the larger value of the average frame width and the average frame height is obtained, and the product of the preset multiple and the larger value is used as the first preset length. Generally, the preset multiple is more than 2 times.

[0121] refer to Figure 7 , Figure 7 Schematic diagram of grid division of an image to be detected in a target detection method provided by an embodiment of the present invention. The image to be detected 73 is divided into grids at intervals of a first preset length D1, and the following is obtained: Figure 7 The multiple grids shown in the to-be-detected image 73 may include initial prediction information of a target (such as 71 ), such as an initial prediction border 72 , or may include prediction information of a target, such as a prediction border.

[0122] Step S602, perform the following processing on each grid:

[0123] Non-maximum suppression processing is performed on the initial prediction border or the prediction border located in the grid to eliminate part of the initial prediction border or part of the prediction border in the grid.

[0124] Specifically, it is necessary to first sort the category confidence of the initial prediction borders or prediction borders in all grids from high to low; then perform non-maximum suppression processing on the initial prediction borders or prediction borders in the same grid to remove redundant initial prediction borders or prediction borders. Since the non-maximum suppression processing is only performed in the same grid, the initial prediction borders or prediction borders that are not in the same grid are not processed, which reduces the amount of calculation during border filtering and improves the border filtering speed. The non-maximum suppression processing can be implemented using a standard NMS algorithm or an improved algorithm based on the NMS algorithm.

[0125] The standard NMS algorithm process is to first select the one with the largest category confidence among all prediction boxes, record it as box_best, and keep it; then calculate the IOU of box_best and the remaining prediction boxes. If this IOU is greater than the threshold set by NMS (usually 0.5), the prediction box is discarded; finally, find the next box_best from the remaining prediction boxes (excluding the previous box_best), and repeat this cycle.

[0126] The second method is to reduce the number of initial predicted bounding boxes or predicted bounding boxes of the target and improve the bounding box filtering speed. For the initial predicted bounding boxes and preset bounding boxes obtained by the anchor-based target detection algorithm, bounding box filtering can be performed based on the feature map of the image to be detected. Figure 8 , Figure 8 is a flow chart of filtering predicted borders in another target detection method provided by an embodiment of the present invention; the target detection method further includes:

[0127] Step S801, dividing a feature map into a plurality of grids at intervals of a second preset length, wherein the feature map is obtained by extracting features from the image to be detected, the grids contain at least one complete target after feature extraction (equivalent to containing a scaled target), and the image to be detected has initial prediction information or prediction information of the target;

[0128] Specifically, after the first preset length is obtained according to the above method, the second preset length can be obtained according to the first preset length. The ratio between the first preset length and the second preset length is the same as the scaling ratio between the image to be detected and the feature map. Therefore, the second preset length can be obtained after scaling the first preset length by the same ratio. Fig. 9 , Fig. 9It is a schematic diagram of gridding a feature map in a target detection method provided by an embodiment of the present invention; the feature map of the image to be detected is divided into multiple grids with a second preset length D2 as intervals, and the feature map includes multiple anchor frames 92 and scaled targets 91. The anchor frame 92 can be an anchor frame corresponding to an initial predicted border, or an anchor frame corresponding to a predicted border.

[0129] Step S802: perform the following processing on each grid:

[0130] Non-maximum suppression processing is performed on the initial prediction border or the prediction border mapped in the grid to eliminate a portion of the initial prediction border or the prediction border mapped in the grid.

[0131] Specifically, it is necessary to first sort the category confidences of all initial prediction bounding boxes or prediction bounding boxes mapped in the grid from high to low; then perform non-maximum suppression processing on the initial prediction bounding boxes or prediction bounding boxes mapped in the same grid, that is, perform non-maximum suppression processing on the anchor boxes (anchor boxes corresponding to the initial prediction bounding boxes or prediction bounding boxes) located in the same grid, and remove redundant anchor boxes to remove redundant initial prediction bounding boxes or prediction bounding boxes. Since non-maximum suppression processing is only performed in the same grid, anchor boxes that are not in the same grid are not processed, which reduces the amount of calculation during border filtering and improves the border filtering speed. The non-maximum suppression processing can be implemented using a standard NMS algorithm or an improved algorithm based on the NMS algorithm.

[0132] refer to Figure 6 and Figure 8 , the non-maximum suppression processing method based on region perception can speed up the calculation speed of repeated border filtering and save calculation cost. In particular, when the target is in a dense scene, that is, there are a large number of small targets in the image to be detected, a large number of predicted borders will be generated. Figure 6 or Figure 8 This method can quickly remove a large number of redundant prediction borders and has high filtering efficiency.

[0133] Optionally, when the detection model is an anchor-based detection model, the embodiment of the present invention proposes a new method for setting anchor points, which reduces the number of anchor points to reduce the generated initial prediction bounding box, reduces the amount of calculation, and improves the efficiency of target detection. Fig.10 and Fig.11 , Fig.10 is a schematic diagram of a process for obtaining initial prediction information of a target in a target detection method provided by an embodiment of the present invention, Fig.11 is a schematic diagram of a feature map in a target detection method provided by an embodiment of the present invention; the initial prediction information of the target is obtained according to the image to be detected, including:

[0134] Step S101, extracting features from the image to be detected to obtain a feature map;

[0135] Generally, the image to be detected is input into a convolutional neural network for feature extraction to obtain a corresponding feature map.

[0136] Step S102, setting a first anchor point at the center of each cell of the feature map, and setting a plurality of first anchor frames with the same anchor frame area and different length-to-width ratios around the first anchor point, wherein the areas of the first anchor frames in different cells are different;

[0137] Specifically, assuming the size of the feature map is d d, then the feature map has d d cells, for example, if d is 5, then 25 cells can be obtained, a first anchor point 112 is set at the center of each cell, and multiple first anchor frames 111 are generated with the first anchor point 112 as the center. Fig.11 In the example, taking each first anchor point 112 generating three first anchor frames 111, a total of 5 5 3=75 first anchor frames 111, wherein each first anchor point 112 generates a first anchor frame 111 with different anchor frame area sizes and the same or different anchor frame length-to-width ratios. In short, the first anchor frames generated by the first anchor points in different cells have different areas, and the anchor frame area and anchor frame length-to-width ratio can be set as required. Fig.11 Take the three cells in the upper left corner of the example, which are cell A, cell B and cell C. Assume that the area of ​​the anchor boxes selected is 16 16, 32 32 and 64 64, the length-to-width ratios of the anchor boxes are set to 1:1, 1:2, and 2:1, respectively. Then, three anchor boxes with an area of ​​16 16 first anchor frames 111 , and the length-to-width ratios of the three first anchor frames 111 are 1:1, 1:2 and 2:1 respectively, so a total of 9 first anchor frames 111 can be obtained.

[0138] Step S103, respectively obtaining intersection-over-union ratios between the first anchor frame and the target frame mapped on the feature map, and obtaining a maximum value of multiple intersection-over-union ratios as a maximum intersection-over-union ratio;

[0139] Specifically, the intersection-over-union (IOU) between all the first anchor frames 111 and the target bounding box (ie, the real bounding box of the target) mapped on the feature map is calculated respectively, and the maximum value of all the IOUs is obtained as the maximum IOU.

[0140] Step S104, setting a plurality of second anchor points on the feature map with the length of the first anchor frame corresponding to the maximum intersection-over-union ratio as a step length, and generating a plurality of second anchor frames at the positions of the second anchor points with the anchor frame area of ​​the first anchor point corresponding to the maximum intersection-over-union ratio and the length-to-width ratio of the anchor frame;

[0141] Among them, reference Fig.11 , when the length of the first anchor box corresponding to the maximum intersection-and-union ratio is D3, the second anchor point 113 is set from left to right on the feature map at intervals of D3, and the second anchor box is generated with the second anchor point 113 as the center according to the anchor box area of ​​the first anchor point corresponding to the maximum intersection-and-union ratio and the length-to-width ratio of the anchor box. For example, assuming that the first anchor box corresponding to the maximum intersection-and-union ratio is Fig.11 The first anchor box 111 in the , then the first anchor point corresponding to the maximum intersection-over-union ratio is Fig.11 The first anchor point 112 in the image is equivalent to generating multiple second anchor frames with the second anchor point 113 as the center based on the anchor frame area corresponding to cell A and the length-to-width ratio of the anchor frame.

[0142] It is worth pointing out that, since the second anchor points are set at intervals of the length of the first anchor frame corresponding to the maximum intersection-over-union ratio in the embodiment of the present invention, that is, the step size changes with the size of the target, it can not only significantly reduce the number of generated anchor frames, effectively reduce the generation of negative anchors, reduce the redundant calculation of a large number of anchors, but also solve the problem of imbalance between positive and negative anchors. In particular, the method of the embodiment of the present invention can also improve the detection accuracy of small targets. When setting the anchor points in accordance with the existing method (such as step S102), since the first anchor points are all set at the center of the cell, the anchor frame may not be able to frame the small target, resulting in the small target being unable to be detected; and using the method of the embodiment of the present invention, when D3 determined by the above steps is less than d, the small target may be detected at this time due to the reduction of the step size.

[0143] In addition, the way of setting anchor points in the embodiment of the present invention will have a prominent improvement effect on the images to be detected in dense scenes (the dense scenes here refer to scenes with a large number of targets, especially scenes with a large number of different targets of the same or similar shapes, such as retail shelf merchandise, traffic vehicles and urban landscape images). In the existing methods, for the images to be detected in dense scenes, a large number of dense anchor points are usually set in order to frame all the targets, resulting in a high degree of overlap between the anchor frames and causing redundant calculations. The way of setting anchor points in the embodiment of the present invention can effectively reduce the number of anchor points and reduce redundant calculations.

[0144] Step S105: generating the initial predicted bounding box of the target according to the second anchor box.

[0145] Specifically, since there is a certain scaling ratio between the feature map and the image to be detected, an initial predicted bounding box of the target can be generated on the image to be detected according to the second anchor frame according to the certain scaling ratio.

[0146] When performing target detection, the embodiment of the present invention screens the initial prediction information based on both the initial prediction category confidence and the initial prediction offset, which can not only reduce the amount of prediction information of the target generated during target detection, but also effectively improve the filtering accuracy of invalid prediction information, retain the valid prediction information about the target, and better improve the detection accuracy of target detection. In addition, especially for target detection in dense scenes, based on regional non-maximum suppression processing, the amount of calculation when filtering redundant prediction borders can be reduced, and the efficiency of target detection can be improved; and based on the new anchor point setting method, the generation of negative anchors can be effectively reduced, the amount of redundant calculations can be reduced, and the efficiency of target detection can be further improved.

[0147] Based on the description of the above target detection method embodiment, the present invention also discloses a target detection device. Fig.12 , Fig.12 1 is a schematic diagram of the structure of a target detection device provided by an embodiment of the present invention, wherein the target detection device comprises an image acquisition unit 121, an initial prediction information acquisition unit 122, and a target prediction information acquisition unit 123; wherein:

[0148] An image acquisition unit 121 is used to acquire an image to be detected;

[0149] An initial prediction information acquisition unit 122 is used to acquire initial prediction information of the target according to the image to be detected, wherein the initial prediction information includes an initial prediction border, an initial prediction category confidence corresponding to the initial prediction border, and an initial prediction offset, and the initial prediction offset is used to reflect the offset between the initial prediction border and the target border;

[0150] A target prediction information acquisition unit 123 is used to perform initial prediction information screening according to the initial prediction category confidence and the initial prediction deviation to obtain prediction information of the target, wherein the prediction information includes a prediction bounding box and a corresponding prediction category confidence;

[0151] The post-processing unit 124 is used to post-process the prediction information of the target.

[0152] The specific functional implementation of the image acquisition unit 121, the initial prediction information acquisition unit 122, the target prediction information acquisition unit 123, and the post-processing unit 124 can be referred to in the above Figure 3 Steps S301 to S304 in the corresponding embodiment, Figure 6 , Figure 8 The description in the corresponding embodiment will not be repeated here.

[0153] In another embodiment, referring to Fig.13 , Fig.13 is a schematic diagram of the structure of another target detection device provided by an embodiment of the present invention; the target detection device further includes a first division unit 131 and a first elimination unit 132, wherein:

[0154] The first division unit 131 is used to divide the image to be detected into a plurality of grids at intervals of a first preset length, wherein the grids contain at least one complete target, and initial prediction information or prediction information of the target is displayed on the image to be detected;

[0155] The first elimination unit 132 is used to perform the following processing on each mesh:

[0156] Non-maximum suppression processing is performed on the initial prediction border or the prediction border located in the grid to eliminate part of the initial prediction border or part of the prediction border in the grid.

[0157] The specific functional implementation of the first division unit 131 and the first elimination unit 132 can be referred to in the above Figure 6 Steps S601 and S602 in the corresponding embodiment will not be described in detail here.

[0158] In another embodiment, reference Fig.14 , Fig.14 is a schematic diagram of the structure of another target detection device provided by an embodiment of the present invention; the target detection device further includes a second division unit 141 and a second elimination unit 142, wherein:

[0159] A second division unit 141 is used to divide the feature map into a plurality of grids at intervals of a second preset length, wherein the feature map is obtained by extracting features from the image to be detected, the grids contain at least one complete target after feature extraction, and the initial prediction information or prediction information of the target is displayed on the image to be detected;

[0160] The second elimination unit 142 is used to perform the following processing on each grid:

[0161] Non-maximum suppression processing is performed on the initial prediction border or the prediction border mapped in the grid to eliminate a portion of the initial prediction border or the prediction border mapped in the grid.

[0162] The specific functional implementation of the second division unit 141 and the second elimination unit 142 can be referred to in the above Figure 8Steps S801 and S802 in the corresponding embodiment will not be described in detail here.

[0163] Further, refer to Fig.15 , Fig.15 1 is a schematic diagram of the structure of an initial prediction information acquisition unit of a target detection device provided by an embodiment of the present invention, wherein the initial prediction information acquisition unit 122 includes a feature extraction module 151, a first anchor frame setting module 152, an intersection-over-union ratio acquisition module 153, a second anchor frame setting module 154, and an initial prediction frame generation module 155, wherein:

[0164] A feature extraction module 151 is used to extract features from the image to be detected to obtain a feature map;

[0165] A first anchor frame setting module 152 is used to set a first anchor point at the center of each cell of the feature map, and set a plurality of first anchor frames with the same anchor frame area and different length-to-width ratios with the first anchor point as the center, wherein the areas of the first anchor frames in different cells are different;

[0166] an IoU acquisition module 153, configured to respectively acquire IoU ratios between the first anchor frame and the target frame mapped on the feature map, and acquire a maximum value among a plurality of IoU ratios as a maximum IoU ratio;

[0167] A second anchor frame setting module 154 is configured to set a plurality of second anchor points on the feature map with a length of the first anchor frame corresponding to the maximum intersection-over-union ratio as a step length, and generate a plurality of second anchor frames at positions of the second anchor points with an anchor frame area of ​​the first anchor point corresponding to the maximum intersection-over-union ratio and a length-to-width ratio of the anchor frame;

[0168] The initial prediction bounding box generating module 155 is configured to generate the initial prediction bounding box of the target according to the second anchor box.

[0169] The specific functional descriptions of the feature extraction module 151, the first anchor frame setting module 152, the intersection-over-union ratio acquisition module 153, the second anchor frame setting module 154, and the initial prediction frame generation module 155 can be found in Fig.10 Steps S101 to S105 in the corresponding embodiment will not be described in detail here.

[0170] It is worth pointing out that Fig.12 , Fig.13 , Fig.14 and Fig.15The various units or modules in the target detection device shown can be combined into one or several other units or modules separately or in whole, or some of the units or modules can be further divided into multiple smaller units or modules in function, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units or modules are divided based on logical functions. In practical applications, the functions of one unit (or module) can also be realized by multiple units (or modules), or the functions of multiple units (or modules) can be realized by one unit (or module).

[0171] Based on the description of the above method embodiment and device embodiment, an embodiment of the present invention further provides a terminal device,

[0172] See also Fig.16 , is a schematic diagram of the structure of a terminal device provided by an embodiment of the present invention. Fig.16 As shown above Figures 12 to 15 The target detection device in the embodiment can be applied to the terminal device 160, and the terminal device 160 may include: a processor 161, a network interface 164 and a memory 165. In addition, the terminal device 160 may also include: a user interface 163, and at least one communication bus 162. The communication bus 162 is used to realize the connection and communication between these components. The user interface 163 may include a display screen (Display), a keyboard (Keyboard), and the user interface 163 may also include a standard wired interface and a wireless interface. The network interface 164 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 165 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 165 may optionally be at least one storage device located away from the aforementioned processor 161. Fig.16 As shown, the memory 165 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0173] exist Fig.16 In the terminal device 160 shown, the network interface 164 can provide a network communication function; the user interface 163 is mainly used to provide an input interface for the user; and the processor 161 can be used to call the device control application stored in the memory 165 to achieve:

[0174] Acquire the image to be detected;

[0175] Acquire initial prediction information of the target according to the image to be detected, wherein the initial prediction information includes an initial prediction border, an initial prediction category confidence corresponding to the initial prediction border, and an initial prediction offset, and the initial prediction offset is used to reflect the degree of offset between the initial prediction border and the target border;

[0176] Initial prediction information is screened according to the initial prediction category confidence and the initial prediction offset to obtain prediction information of the target, wherein the prediction information includes a prediction bounding box and a corresponding prediction category confidence.

[0177] In one embodiment, when the processor 161 performs initial prediction information screening according to the initial prediction category confidence and the initial prediction offset to obtain prediction information of the target, wherein the prediction information includes a prediction bounding box and a corresponding prediction category confidence, the processor 161 specifically performs the following steps:

[0178] Constructing a two-dimensional Gaussian distribution function corresponding to the initial predicted border of the target according to the position information of the initial predicted border;

[0179] Constructing a first mixed Gaussian distribution function based on the two-dimensional Gaussian distribution function and the corresponding weight, wherein the weight of the two-dimensional Gaussian distribution function is the ratio of the bias product to the total bias product, wherein the bias product is the product of the initial prediction category confidence and the initial prediction offset of the initial prediction bounding box corresponding to the two-dimensional Gaussian distribution function, and the total bias product is the sum of all the bias products in the image to be detected;

[0180] Processing the first mixed Gaussian distribution function according to an estimation algorithm to obtain a second mixed Gaussian distribution function, wherein the estimation algorithm is used to find a set of two-dimensional Gaussian distribution functions and corresponding weights from the first mixed Gaussian distribution function to form the second mixed Gaussian distribution function, so that the second mixed Gaussian distribution function represents the first mixed Gaussian distribution function, and the number of items of the second mixed Gaussian distribution function is less than the number of items of the first mixed Gaussian distribution function;

[0181] Filter each weight in the second mixed Gaussian distribution function according to a preset weight range, and obtain a weight within the preset weight range as a first weight;

[0182] According to the two-dimensional Gaussian distribution function corresponding to the first weight, the corresponding initial prediction bounding box and the initial prediction category confidence are obtained, and the obtained initial prediction bounding box and the initial prediction category confidence are used as the prediction bounding box and the prediction category confidence of the target.

[0183] In one embodiment, when the processor 161 acquires initial prediction information of the target according to the image to be detected, the processor 161 specifically performs the following steps:

[0184] Initial prediction information of the target is obtained according to the image to be detected and the detection model, wherein the detection model is trained using a training image containing annotation information based on a loss function that reflects the difference between position information of an initial prediction bounding box obtained by the detection model and position information of a target bounding box, the difference between an initial prediction category confidence obtained by the detection model and target category information, and the difference between an initial prediction offset degree obtained by the detection model and a target offset degree, wherein the annotation information includes position information of a target bounding box of the training image, target category information and target offset degree.

[0185] In one embodiment, the processor 161 further performs the following steps:

[0186] Clustering is performed on the training image containing the position information of the target frame to obtain multiple clustering frame widths and multiple clustering frame heights;

[0187] Calculate the average box width of multiple cluster boxes and the average box height of multiple cluster boxes;

[0188] The larger value of the average frame width and the average frame height is obtained, and the product of a preset multiple and the larger value is used as the first preset length.

[0189] In one embodiment, the processor 161 further performs the following steps:

[0190] Dividing the image to be detected into a plurality of grids at intervals of a first preset length, wherein the grids contain at least one complete target, and the image to be detected has initial prediction information or prediction information of the target;

[0191] Perform the following processing on each mesh separately:

[0192] Non-maximum suppression processing is performed on the initial prediction border or the prediction border located in the grid to eliminate part of the initial prediction border or part of the prediction border in the grid.

[0193] In one embodiment, the processor 161 further performs the following steps:

[0194] Dividing the feature map into a plurality of grids at intervals of a second preset length, wherein the feature map is obtained by extracting features from the image to be detected, the grids contain at least one complete target after feature extraction, and the image to be detected has initial prediction information or prediction information of the target;

[0195] Perform the following processing on each mesh separately:

[0196] Non-maximum suppression processing is performed on the initial prediction border or the prediction border mapped in the grid to eliminate a portion of the initial prediction border or the prediction border mapped in the grid.

[0197] In one embodiment, when the processor 161 acquires initial prediction information of the target according to the image to be detected, the processor 161 specifically performs the following steps:

[0198] Performing feature extraction on the image to be detected to obtain a feature map;

[0199] Setting a first anchor point at the center of each cell of the feature map, and setting a plurality of first anchor frames with the same anchor frame area and different length-to-width ratios around the first anchor point, wherein the areas of the first anchor frames in different cells are different;

[0200] Obtaining intersection-and-union ratios between the first anchor frame and the target frame mapped on the feature map respectively, and obtaining a maximum value of multiple intersection-and-union ratios as a maximum intersection-and-union ratio;

[0201] Taking the length of the first anchor frame corresponding to the maximum intersection-over-union ratio as a step length, setting a plurality of second anchor points on the feature map, and generating a plurality of second anchor frames at the positions of the second anchor points according to the anchor frame area of ​​the first anchor point corresponding to the maximum intersection-over-union ratio and the length-to-width ratio of the anchor frame;

[0202] The initial predicted bounding box of the target is generated according to the second anchor box.

[0203] It should be understood that the terminal device 160 described in the embodiment of the present invention can execute the above Figures 3 to 11 The description of the target detection method in the corresponding embodiment can also be performed as described above. Figures 12 to 15 The description of the target detection device in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here.

[0204] In addition, it should be pointed out here that: an embodiment of the present invention further provides a computer storage medium, and the computer storage medium stores a computer program executed by the target detection device mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figures 3 to 11 The description of the target detection method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer storage medium embodiment involved in the present invention, please refer to the description of the method embodiment of the present invention.

[0205] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0206] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A target detection method, characterized in that: include: Acquire the image to be detected; Acquire initial prediction information of the target according to the image to be detected, wherein the initial prediction information includes an initial prediction border, an initial prediction category confidence corresponding to the initial prediction border, and an initial prediction offset, and the initial prediction offset is used to reflect the degree of offset between the initial prediction border and the target border; Performing initial prediction information screening according to the initial prediction category confidence and the initial prediction deviation to obtain prediction information of the target, wherein the prediction information includes a prediction bounding box and a corresponding prediction category confidence; The initial prediction information screening is performed according to the initial prediction category confidence and the initial prediction deviation to obtain the prediction information of the target, including: Constructing a two-dimensional Gaussian distribution function corresponding to the initial predicted border of the target according to the position information of the initial predicted border; Constructing a first mixed Gaussian distribution function based on the two-dimensional Gaussian distribution function and the corresponding weight, wherein the weight of the two-dimensional Gaussian distribution function is the ratio of the bias product to the total bias product, wherein the bias product is the product of the initial prediction category confidence and the initial prediction offset of the initial prediction bounding box corresponding to the two-dimensional Gaussian distribution function, and the total bias product is the sum of all the bias products in the image to be detected; Processing the first mixed Gaussian distribution function according to an estimation algorithm to obtain a second mixed Gaussian distribution function, wherein the estimation algorithm is used to find a set of two-dimensional Gaussian distribution functions and corresponding weights from the first mixed Gaussian distribution function to form the second mixed Gaussian distribution function, so that the second mixed Gaussian distribution function represents the first mixed Gaussian distribution function, and the number of items of the second mixed Gaussian distribution function is less than the number of items of the first mixed Gaussian distribution function; Filter each weight in the second mixed Gaussian distribution function according to a preset weight range, and obtain a weight within the preset weight range as a first weight; According to the two-dimensional Gaussian distribution function corresponding to the first weight, the corresponding initial prediction bounding box and the initial prediction category confidence are obtained, and the obtained initial prediction bounding box and the initial prediction category confidence are used as the prediction bounding box and the prediction category confidence of the target.

2. The method according to claim 1, characterized in that The estimation algorithm is an estimation algorithm that performs density estimation on a mixed Gaussian distribution function through iteration.

3. The method according to claim 1, characterized in that: The obtaining of initial prediction information of the target according to the image to be detected includes: Initial prediction information of the target is obtained according to the image to be detected and the detection model, wherein the detection model is trained using a training image containing annotation information based on a loss function that reflects the difference between position information of an initial prediction bounding box obtained by the detection model and position information of a target bounding box, the difference between an initial prediction category confidence obtained by the detection model and target category information, and the difference between an initial prediction offset degree obtained by the detection model and a target offset degree, wherein the annotation information includes position information of a target bounding box of the training image, target category information and target offset degree.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Dividing the image to be detected into a plurality of grids at intervals of a first preset length, wherein the grids contain at least one complete target, and the image to be detected has initial prediction information or prediction information of the target; Perform the following processing on each mesh separately: Non-maximum suppression processing is performed on the initial prediction border or the prediction border located in the grid to eliminate part of the initial prediction border or part of the prediction border in the grid.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Dividing the feature map into a plurality of grids at intervals of a second preset length, wherein the feature map is obtained by extracting features from the image to be detected, the grids contain at least one complete target after feature extraction, and the image to be detected has initial prediction information or prediction information of the target; Perform the following processing on each mesh separately: Non-maximum suppression processing is performed on the initial prediction border or the prediction border mapped in the grid to eliminate a portion of the initial prediction border or the prediction border mapped in the grid.

6. The method according to claim 4, characterized in that The method further comprises: Clustering is performed on the training image containing the position information of the target frame to obtain multiple clustering frame widths and multiple clustering frame heights; Calculate the average box width of multiple cluster boxes and the average box height of multiple cluster boxes; The larger value of the average frame width and the average frame height is obtained, and the product of a preset multiple and the larger value is used as the first preset length.

7. The method according to any one of claims 1 to 3, characterized in that: The obtaining of initial prediction information of the target according to the image to be detected includes: Performing feature extraction on the image to be detected to obtain a feature map; Setting a first anchor point at the center of each cell of the feature map, and setting a plurality of first anchor frames with the same anchor frame area and different length-to-width ratios around the first anchor point, wherein the areas of the first anchor frames in different cells are different; Obtaining intersection-and-union ratios between the first anchor frame and the target frame mapped on the feature map respectively, and obtaining a maximum value of multiple intersection-and-union ratios as a maximum intersection-and-union ratio; Taking the length of the first anchor frame corresponding to the maximum intersection-over-union ratio as a step length, setting a plurality of second anchor points on the feature map, and generating a plurality of second anchor frames at the positions of the second anchor points according to the anchor frame area of ​​the first anchor point corresponding to the maximum intersection-over-union ratio and the length-to-width ratio of the anchor frame; The initial predicted bounding box of the target is generated according to the second anchor box.

8. A target detection device, characterized in that: include: An image acquisition unit, used for acquiring an image to be detected; An initial prediction information acquisition unit, used to acquire initial prediction information of the target according to the image to be detected, wherein the initial prediction information includes an initial prediction border, an initial prediction category confidence corresponding to the initial prediction border, and an initial prediction deviation, and the initial prediction deviation is used to reflect the degree of deviation between the initial prediction border and the target border; A target prediction information acquisition unit is used to construct a two-dimensional Gaussian distribution function corresponding to the initial prediction border of the target according to the position information of the initial prediction border; construct a first mixed Gaussian distribution function based on the two-dimensional Gaussian distribution function and the corresponding weight, the weight of the two-dimensional Gaussian distribution function is the ratio of the set bias product to the total set bias product, wherein the set bias product is the product of the initial prediction category confidence and the initial prediction offset of the initial prediction border corresponding to the two-dimensional Gaussian distribution function, and the total set bias product is the sum of all the set bias products in the image to be detected; the first mixed Gaussian distribution function is processed according to an estimation algorithm to obtain a second mixed Gaussian distribution function, and the estimation algorithm is used to obtain a second mixed Gaussian distribution function from the first mixed Gaussian distribution function Find a set of two-dimensional Gaussian distribution functions and corresponding weights to constitute the second mixed Gaussian distribution function, so that the second mixed Gaussian distribution function represents the first mixed Gaussian distribution function, and the number of items of the second mixed Gaussian distribution function is less than the number of items of the first mixed Gaussian distribution function; filter each weight in the second mixed Gaussian distribution function according to a preset weight range, and obtain the weight within the preset weight range as the first weight; according to the two-dimensional Gaussian distribution function corresponding to the first weight, filter and obtain the corresponding initial prediction bounding box and the initial prediction category confidence, and use the filtered initial prediction bounding box and the initial prediction category confidence as the prediction bounding box and prediction category confidence of the target.

9. The device according to claim 8, characterized in that The device also includes: A first division unit is used to divide the image to be detected into a plurality of grids at intervals of a first preset length, wherein the grids contain at least one complete target, and initial prediction information or prediction information of the target is displayed on the image to be detected; The first culling unit is used to perform the following processing on each mesh separately: Non-maximum suppression processing is performed on the initial prediction border or the prediction border located in the grid to eliminate part of the initial prediction border or part of the prediction border in the grid.

10. The device according to claim 8, characterized in that The device also includes: a second division unit, configured to divide the feature map into a plurality of grids at intervals of a second preset length, wherein the feature map is obtained by extracting features from the image to be detected, the grids contain at least one complete target after feature extraction, and initial prediction information or prediction information of the target is displayed on the image to be detected; The second culling unit is used to perform the following processing on each mesh separately: Non-maximum suppression processing is performed on the initial prediction border or the prediction border mapped in the grid to eliminate a portion of the initial prediction border or the prediction border mapped in the grid.

11. The device according to any one of claims 8 to 10, characterized in that The initial prediction information acquisition unit comprises: A feature extraction module, used for extracting features from the image to be detected to obtain a feature map; A first anchor frame setting module, configured to respectively set a first anchor point at the center of each cell of the feature map, and to set a plurality of first anchor frames with the same anchor frame area and different length-to-width ratios around the first anchor point, wherein the areas of the first anchor frames in different cells are different; an IoU acquisition module, used to respectively acquire IoU ratios between the first anchor frame and a target frame mapped on the feature map, and acquire a maximum value among a plurality of IoU ratios as a maximum IoU ratio; A second anchor frame setting module, configured to set a plurality of second anchor points on the feature map with a length of the first anchor frame corresponding to the maximum intersection-and-union ratio as a step length, and generate a plurality of second anchor frames at positions of the second anchor points with an anchor frame area of ​​the first anchor point corresponding to the maximum intersection-and-union ratio and a length-to-width ratio of the anchor frame; An initial prediction box generation module is used to generate the initial prediction box of the target according to the second anchor box.

12. A terminal device, characterized in that: include: Processor and memory; The processor is connected to a memory, wherein the memory is used to store program code, and the processor is used to call the program code to execute the target detection method according to any one of claims 1-7.

13. A computer storage medium, characterized in that: The computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the target detection method according to any one of claims 1 to 7 is executed.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the target detection method according to any one of claims 1 to 7 are implemented.

Images