Target detection model training method, target detection method, and related device

Abstract

The embodiment of the present application provides a kind of target detection model training method, obtains the target detection model to be trained and target data set;Sample image is handled by feature extraction layer and feature fusion layer, and the fusion feature map of sample image is obtained;The fusion feature map is input into main prediction layer and is handled, and the prediction frame of sample image is obtained;The fusion feature map is input into auxiliary prediction layer and is handled, and the prediction area of sample image is obtained;Based on prediction frame, prediction area and label frame, the parameter adjustment of target detection model to be trained is carried out, and the trained target detection model is obtained after parameter adjustment is completed.The training of main prediction layer is assisted by auxiliary prediction layer, so that main prediction layer can better carry out feature learning, and then the training speed and recognition accuracy of target detection model are improved.

Description

Technical Field

[0001] This invention relates to the field of image detection, and more particularly to a method for training a target detection model, a target detection method, and related equipment. Background Technology

[0002] With advancements in deep learning, object detection algorithms have achieved significant technological breakthroughs. These algorithms are essential for image recognition in many fields. However, existing algorithms are typically based on Region Proposal-based R-CNN systems. While R-CNN is a two-stage algorithm, it offers high accuracy but is relatively slow, making it inefficient for image recognition and unsuitable for scenarios requiring high-efficiency image identification. Another type is the YOLO or SSD system. YOLO-based algorithms are one-stage algorithms, offering faster recognition speeds but slower convergence, lower target localization accuracy, and a higher rate of false background detection. Therefore, developing a detection model that is both fast and accurate to achieve efficient and high-accuracy object detection has become a pressing issue. Summary of the Invention

[0003] This invention provides a method for training an object detection model, aiming to address the problems of low accuracy or slow recognition speed in existing object detection algorithms. The method processes the image to be detected through a feature extraction layer and a feature fusion layer to obtain a fused feature map. This fused feature map is then input into the main prediction layer and auxiliary prediction layer for prediction processing, resulting in corresponding prediction boxes and prediction regions. The influence of the prediction region on the prediction result of the prediction box is comprehensively considered, ultimately completing the training of the object detection model to achieve high efficiency and high accuracy in object recognition of the image to be detected.

[0004] In a first aspect, embodiments of the present invention provide a method for training a target detection model, the method comprising:

[0005] Obtain the target detection model to be trained and the target dataset. The target dataset includes sample images and bounding boxes. The sample images include the target to be detected. The target detection model to be trained includes a feature extraction layer, a feature fusion layer and a prediction layer. The prediction layer includes a main prediction layer and an auxiliary prediction layer.

[0006] The sample image is processed by the feature extraction layer and the feature fusion layer to obtain the fused feature map of the sample image;

[0007] The fused feature map is input into the main prediction layer for processing to obtain the prediction bounding box of the sample image;

[0008] The fused feature map is input into the auxiliary prediction layer for processing to obtain the predicted region of the sample image;

[0009] Based on the prediction box, the prediction region, and the annotation box, the parameters of the target detection model to be trained are adjusted, and a trained target detection model is obtained after the parameter adjustment is completed. The trained target detection model includes a trained feature extraction layer, a trained feature fusion layer, and a trained main prediction layer.

[0010] Optionally, the step of inputting the fused feature map into the main prediction layer for processing to obtain the prediction bounding box of the sample image includes:

[0011] The fused feature map is input into the main prediction layer for processing to obtain the label assignment result corresponding to the sample image;

[0012] Based on the label assignment results, the prediction bounding box of the sample image is determined.

[0013] Optionally, the main prediction layer includes a label matching strategy, which inputs the fused feature map into the main prediction layer for processing to obtain the label assignment result of the sample image, including:

[0014] The fused feature map is input into the main prediction layer for processing to obtain the first predicted feature of the sample image;

[0015] Based on the label matching strategy of the main prediction layer and the bounding box, the label assignment result of the sample image is determined in the first prediction feature of the sample image.

[0016] Optionally, the step of inputting the fused feature map into the auxiliary prediction layer for processing to obtain the prediction region of the sample image includes:

[0017] The fused feature map is input into the auxiliary prediction layer for processing to obtain the second prediction feature of the sample image, the second prediction feature including the category channel corresponding to the detection category;

[0018] Based on the second prediction feature of the sample image, the prediction region of the sample image is determined.

[0019] Optionally, adjusting the parameters of the target detection model to be trained based on the predicted bounding box, the predicted region, and the labeled bounding box includes:

[0020] Based on the predicted bounding box and the labeled bounding box, a first loss is determined;

[0021] Based on the predicted region and the labeled box, a second loss is determined;

[0022] Based on the first loss and the second loss, determine the total loss;

[0023] The parameters of the target detection model to be trained are adjusted based on the total loss.

[0024] Optionally, determining the second loss based on the predicted region and the bounding box includes:

[0025] The bounding box is mapped to the second predicted feature to obtain the labeled region of the sample image;

[0026] A second loss is determined based on the predicted region and the labeled region.

[0027] Secondly, embodiments of the present invention provide a target detection method, the target detection method comprising:

[0028] Acquire the image to be detected;

[0029] The image to be detected is input into a trained target detection model for processing to obtain the target detection result of the image to be detected. The trained target detection model is obtained according to the target detection model training method.

[0030] Thirdly, embodiments of the present invention also provide a target detection model training device, the target detection model training device comprising:

[0031] The first acquisition module is used to acquire the target detection model to be trained and the target dataset. The target dataset includes sample images and bounding boxes. The sample images include the target to be detected. The target detection model to be trained includes a feature extraction layer, a feature fusion layer and a prediction layer. The prediction layer includes a main prediction layer and an auxiliary prediction layer.

[0032] The first processing module is used to process the sample image through the feature extraction layer and the feature fusion layer to obtain the fused feature map of the sample image;

[0033] The second processing module is used to input the fused feature map into the main prediction layer for processing to obtain the prediction box of the sample image;

[0034] The third processing module is used to input the fused feature map into the auxiliary prediction layer for processing to obtain the prediction region of the sample image;

[0035] The adjustment module is used to adjust the parameters of the target detection model to be trained based on the prediction box, the prediction region, and the annotation box, and to obtain a trained target detection model after the parameter adjustment is completed. The trained target detection model includes a trained feature extraction layer, a trained feature fusion layer, and a trained main prediction layer.

[0036] Fourthly, embodiments of the present invention also provide a target detection device, the target detection device comprising:

[0037] The second acquisition module is used to acquire the image to be detected;

[0038] The fourth processing module is used to input the image to be detected into the trained target detection model for processing, and to obtain the target detection result of the image to be detected. The trained target detection model is obtained according to the target detection model training method.

[0039] Fifthly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the target detection model training method provided in the embodiments of the present invention or the steps in the target detection method provided in the embodiments of the present invention.

[0040] Sixthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps in the target detection model training method provided in embodiments of the present invention or the steps in the target detection method provided in embodiments of the present invention.

[0041] In this embodiment of the invention, a target detection model to be trained and a target dataset are obtained; sample images are processed through a feature extraction layer and a feature fusion layer to obtain a fused feature map of the sample images; the fused feature map is input into the main prediction layer for processing to obtain the predicted bounding boxes of the sample images; the fused feature map is input into the auxiliary prediction layer for processing to obtain the predicted regions of the sample images; based on the predicted bounding boxes, predicted regions, and labeled boxes, the parameters of the target detection model to be trained are adjusted, and a trained target detection model is obtained after the parameter adjustment is completed. The target image to be detected is processed through a feature extraction layer and a feature fusion layer to obtain a fused feature map of the target image. The fused feature map of the target image is then input into the main prediction layer and the auxiliary prediction layer for prediction processing. The auxiliary prediction layer assists the training of the main prediction layer, enabling the main prediction layer to better learn features, thereby improving the training speed and recognition accuracy of the target detection model. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of a target detection model training method provided in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the structure of a target detection model to be trained according to an embodiment of the present invention;

[0045] Figure 3 This is a flowchart of a target detection method provided in an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of the structure of a target detection model training device provided in an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of the structure of a target detection device provided in an embodiment of the present invention;

[0048] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] like Figure 1 As shown, Figure 1 This is a flowchart of a method for training an object detection model according to an embodiment of the present invention. The object detection model training method includes the following steps:

[0051] 101. Obtain the target detection model to be trained and the target dataset.

[0052] In this embodiment of the invention, the target detection model can be deployed on a server or server cluster. The server or server cluster can be an electronic device with functions such as image processing, image storage, image recognition, and data transmission. The target dataset can be a sample dataset used to train the target detection model. The target dataset includes sample images and bounding boxes. Specifically, the sample images can refer to image examples used for training, testing, or demonstration purposes. The bounding boxes can be data annotation tools used to locate and identify target objects in the sample images. By defining the target object region in the sample images using bounding boxes, the location and size of the target can be accurately indicated. The bounding boxes can be determined manually or through neural network algorithms such as R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN.

[0053] The aforementioned object detection can detect targets such as faces, bodies, and vehicles. Different detection targets correspond to different sample images. When detecting faces, the sample image can be a face image; when detecting bodies, the sample image can be a body image; and when detecting vehicles, the sample image can be a vehicle image. The aforementioned object detection can also be multi-target detection, simultaneously detecting targets such as faces, bodies, and vehicles. In this case, the sample image can include at least one target such as a face, body, or vehicle.

[0054] In this embodiment of the invention, the target detection model to be trained can be a target detection model based on a convolutional neural network. Specifically, the target detection model includes a feature extraction layer, a feature fusion layer, and a prediction layer. The prediction layer can include a main prediction layer and an auxiliary prediction layer. The main prediction layer and the auxiliary prediction layer are prediction layers with different prediction modes. Further, the main prediction layer, the feature extraction layer, and the feature fusion layer are different structural layers within the same target detection model; the main prediction layer and the auxiliary prediction layer are prediction layers in different target detection models. For example, the main prediction layer can be a YOLOv series prediction layer, whose prediction mode is the predicted bounding box of the target; the auxiliary prediction layer can be a CenterNet prediction layer, whose prediction mode is the predicted center point and offset of the target.

[0055] The aforementioned feature extraction layer includes convolutional layers and pooling layers. The convolutional layers are used to perform convolution operations on the sample image to extract the spatial and semantic information of the sample image, resulting in a feature map containing both spatial and semantic information. The pooling layers are used to perform downsampling operations on the feature map output by the convolutional layers to obtain a feature map with a smaller scale. Through multiple convolutional and downsampling operations, the sample image can be processed into feature maps of multiple scales. These feature maps of multiple scales contain spatial and semantic information at multiple scales. Downsampling can reduce image storage space, reduce computational complexity, and improve the speed of sample image processing. The downsampling operation can be methods such as average pooling and max pooling.

[0056] The aforementioned feature fusion layer is used to perform image feature fusion operations on feature maps of different scales. Specifically, the image feature fusion operation combines and organizes feature maps of different scales to obtain a more comprehensive and accurate feature map. This image feature fusion operation can include methods such as concatenation, weighted summation, averaging, and element-wise operations. It is understood that larger-scale feature maps can retain richer spatial and semantic details, while smaller-scale feature maps have a larger receptive field. By performing feature fusion operations on feature maps of different scales, the resulting fused feature map can more accurately represent the features of the target object.

[0057] The aforementioned prediction layer can be the output layer used to regress the prediction results of the sample image. Specifically, the output of the main prediction layer can be a predicted bounding box, and the output of the auxiliary prediction layer can be the center point and offset. In actual use after training, the auxiliary prediction layer can be discarded, thus retaining only the main prediction layer. One main prediction layer can output predicted bounding boxes for multiple objects. The predicted bounding box can be a colored rectangular bounding box used to define the target location in the sample image. The predicted bounding box can be represented by four values: (x, y, width, height). Where (x, y) are the coordinates of the top-left corner of the predicted bounding box, and width and height are the width and height of the predicted bounding box. Using these four values, a rectangle can be accurately drawn on the sample image to mark the target object.

[0058] 102. The sample image is processed through a feature extraction layer and a feature fusion layer to obtain the fused feature map of the sample image.

[0059] In this embodiment of the invention, the fusion feature map of the above sample image can be obtained by fusing feature maps of different scales of the above sample image.

[0060] Specifically, the sample image can be input into the feature extraction layer for feature extraction processing to obtain feature maps of multiple scales corresponding to the sample image. The feature maps of multiple scales corresponding to the sample image can then be input into the feature fusion layer for feature fusion processing to obtain fused feature maps of multiple scales of the sample image.

[0061] In one possible embodiment, after the server or server cluster acquires the target monitoring model to be trained and the target dataset, the sample images in the target dataset are input into the feature extraction layer for convolution and pooling processing to obtain the feature map corresponding to the sample image. The feature map corresponding to the sample image is then input into the feature fusion layer for pooling processing to obtain the fused feature map corresponding to the image features. The pooling processing may include upsampling and downsampling.

[0062] More specifically, such as Figure 2 As shown, Figure 2 This is a schematic diagram of a target detection model to be trained according to an embodiment of the present invention. Figure 2 In this process, the sample image is input into the feature extraction layer for feature extraction. The convolutional layer in the feature extraction layer performs a convolution operation on the sample image to obtain the feature map of the sample image. The pooling layer performs a downsampling operation on the feature map to obtain feature maps at different scales. The downsampling operations include 2x downsampling (2×), 4x downsampling (4×), 8x downsampling (8×), 16x downsampling (16×), and 32x downsampling (32×). Taking the sample image scale as 1024*1024 as an example, after the 2x downsampling operation, a 512*512 feature map is obtained; after the 4x downsampling operation, a 256*256 feature map is obtained; after the 8x downsampling operation, a 128*128 feature map is obtained; after the 16x downsampling operation, a 64*64 feature map is obtained; and after the 32x downsampling operation, a 32*32 feature map is obtained. In the feature fusion layer, feature fusion can be performed at scales of 128*128, 64*64, and 32*32. Figure 2In this process, the 32*32 feature map is upsampled to obtain a 64*64 feature map. This 64*64 feature map is then fused with the original 64*64 feature map for the first time, resulting in a fused 64*64 feature map. This fused 64*64 feature map is then upsampled again to obtain a 128*128 feature map. This fused 128*128 feature map is then fused with the original 128*128 feature map for the second time, resulting in a fused 64*64 feature map. This fused 64*64 feature map is then downsampled to obtain a 64*64 feature map. This fused 64*64 feature map is then downsampled again to obtain a 32*32 feature map. This fused 32*32 feature map is then fused with the original 32*32 feature map for the third time. Through the above feature fusion layer, a 32*32 fused feature map, a 64*64 fused feature map, and a 128*128 fused feature map are obtained. The above fusion can be superimposed fusion. Specifically, each feature map can be regarded as a channel and the channels can be superimposed to achieve feature fusion.

[0063] 103. Input the fused feature map into the main prediction layer for processing to obtain the prediction box of the sample image.

[0064] In this embodiment of the invention, the number of the aforementioned main prediction layers can be multiple, and the multiple main prediction layers are fused with the aforementioned multiple scales of features. Figure 1 There is a one-to-one correspondence, with each fused feature map corresponding to a main prediction layer. This main prediction layer can be a prediction structure within a deep convolutional neural network, such as the prediction portion of a YOLOv7 or YOLOv5 deep convolutional neural network.

[0065] In one possible embodiment, after the sample image is processed by the feature extraction layer and the feature fusion layer to obtain the fused feature map, the fused feature map is input to the main prediction layer for prediction processing to obtain the (x, y, width, height) of the prediction box. Based on the (x, y, width, height), the coordinates of the upper left corner, the width, and the height of the prediction box are determined. Using the coordinates of the upper left corner, the width, and the height of the prediction box, a rectangle is accurately drawn on the sample image to mark the target object. The rectangle can be a color rectangle, and the drawing color of the rectangle can correspond to the type of the target object. Each target object corresponds to a type, and each type corresponds to a drawing color. For example, if the target object is a dog, the corresponding type can be animal, and the corresponding drawing color can be green. If the target object is a pine tree, the corresponding type can be plant, and the corresponding drawing color can be yellow.

[0066] 104. Input the fused feature map into the auxiliary prediction layer for processing to obtain the prediction region of the sample image.

[0067] In this embodiment of the invention, the number of auxiliary prediction layers is multiple, and the multiple auxiliary prediction layers can be two or more layers. These multiple prediction layers can be fused with the features of the multiple scales. Figure 1 In a one-to-one correspondence, each of the above-mentioned fused feature maps corresponds to an auxiliary prediction layer. The auxiliary prediction layer may include multiple prediction channels. The number of channels in each auxiliary prediction layer can be determined according to the total number of prediction categories corresponding to the above-mentioned sample images. Specifically, in an auxiliary prediction layer, each channel corresponds to a prediction category. The prediction category can be determined according to the heatmap of the above-mentioned sample images. Gaussian prediction regions can be constructed on the corresponding prediction channels based on the above-mentioned fused feature maps, and the prediction region of the above-mentioned sample images can be determined based on the above-mentioned Gaussian prediction regions.

[0068] In one possible embodiment, after the sample image is processed by the feature extraction layer and the feature fusion layer to obtain the fused feature map, the fused feature map is input to the auxiliary prediction layer for prediction processing to obtain the Gaussian prediction region, and the prediction region of the sample image is determined based on the Gaussian prediction region.

[0069] 105. Based on the predicted bounding box, predicted region, and labeled bounding box, adjust the parameters of the target detection model to be trained, and obtain the trained target detection model after the parameter adjustment is completed.

[0070] In this embodiment of the invention, the above-mentioned bounding boxes are the bounding boxes corresponding to the sample images. Based on the above-mentioned predicted boxes and the above-mentioned bounding boxes, a first loss between the above-mentioned predicted boxes and the above-mentioned bounding boxes is determined. The above-mentioned bounding boxes are mapped to prediction channels of different scales to obtain the labeled regions corresponding to the bounding boxes. Based on the above-mentioned predicted regions and the above-mentioned labeled regions, a second loss between the above-mentioned predicted regions and the labeled regions is determined. Based on the above-mentioned first loss and the above-mentioned second loss, the total loss corresponding to the target detection model to be trained is determined. Based on the total loss corresponding to the target detection model to be trained, the parameters of the target detection model to be trained are adjusted to obtain the above-mentioned trained target detection model.

[0071] In one possible embodiment, after obtaining the predicted bounding box and the predicted region, a first loss corresponding to the predicted bounding box and the corresponding labeled bounding box is determined based on the predicted bounding box and the corresponding labeled bounding box. The labeled bounding box is mapped to a labeled region. Specifically, the center point of the labeled bounding box can be mapped to the center point of the labeled region. A Gaussian prediction region is constructed based on the center point to obtain the predicted region. A second loss corresponding to the predicted region and the labeled region is determined based on the predicted region and the labeled region. The total loss corresponding to the target detection model to be trained is determined based on the first loss and the second loss. Based on the total loss corresponding to the target detection model to be trained, the parameter adjustment process is iterated with minimizing the total loss as the optimization objective until the total loss converges at the minimum or the number of iterations reaches a preset value. Training is then stopped, and a trained target detection model is obtained.

[0072] Specifically, the target dataset can be input into the target detection model to be trained for processing. Feature extraction is performed through the feature extraction layer to obtain image features corresponding to the sample images. Feature fusion is then performed on these image features through the feature fusion layer to obtain fused feature maps. These fused feature maps are then input into the main prediction layer and the auxiliary prediction layer for main and auxiliary prediction processing, respectively, to obtain the prediction boxes corresponding to the main prediction layer and the prediction regions corresponding to the auxiliary prediction layer. Based on the prediction boxes and their corresponding labeled boxes, a first loss is determined, and the labeled boxes are mapped to labeled regions. Based on the predicted regions and their corresponding labeled regions, a second loss is determined. Based on the first and second losses, the total loss corresponding to the target detection model to be trained is determined. Based on the total loss corresponding to the target detection model to be trained, the parameter adjustment process is iterated with minimizing the total loss as the optimization objective until the total loss converges at its minimum or the number of iterations reaches a preset value. Training is then stopped, resulting in a trained target detection model.

[0073] In another possible embodiment, if the object to be detected is a face, then the object detection model can be a face detection model, the sample images corresponding to the target dataset can be face sample images, and the bounding boxes corresponding to the target dataset can be face bounding boxes. The face sample images and the face bounding boxes are input into the face detection model to be trained for processing. Feature extraction is performed through the feature extraction layer to obtain face image features corresponding to the face sample images. Feature fusion is then performed on the face image features through the feature fusion layer to obtain a face fusion feature map corresponding to the face image features. The face fusion feature map is then input into the main prediction layer and the auxiliary prediction layer for main prediction and auxiliary prediction processing, respectively, to obtain the above... The main prediction layer defines the face prediction bounding box and the auxiliary prediction layer defines the face prediction region. Based on the face prediction bounding box and the corresponding face annotation box, a first loss is determined. The face annotation box is mapped to a face annotation region. Based on the face prediction region and the corresponding face annotation region, a second loss is determined. Based on the first loss and the second loss, the total loss of the face detection model to be trained is determined. Based on the total loss of the face detection model to be trained, the parameter adjustment process is iterated with minimizing the total loss as the optimization objective until the total loss converges at the minimum or the number of iterations reaches a preset value. Training is then stopped, and a trained face detection model is obtained.

[0074] In this embodiment of the invention, a target detection model to be trained and a target dataset are obtained; sample images are processed through a feature extraction layer and a feature fusion layer to obtain a fused feature map of the sample images; the fused feature map is input into the main prediction layer for processing to obtain the predicted bounding boxes of the sample images; the fused feature map is input into the auxiliary prediction layer for processing to obtain the predicted regions of the sample images; based on the predicted bounding boxes, predicted regions, and labeled boxes, the parameters of the target detection model to be trained are adjusted, and a trained target detection model is obtained after the parameter adjustment is completed. The target image to be detected is processed through a feature extraction layer and a feature fusion layer to obtain a fused feature map of the target image. The fused feature map of the target image is then input into the main prediction layer and the auxiliary prediction layer for prediction processing. The auxiliary prediction layer assists the training of the main prediction layer, enabling the main prediction layer to better learn features, thereby improving the training speed and recognition accuracy of the target detection model.

[0075] Optionally, in the process of inputting the fused feature map into the main prediction layer for processing to obtain the predicted bounding box of the sample image, the fused feature map can be input into the main prediction layer for processing to obtain the label assignment result corresponding to the sample image, and the predicted bounding box of the sample image can be determined based on the label assignment result.

[0076] In this embodiment of the invention, the label allocation result can be obtained by performing positive and negative sample allocation on the fused feature map using a positive and negative sample allocation method. The prediction box of the sample image is determined based on the label allocation result. Specifically, the fused feature map can be input into the main prediction layer for positive and negative sample allocation processing to obtain the label allocation result corresponding to the sample image. The label allocation result can correspond to a fused feature map of one scale or to a fused feature map of multiple scales.

[0077] In one possible embodiment, after obtaining the target recognition model to be trained and the target dataset, the fused feature map is input into the main prediction layer. The fused feature map is then assigned positive and negative samples through the positive and negative sample assignment to obtain the label assignment result. The prediction box of the sample image is determined based on the label assignment result. The prediction box of the sample image can be one or more. The specific number of prediction boxes of the sample image can be determined based on the label assignment result. If the label assignment result corresponds to the fused feature map of one scale, the number of prediction boxes of the sample image is 1. If the label assignment result corresponds to the fused feature map of two scales, the number of prediction boxes of the sample image is 2.

[0078] Optionally, in the process of inputting the fused feature map into the main prediction layer for processing to obtain the label assignment result of the sample image, the fused feature map can be input into the main prediction layer for processing to obtain the first predicted feature of the sample image. Based on the label matching strategy and bounding box of the main prediction layer, the label assignment result of the sample image is determined in the first predicted feature of the sample image.

[0079] In this embodiment of the invention, the above-mentioned inputting the fused feature map into the main prediction layer for processing can be to input the fused feature map into the main prediction layer for linear transformation processing to obtain the first prediction feature of the sample image. The first prediction feature can include multiple anchor boxes. Each pixel in the fused feature map can include multiple anchor boxes to form a candidate box. The label matching strategy can be a positive and negative sample allocation method. The fused feature map is processed through linear transformation to obtain the candidate boxes. The labeled boxes are compared and matched with the candidate boxes. The candidate boxes that successfully match the labeled boxes are positive samples, and the candidate boxes that fail to match the labeled boxes are negative samples. The positive samples and negative samples are used as the label allocation results.

[0080] In one possible embodiment, the fused feature map can be a fused feature map of multiple scales. The fused feature map of multiple scales is input into the main prediction layer for linear transformation processing to obtain multiple anchor boxes corresponding to the fused feature map of multiple scales. Multiple candidate boxes are constructed based on the multiple anchor boxes. The multiple candidate boxes are compared and matched with the labeled boxes. The candidate boxes that successfully match the labeled boxes are taken as positive samples, and the candidate boxes that fail to match the labeled boxes are taken as negative samples. The positive samples and negative samples are taken as the label assignment results. Alternatively, the positive samples and negative samples can be re-labeled using a multi-class label assignment method.

[0081] Optionally, when processing the fused feature map into the auxiliary prediction layer to obtain the prediction region of the sample image, the fused feature map can be processed into the auxiliary prediction layer to obtain the second prediction feature of the sample image. The second prediction feature includes the category channel corresponding to the detection category. Based on the second prediction feature of the sample image, the prediction region of the sample image is determined.

[0082] In this embodiment of the invention, the fused feature map can be input into the auxiliary prediction layer for processing to obtain a heatmap corresponding to the fused feature map. The heatmap is used as the second prediction feature. The processing can be to construct the heatmap using methods such as kernel density estimation or point density estimation. The detection categories can be multiple or single, and each detection category corresponds to one category channel. If there are three detection categories, there are also three category channels. Each auxiliary prediction layer can include multiple category channels, and each category channel corresponds to one detection category. The number of each category channel is equal to the number of detection categories. The number of detection categories is the total number of categories that need to be detected in the sample image.

[0083] In one possible embodiment, after the fusion processing is performed through the feature fusion layer to obtain the fused feature map, the fused feature map is input into the auxiliary prediction layer. A heatmap corresponding to the fused feature map is constructed using the kernel density estimation method or the point density estimation method. The heatmap is used as the second prediction feature, and the positive and negative sample allocation method is performed based on the second prediction feature to determine the prediction region of the sample image.

[0084] Specifically, the aforementioned fused feature map may include fused feature maps at three scales. The aforementioned auxiliary prediction layers are the corresponding three auxiliary prediction layers. The aforementioned fused feature maps at the three scales are respectively input into the corresponding scale auxiliary prediction layers for processing to obtain three scale heatmaps. The aforementioned three scale heatmaps are respectively input into all category channels of the corresponding scale to construct Gaussian regions, and the aforementioned Gaussian regions are used as the aforementioned prediction regions.

[0085] Optionally, in adjusting the parameters of the target detection model to be trained based on the predicted bounding box, the predicted region, and the labeled bounding box, a first loss can be determined based on the predicted bounding box and the labeled bounding box, a second loss can be determined based on the predicted region and the labeled bounding box, a total loss can be determined based on the first loss and the second loss, and the parameters of the target detection model to be trained can be adjusted based on the total loss.

[0086] In this embodiment of the invention, the first loss may include bounding box loss, objness_loss, and cls_loss. The bounding box loss can be calculated using the CIOU_loss loss function, and the objness_loss and cls_loss can be calculated using the cross-entropy loss function. The objness_loss can be used to determine whether the detection category present in the labeled box is consistent with the detection category present in the predicted box. The second loss may be heatmap_loss, which can be calculated using guassion_focal_loss.

[0087] Specifically, the total loss mentioned above can be calculated using the following formula:

[0088] loss=main_loss+β*heatmap_loss

[0089] Wherein, the above loss is the total loss, the above main_loss is the first loss, the above heatmap_loss is the second loss, and the above β is the weight between the above first loss and the above second loss, and the above weight can be 0.1.

[0090] In one possible embodiment, after obtaining the predicted bounding box and the predicted region, the bounding box loss between the predicted bounding box and the labeled bounding box is calculated based on the bounding box loss, the detection confidence loss between the predicted bounding box and the labeled bounding box is calculated based on the objness_loss, the classification loss between the predicted bounding box and the labeled bounding box is calculated based on the cls_loss, the first loss is determined based on the bounding box loss, the detection confidence loss, and the classification loss, the Gaussian focus loss between the predicted region and the labeled bounding box is calculated based on the heatmap_loss, the second loss is determined, and the total loss is determined based on the first loss and the second loss. The parameters of the target detection model to be trained are adjusted iteratively based on the total loss until the total loss converges at the minimum or reaches a preset number of iterations, thus completing the training and obtaining the trained target detection model.

[0091] Optionally, in determining the second loss based on the predicted region and the bounding box, the bounding box can be mapped to the second predicted feature to obtain the labeled region of the sample image, and the second loss can be determined based on the predicted region and the labeled region.

[0092] In this embodiment of the invention, the labeled region may be the category region corresponding to the labeled box and the second predicted feature. The labeled box is mapped to the second predicted feature. The center point of the labeled region is determined based on the center point of the labeled box. The X offset and Y offset of the labeled region are determined based on the width and height values ​​of the labeled box. The labeled region is determined based on the center point, X offset, and Y offset. The labeled region determines the second loss based on the predicted region and the labeled region.

[0093] In one possible embodiment, after the predicted region is obtained, the bounding box is mapped to the second predicted feature corresponding to the predicted region to obtain the labeled region of the sample image, and the second loss is determined based on the Gaussian focus loss between the predicted region and the labeled region.

[0094] It should be noted that the target detection model training method provided in this embodiment of the invention can be applied to devices such as smartphones, computers, and servers that can perform target detection model training.

[0095] like Figure 3 As shown, Figure 3 This is a flowchart of a target detection method provided in an embodiment of the present invention. The target detection method includes the following steps:

[0096] 301. Obtain the image to be detected;

[0097] In this embodiment of the invention, the target detection method described above can be deployed to a server or server cluster. The server or server cluster can be any electronic device with image processing, image recognition, image storage, and data transmission capabilities. The image to be detected can be an image sequence that needs to be detected. The server can obtain the image to be detected through the data transmission function described above.

[0098] 302. Input the image to be detected into the trained object detection model for processing to obtain the object detection result of the image to be detected.

[0099] In this embodiment of the invention, the trained target detection model can be obtained by the target detection model training method, and the target detection result can be obtained by processing the target monitoring model.

[0100] Specifically, the structural diagram of the above-mentioned trained object detection model includes a feature extraction layer (Backbone), a feature fusion layer (Neck), and prediction layers (Heads). The prediction layers may include a main prediction layer, which includes prediction layer 1, prediction layer 2, and prediction layer 3.

[0101] In one possible embodiment, the image to be detected is preprocessed to obtain a preprocessed result. The preprocessing operation may include random scaling, random cropping, or other preprocessing operations. The preprocessed result is then input into the feature extraction layer for feature extraction at different scales. These different scales may be 2*, 4*, 8*, 16*, and 32* scales. The image features corresponding to the 2* scale can be extracted based on the image to be detected, the image features corresponding to the 4* scale can be extracted based on the image features corresponding to the 2* scale, the image features corresponding to the 8* scale can be extracted based on the image features corresponding to the 4* scale, and so on. The image features corresponding to the 16* scale can be extracted based on the image to be detected. The image features corresponding to the 8* scale are extracted, and the image features corresponding to the 32* scale can be extracted based on the image features corresponding to the 16* scale. The image features corresponding to the 8* scale, the 16* scale, and the 32* scale are respectively input into the scale corresponding to the feature fusion layer to obtain the fused feature map of the image to be detected. The fused feature map of the image to be detected includes the fused feature map corresponding to the 8* scale, the fused feature map corresponding to the 16* scale, and the fused feature map corresponding to the 32* scale. The fused feature map is respectively input into the prediction layer 1, prediction layer 2, and prediction layer 3 to obtain the prediction box corresponding to the main prediction layer. Based on the prediction box, the target detection result of the image to be detected is determined.

[0102] In another possible embodiment, if the image to be detected is a face image, then the target detection model can be a face detection model. The face image is preprocessed to obtain a preprocessed result. This preprocessing operation can be random scaling, random cropping, etc. The preprocessed result is then input to the feature extraction layer for feature extraction at different scales. These different scales can be 2*, 4*, 8*, 16*, and 32* scales. The face image features corresponding to the 2* scale can be extracted based on the face image, the face image features corresponding to the 4* scale can be extracted based on the face image features corresponding to the 2* scale, the face image features corresponding to the 8* scale can be extracted based on the face image features corresponding to the 4* scale, and so on. The face image features corresponding to the 6* scale can be extracted based on the face image features corresponding to the 8* scale, and the face image features corresponding to the 32* scale can be extracted based on the face image features corresponding to the 16* scale. The face image features corresponding to the 8*, 16*, and 32* scales are respectively input into the scales corresponding to the feature fusion layer to obtain the face fusion feature map of the face image. The face fusion feature map of the face image includes the face fusion feature map corresponding to the 8*, 16*, and 32* scales. The face fusion feature map is then input into prediction layer 1, prediction layer 2, and prediction layer 3 to obtain the face prediction bounding box corresponding to the main prediction layer. Based on the face prediction bounding box, the face detection result of the face image is determined.

[0103] In this embodiment of the invention, an image to be detected is acquired; the image to be detected is input into a trained target detection model for processing to obtain the target detection result of the image to be detected. By obtaining a trained target detection model through the above-described target detection model training method, and inputting the image to be detected into the trained target detection model for processing, the problem of efficient and high-accuracy target recognition of the image to be detected is solved.

[0104] It should be noted that the target detection method provided in this embodiment of the invention can be applied to devices such as smartphones, computers, and servers that are capable of target detection.

[0105] like Figure 4 As shown, this embodiment of the invention provides a target detection model training device, which includes:

[0106] The first acquisition module 401 is used to acquire the target detection model to be trained and the target dataset. The target dataset includes sample images and bounding boxes. The sample images include the target to be detected. The target detection model to be trained includes a feature extraction layer, a feature fusion layer and a prediction layer. The prediction layer includes a main prediction layer and an auxiliary prediction layer.

[0107] The first processing module 402 is used to process the sample image through the feature extraction layer and the feature fusion layer to obtain the fused feature map of the sample image;

[0108] The second processing module 403 is used to input the fused feature map into the main prediction layer for processing to obtain the prediction box of the sample image;

[0109] The third processing module 404 is used to input the fused feature map into the auxiliary prediction layer for processing to obtain the prediction region of the sample image;

[0110] The adjustment module 405 is used to adjust the parameters of the target detection model to be trained based on the prediction box, the prediction region and the annotation box, and obtain the trained target detection model after the parameter adjustment is completed. The trained target detection model includes a trained feature extraction layer, a trained feature fusion layer and a trained main prediction layer.

[0111] Optionally, the second processing module 403 includes:

[0112] The first processing submodule is used to input the fused feature map into the main prediction layer for processing to obtain the label assignment result corresponding to the sample image;

[0113] The first determining submodule is used to determine the prediction bounding box of the sample image based on the label allocation result.

[0114] Optionally, the first processing submodule includes:

[0115] The processing unit is used to input the fused feature map into the main prediction layer for processing to obtain the first prediction feature of the sample image;

[0116] The first determining unit is used to determine the label assignment result of the sample image in the first prediction feature of the sample image based on the label matching strategy of the main prediction layer and the annotation box.

[0117] Optionally, the third processing module 404 includes:

[0118] The second processing submodule is used to input the fused feature map into the auxiliary prediction layer for processing to obtain the second prediction feature of the sample image, wherein the second prediction feature includes the category channel corresponding to the detection category.

[0119] The second determining submodule is used to determine the prediction region of the sample image based on the second prediction feature of the sample image.

[0120] Optionally, the adjustment module 405 includes:

[0121] The third determination submodule is used to determine the first loss based on the prediction box and the annotation box;

[0122] The fourth determination submodule is used to determine the second loss based on the predicted region and the labeled box;

[0123] The fifth determination submodule is used to determine the total loss based on the first loss and the second loss; the adjustment submodule is used to adjust the parameters of the target detection model to be trained based on the total loss.

[0124] Optionally, the fourth determining submodule includes:

[0125] A mapping unit is used to map the bounding box to the second predicted feature to obtain the labeled region of the sample image;

[0126] The second determining unit is used to determine the second loss based on the predicted region and the labeled region.

[0127] It should be noted that the target detection model training device provided in this embodiment of the invention can be applied to devices such as smartphones, computers, and servers that can perform target detection model training.

[0128] The target detection model training apparatus provided in this embodiment of the invention can implement all the processes of the target detection model training method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, further details are omitted here.

[0129] like Figure 5 As shown, an embodiment of the present invention provides a target detection device, which includes:

[0130] The second acquisition module 501 is used to acquire the image to be detected;

[0131] The fourth processing module 502 inputs the image to be detected into the trained target detection model for processing to obtain the target detection result of the image to be detected. The trained target detection model is obtained according to the target detection model training method.

[0132] It should be noted that the target detection device provided in this embodiment of the invention can be applied to devices such as smartphones, computers, and servers that can perform target detection.

[0133] The target detection device provided in this embodiment of the invention can implement all the processes of the target detection model training method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, further details are omitted here.

[0134] See Figure 6 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 6 As shown, it includes: a memory 602, a processor 601, and a computer program or object detection method for training an object detection model stored in the memory 602 and capable of running on the processor 601, wherein:

[0135] Processor 601 is used to call the target detection model training computer program stored in memory 602 and execute the following steps:

[0136] Obtain the target detection model to be trained and the target dataset. The target dataset includes sample images and bounding boxes. The sample images include the target to be detected. The target detection model to be trained includes a feature extraction layer, a feature fusion layer and a prediction layer. The prediction layer includes a main prediction layer and an auxiliary prediction layer.

[0137] The sample image is processed by the feature extraction layer and the feature fusion layer to obtain the fused feature map of the sample image;

[0138] The fused feature map is input into the main prediction layer for processing to obtain the prediction bounding box of the sample image;

[0139] The fused feature map is input into the auxiliary prediction layer for processing to obtain the predicted region of the sample image;

[0140] Based on the prediction box, the prediction region, and the annotation box, the parameters of the target detection model to be trained are adjusted, and a trained target detection model is obtained after the parameter adjustment is completed. The trained target detection model includes a trained feature extraction layer, a trained feature fusion layer, and a trained main prediction layer.

[0141] Optionally, the step of processor 601 inputting the fused feature map into the main prediction layer for processing to obtain the prediction bounding box of the sample image includes:

[0142] The fused feature map is input into the main prediction layer for processing to obtain the label assignment result corresponding to the sample image;

[0143] Based on the label assignment results, the prediction bounding box of the sample image is determined.

[0144] Optionally, the main prediction layer executed by the processor 601 includes a label matching strategy, inputting the fused feature map into the main prediction layer for processing to obtain the label assignment result of the sample image, including:

[0145] The fused feature map is input into the main prediction layer for processing to obtain the first predicted feature of the sample image;

[0146] Based on the label matching strategy of the main prediction layer and the bounding box, the label assignment result of the sample image is determined in the first prediction feature of the sample image.

[0147] Optionally, the step of processor 601 inputting the fused feature map into the auxiliary prediction layer for processing to obtain the prediction region of the sample image includes:

[0148] The fused feature map is input into the auxiliary prediction layer for processing to obtain the second prediction feature of the sample image, the second prediction feature including the category channel corresponding to the detection category;

[0149] Based on the second prediction feature of the sample image, the prediction region of the sample image is determined.

[0150] Optionally, the parameter adjustment of the target detection model to be trained based on the predicted bounding box, the predicted region, and the labeled bounding box, performed by the processor 601, includes:

[0151] Based on the predicted bounding box and the labeled bounding box, a first loss is determined;

[0152] Based on the predicted region and the labeled box, a second loss is determined;

[0153] Based on the first loss and the second loss, determine the total loss;

[0154] The parameters of the target detection model to be trained are adjusted based on the total loss.

[0155] Optionally, the target detection model training method described in processor 601 is characterized in that determining the second loss based on the predicted region and the bounding box includes:

[0156] The bounding box is mapped to the second predicted feature to obtain the labeled region of the sample image;

[0157] Based on the predicted region and the labeled region, a second loss is determined.

[0158] Processor 601 is used to call the target detection computer program stored in memory 602 and execute the following steps:

[0159] The second acquisition module is used to acquire the image to be detected;

[0160] The fourth processing module is used to input the image to be detected into the trained target detection model for processing, and to obtain the target detection result of the image to be detected. The trained target detection model is obtained according to the target detection model training method.

[0161] It should be noted that the electronic device provided in the embodiments of the present invention can be applied to devices such as smartphones, computers, and servers that can perform object detection model training and generation methods.

[0162] The electronic device provided in this embodiment of the invention can implement all the processes of the target detection model training method or the target detection method implemented in the above method embodiments, and can achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0163] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the target detection model training method or target detection method of this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0164] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0165] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for training an object detection model, characterized in that, The method includes the following steps: Obtain the target detection model to be trained and the target dataset. The target dataset includes sample images and bounding boxes. The sample images include the target to be detected. The target detection model to be trained includes a feature extraction layer, a feature fusion layer and a prediction layer. The prediction layer includes a main prediction layer and an auxiliary prediction layer. The sample image is processed by the feature extraction layer and the feature fusion layer to obtain the fusion feature map of the sample image. The fusion feature map includes fusion feature maps of three scales, and the auxiliary prediction layer consists of three corresponding auxiliary prediction layers. The fused feature map is input into the main prediction layer for processing to obtain the prediction bounding box of the sample image; The fused feature map is input into the auxiliary prediction layer for processing to obtain the predicted region of the sample image; specifically, this includes: inputting the fused feature maps of the three scales into the auxiliary prediction layer of the corresponding scale for processing; constructing heatmaps of the three scales corresponding to the fused feature maps of the three scales using kernel density estimation or point density estimation methods; using the heatmaps of the three scales as the second predicted feature of the sample image; inputting the heatmaps of the three scales into all category channels of the corresponding scale to construct Gaussian regions; and using the Gaussian regions as the predicted regions. Based on the prediction box, the prediction region, and the annotation box, the parameters of the target detection model to be trained are adjusted, and a trained target detection model is obtained after the parameter adjustment is completed. The trained target detection model includes a trained feature extraction layer, a trained feature fusion layer, and a trained main prediction layer.

2. The target detection model training method as described in claim 1, characterized in that, The step of inputting the fused feature map into the main prediction layer for processing to obtain the prediction bounding box of the sample image includes: The fused feature map is input into the main prediction layer for processing to obtain the label assignment result corresponding to the sample image; Based on the label assignment results, the prediction bounding box of the sample image is determined.

3. The target detection model training method as described in claim 2, characterized in that, The main prediction layer includes a label matching strategy. The fused feature map is input into the main prediction layer for processing to obtain the label assignment result of the sample image, including: The fused feature map is input into the main prediction layer for processing to obtain the first predicted feature of the sample image; Based on the label matching strategy of the main prediction layer and the bounding box, the label assignment result of the sample image is determined in the first prediction feature of the sample image.

4. The target detection model training method as described in claim 1, characterized in that, The step of adjusting the parameters of the target detection model to be trained based on the predicted bounding box, the predicted region, and the labeled bounding box includes: Based on the predicted bounding box and the labeled bounding box, a first loss is determined; Based on the predicted region and the labeled box, a second loss is determined; Based on the first loss and the second loss, determine the total loss; The parameters of the target detection model to be trained are adjusted based on the total loss.

5. The target detection model training method as described in claim 4, characterized in that, The step of determining the second loss based on the predicted region and the labeled bounding box includes: The bounding box is mapped to the second predicted feature to obtain the labeled region of the sample image; Based on the predicted region and the labeled region, a second loss is determined.

6. A target detection method, characterized in that, The method includes the following steps: Acquire the image to be detected; The image to be detected is input into a trained target detection model for processing to obtain the target detection result of the image to be detected. The trained target detection model is obtained according to any one of the target detection model training methods in claims 1 to 5.

7. A target detection model training device, characterized in that, The target detection model training device includes: The first acquisition module is used to acquire the target detection model to be trained and the target dataset. The target dataset includes sample images and bounding boxes. The sample images include the target to be detected. The target detection model to be trained includes a feature extraction layer, a feature fusion layer and a prediction layer. The prediction layer includes a main prediction layer and an auxiliary prediction layer. The first processing module is used to process the sample image through the feature extraction layer and the feature fusion layer to obtain a fusion feature map of the sample image. The fusion feature map includes fusion feature maps of three scales, and the auxiliary prediction layer consists of three corresponding auxiliary prediction layers. The second processing module is used to input the fused feature map into the main prediction layer for processing to obtain the prediction box of the sample image; The third processing module is used to input the fused feature map into the auxiliary prediction layer for processing to obtain the predicted region of the sample image; specifically, it includes: inputting the fused feature maps of the three scales into the auxiliary prediction layer of the corresponding scale for processing; constructing heatmaps of the three scales corresponding to the fused feature maps of the three scales using kernel density estimation method or point density estimation method; using the heatmaps of the three scales as the second predicted feature of the sample image; inputting the heatmaps of the three scales into all category channels of the corresponding scale to construct Gaussian regions; and using the Gaussian regions as the predicted regions. The adjustment module is used to adjust the parameters of the target detection model to be trained based on the prediction box, the prediction region, and the annotation box, and to obtain a trained target detection model after the parameter adjustment is completed. The trained target detection model includes a trained feature extraction layer, a trained feature fusion layer, and a trained main prediction layer.

8. A target detection device, characterized in that, The target detection device includes: The second acquisition module is used to acquire the image to be detected; The fourth processing module is used to input the image to be detected into the trained target detection model for processing, and to obtain the target detection result of the image to be detected. The trained target detection model is obtained according to any one of the target detection model training methods in claims 1 to 5.

9. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the target detection model training method as described in any one of claims 1 to 5 or the steps in the target detection method as described in claim 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps in the target detection model training method as described in any one of claims 1 to 5 or the steps in the target detection method as described in claim 6.

Images