Model training method and device, target detection method and device, equipment and storage medium

Abstract

The invention provides a model training method, a target detection method, equipment and a storage medium, and the method comprises the steps: inputting a sample image into a target detection model, and outputting a target frame graph comprising a target object; inputting the target frame image into a recognition model, and obtaining a prediction classification result and a prediction mask of the target frame image; determining a loss function value of the identification model based on the prediction classification result and the prediction mask; and adjusting model parameters of the recognition model based on the loss function value, and continuing training until a preset training completion condition is met to obtain a trained recognition model. According to the embodiment of the invention, the loss function value of the recognition model is determined based on the prediction classification result and the prediction mask, so that the contour features with high discrimination between targets extracted by classification and the mask are simultaneously paid attention to in the model training process, and the recognition accuracy of the recognition model is improved.

Description

Technical Field

[0001] The present application belongs to the field of model training technology, and specifically relates to a model training method, device, equipment and storage medium. Background Art

[0002] Object detection is an important task in the field of computer vision. Its goal is to detect and identify the locations and categories of multiple objects in images or videos. Since object detection not only expands the capabilities of image processing, but also enables computers to understand and process visual information more intelligently, object detection has a wide range of applications in many fields.

[0003] However, in some small sample target detection and fine-grained recognition fields, due to the particularity of the application, it is difficult to obtain a lot of sample data, resulting in the inability to extract image features during the training phase of the target detection task or the extracted image features are relatively scarce, resulting in the target detection task ultimately identifying the target inaccurately. Summary of the invention

[0004] The present application proposes a model training method, device, equipment and storage medium, which can solve the technical problem that the target finally identified in the target detection task is not accurate enough.

[0005] The first embodiment of the present application proposes a model training method, including:

[0006] Input the sample image into the target detection model, and output the target bounding box map including the target object;

[0007] Inputting the target frame image into a recognition model to obtain a predicted classification result and a predicted mask of the target frame image;

[0008] Determining a loss function value of the recognition model based on the predicted classification result and the predicted mask;

[0009] The model parameters of the recognition model are adjusted based on the loss function value, and the training is continued until a preset training completion condition is met to obtain a trained recognition model.

[0010] The embodiment of the second aspect of the present application provides a model training device, including:

[0011] An input module, used for inputting a sample image into a target detection model and outputting a target bounding box map including a target object;

[0012] The input module is further used to input the target frame image into a recognition model to obtain a predicted classification result and a predicted mask of the target frame image;

[0013] A determination module, configured to determine a loss function value of the recognition model based on the predicted classification result and the predicted mask;

[0014] The adjustment module is used to adjust the model parameters of the recognition model based on the loss function value, and continue training until the preset training completion conditions are met to obtain a trained recognition model.

[0015] The third aspect of the present application provides a target detection method, including:

[0016] Input the image to be identified into the target detection model, and output the target bounding box map including the target object;

[0017] The target frame image is input into a trained recognition model, and the type and position information of the object to be identified in the image to be identified are output. The recognition model is trained using the model training method described in the first aspect.

[0018] The fourth aspect of the present application provides an object detection device, including:

[0019] An input module is used to input the image to be identified into the target detection model and output a target bounding box image including the target object;

[0020] The input module is used to input the target frame image into a trained recognition model and output the type and position information of the object to be identified in the image to be identified. The recognition model is trained using the model training method described in the first aspect.

[0021] An embodiment of the fifth aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect or the third aspect above.

[0022] The embodiment of the sixth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect or the third aspect above.

[0023] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0024] The present application proposes a model training method, a target detection method, a device and a storage medium, including: inputting a sample image into a target detection model, and outputting a target frame image including a target object; inputting the target frame image into a recognition model, and obtaining a predicted classification result and a predicted mask of the target frame image; determining the loss function value of the recognition model based on the predicted classification result and the predicted mask; adjusting the model parameters of the recognition model based on the loss function value, and continuing training until the preset training completion conditions are met to obtain a trained recognition model. The embodiment of the present application determines the loss function value of the recognition model based on the predicted classification result and the predicted mask, so as to pay attention to the highly discriminative contour features between the classification and mask extraction targets during the model training process, thereby improving the recognition accuracy of the recognition model.

[0025] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0027] Figure 1 A flow chart of a model training method provided by an embodiment of the present application is shown;

[0028] Figure 2 A structural schematic diagram of a recognition model provided by an embodiment of the present application is shown;

[0029] Figure 3 A schematic diagram of the structure of a mask output network provided by an embodiment of the present application is shown;

[0030] Figure 4 A schematic diagram of the structure of a CRB module provided in an embodiment of the present application is shown;

[0031] Figure 5 A schematic diagram of the structure of a preset downsampling unit provided in an embodiment of the present application is shown;

[0032] Figure 6 A schematic diagram of the structure of an attention mechanism module provided by an embodiment of the present application is shown;

[0033] Figure 7 A flow chart of a target detection method provided by an embodiment of the present application is shown;

[0034] Figure 8A schematic diagram of the structure of a model training device provided in one embodiment of the present application is shown;

[0035] Fig. 9 A schematic diagram showing the structure of a target detection device provided by an embodiment of the present application is shown.

[0036] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0037] Fig.11 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0039] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0040] The model training method of the present application can be executed by a computing device, which can be a server, such as one server, multiple servers, a server cluster, a cloud computing platform, etc. Optionally, the computing device can also be a terminal device, such as a mobile phone, a tablet computer, a game console, a portable computer, a desktop computer, an advertising machine, an all-in-one machine, etc. The present application does not limit the device type and number of computing devices.

[0041] The present application proposes a model training method, a target detection method, a device and a storage medium, including: inputting a sample image into a target detection model, and outputting a target frame image including a target object; inputting the target frame image into a recognition model, and obtaining a predicted classification result and a predicted mask of the target frame image; determining the loss function value of the recognition model based on the predicted classification result and the predicted mask; adjusting the model parameters of the recognition model based on the loss function value, and continuing training until the preset training completion conditions are met to obtain a trained recognition model. The embodiment of the present application determines the loss function value of the recognition model based on the predicted classification result and the predicted mask, so as to pay attention to the highly discriminative contour features between the classification and mask extraction targets during the model training process, thereby improving the recognition accuracy of the recognition model.

[0042] A model training method proposed according to an embodiment of the present application is described below in conjunction with the accompanying drawings.

[0043] See also Figure 1 , the method specifically comprises the following steps:

[0044] S101, input a sample image into a target detection model, and output a target bounding box map including a target object.

[0045] The sample image may be an image including a target object, and the sample image may be a sample image in a small sample field, and the small sample field may be a medical field, a military field, an aviation field, and the like.

[0046] The target detection model can determine the position and size of the target object in the sample image, and further determine the corresponding target border map based on the position and size of the target object, thereby outputting the target border map.

[0047] S102: Input the target border image into the recognition model to obtain the predicted classification result and the predicted mask of the target border image.

[0048] Among them, the recognition model is used to output the type of target object in the target bounding box image, that is, what the target object is specifically. However, in small sample fine-grained recognition tasks, it is difficult for the network to learn highly discriminative features between targets. The similarity between different models of targets of the same type is high, and the features used to accurately distinguish different models of targets are difficult to extract. It is difficult for the network to learn highly discriminative features from limited samples, and the classification accuracy is low. Therefore, the recognition model can be used to output the predicted classification results corresponding to the target bounding box image and the predicted mask of the target object, so that the recognition model can learn the highly discriminative contour features between targets and improve the accuracy of classification.

[0049] S103: Determine the loss function value of the recognition model based on the predicted classification result and the predicted mask.

[0050] The loss function value of the recognition model includes two parts, namely, a first loss function value corresponding to the predicted classification result and a second loss function value corresponding to the predicted mask. The first loss function value and the second loss function value can be assigned corresponding weights to obtain the loss function value of the recognition model.

[0051] S104: Adjust the model parameters of the recognition model based on the loss function value, and continue training until the preset training completion conditions are met to obtain a trained recognition model.

[0052] The present application proposes a model training method, a target detection method, a device and a storage medium, including: inputting a sample image into a target detection model, and outputting a target frame image including a target object; inputting the target frame image into a recognition model, and obtaining a predicted classification result and a predicted mask of the target frame image; determining the loss function value of the recognition model based on the predicted classification result and the predicted mask; adjusting the model parameters of the recognition model based on the loss function value, and continuing training until the preset training completion conditions are met to obtain a trained recognition model. The embodiment of the present application determines the loss function value of the recognition model based on the predicted classification result and the predicted mask, so as to pay attention to the highly discriminative contour features between the classification and mask extraction targets during the model training process, thereby improving the recognition accuracy of the recognition model.

[0053] In some embodiments, inputting the target border image into the recognition model to obtain the predicted classification result and the predicted mask of the target border image includes:

[0054] The feature extraction network of the recognition model is used to obtain the image features of the target border image;

[0055] The feature extraction network is used to input the image features into the classification output network and the mask output network respectively, so that the classification output network outputs the predicted classification result of the target border map and the mask output network outputs the predicted mask of the target border map.

[0056] In some embodiments, the recognition model includes a feature extraction network, a classification output network, and a mask output network. Mask loss is introduced during the training process to supervise the recognition model, so that the feature extraction network can extract features that meet both the classification task and the mask extraction task, thereby improving the feature extraction network's attention to the target area features, and learning the features related to the target and the contour, thereby improving the classification accuracy.

[0057] Specifically, the feature extraction network is used to extract image features of the target border map, the classification output network is used to output the classification result of the target object based on the image features, that is, what the target object is, and the mask output network is used to output the mask result of the target object based on the image features, that is, the outline of the target object.

[0058] In some embodiments, Figure 2 As shown, the recognition model includes a feature extraction network, a classification output network, and a mask output network.

[0059] Among them, the feature extraction network includes multiple downsampling units for extracting image features of different scales.

[0060] The classification output network includes a global average pooling (GAP) layer and a fully connected (FC) layer. The GAP layer is used to average all elements of each feature map to obtain a scalar output for each feature map. The result of this operation is to convert the entire feature map into a vector of fixed length, which represents the overall information of the feature map. The FC layer is used to integrate and classify the features extracted by the previous layers and finally output the prediction results.

[0061] The mask output network includes multiple upsampling units, which are used to restore the input image features, extract feature images of multiple scales, and output corresponding mask results.

[0062] like Figure 2 As shown, the feature extraction network includes 4 down-sampling units, and the mask output network includes 4 up-sampling units.

[0063] In some embodiments, the structure of the mask output network is as follows: Figure 3 As shown, the mask output network includes multiple upsampling units, transposed convolution layers and convolution layers, wherein the upsampling unit includes a transposed convolution layer and a channel bidirectional convolution cycle (Channel-wise Bi-directional Convolutional Recurrent, referred to as CRB) module, wherein the upsampling multiples of all transposed convolution layers in the mask output network are the same as the downsampling multiples of the input feature image.

[0064] The input of the mask output network is a feature map of size (C×(H / r)×(W / r)), where C is the number of channels of the feature map of this layer, H and W are the height and width of the image when input to the network, and r is the downsampling multiple of the feature map relative to the input image. The output of the mask guidance branch is the predicted mask image. The predicted mask height and width are the same as the input image height and width, the number of channels is 2, and the fraction of pixels in the image that are background pixels and target pixels is output respectively.

[0065] The input feature map is upsampled using transposed convolution. The height and width of the upsampled feature map are twice as large as the original ones, and the number of channels is reduced to half. A total of r transposed convolutions are used in the mask output network, so that the output mask has the same width and height as the input image.

[0066] In some embodiments, the structure of the CRB module is as follows: Figure 4As shown in the figure, the CBR module consists of a 3×3 convolutional layer, a BN layer, and a ReLU activation function. The upsampled feature maps are fused to enhance the network's expressiveness. It should be noted that the output of this module is the same size as the input feature map, and no additional downsampling is introduced. The feature map obtained by the last upsampling is subjected to a 1×1 convolution to obtain the output mask.

[0067] In some embodiments, the loss function value of the recognition model is determined based on the predicted classification result and the predicted mask, including: obtaining the label classification result and the label mask of the target bounding box image; determining the first loss function value of the recognition model based on the label classification result and the predicted classification result; determining the second loss function value of the recognition model based on the label mask and the predicted mask; determining the loss function value of the recognition model based on the first loss function value and the second loss function value.

[0068] After obtaining the target bounding box map, the label classification result and label mask of the target object can be obtained, and the first loss function value of the recognition model can be determined based on the label classification result and the predicted classification result; the second loss function value of the recognition model can be determined based on the label mask and the predicted mask, thereby realizing supervision of the training of the recognition model based on the label classification result and the label mask.

[0069] In some embodiments, the loss function value of the recognition model can be calculated by formula (1):

[0070] L=L cls +λL mask (1)

[0071] Among them, L is the loss function of the recognition model, L cls is the first loss function, calculated using cross entropy loss, L Mask is the second loss function, which is calculated using cross entropy loss, and λ is the weight of mask loss.

[0072] Among them, the first loss function is shown in formula (2):

[0073]

[0074] Among them, m is the number of samples, n is the number of categories, The i-th sample is the label of the k-th category, The probability of predicting the kth category for the i-th sample.

[0075] The second loss function is shown in formula (3):

[0076]

[0077] Where m is the number of samples, H and W are the height and width of the image, is the label of the pixel at position (u,v) in the i-th sample as background or target, is the probability that the pixel at position (u, v) in the i-th sample is the background or target.

[0078] In some embodiments, a sample image is input into a target detection model, and a target bounding box map of a target object is output, including: inputting the sample image into a backbone network of the target detection model, and outputting first image features of different scales; using the backbone network to input the first image features with the smallest scale into a weighted fusion network, and outputting second image features of different scales; using the weighted fusion network to input second image features of different scales into a detection layer of the target detection model, and outputting a target bounding box map of the target object.

[0079] In some embodiments, the target detection model includes a backbone network, a weighted fusion network and a detection layer, wherein the backbone network includes multiple downsampling units to extract first image features of different scales, and the weighted fusion network further performs an upsampling operation on the first image features with the smallest scale to obtain second image features of different scales. The weighted fusion network is used to input the second image features of different scales into the detection layer of the target detection model, and the target bounding box map of the target object is output.

[0080] In some embodiments, a sample image is input into a backbone network of a target detection model, and a first image feature is output, including:

[0081] The sample image is input into the backbone network of the target detection model. During the process of downsampling the sample image using multiple downsampling units of the backbone network, an attention mechanism is introduced into the preset downsampling units to obtain the first image features of different scales.

[0082] In some embodiments, the backbone network includes a plurality of downsampling units, each of which is used to perform a downsampling operation on a corresponding output according to a corresponding downsampling multiple to obtain first image features of different scales.

[0083] However, when the task is heavy with small samples, it is difficult for the backbone network to extract the features of the target area. Therefore, the EMA attention mechanism can be introduced into the backbone network to make the backbone network pay more attention to the features of the target area in the image.

[0084] In order to simplify the structure of the backbone network and make the backbone network pay more attention to the characteristics of the target area in the image, an attention mechanism can be introduced in the preset downsampling unit.

[0085] Among them, the downsampling unit generally includes two branches, namely the convolution branch and the Bottleneck branch. Both branches use 1×1 convolution for channel dimension reduction. The Bottleneck branch introduces a residual structure, which can avoid the problem of gradient disappearance while increasing the network depth. Finally, the output results of the two branches are spliced ​​in the channel dimension, the original features and the extracted deep features are fused, and the final output is obtained through a convolution.

[0086] In order to improve the network's ability to extract features of smaller target areas, the EMA attention mechanism is introduced in the preset downsampling unit containing rich local information. In order to avoid increasing too much computation, the EMA module is only introduced in the convolution branch. The structure of the preset downsampling unit is as follows: Figure 5 The EMA attention mechanism is used to increase the feature weight of the target area in the original features, and the characteristics of the preset downsampling unit cross-stage connection are used to improve the accuracy of the final output features of the module.

[0087] In some embodiments, the attention mechanism module structure is as follows Figure 6 As shown in the figure, the EMA attention mechanism module is improved based on the Coordinate Attention (CA) module. In order to preserve the information on each channel of the feature map and reduce the computational cost, the EMA module groups the input feature map channels evenly on the basis of the CA module to obtain G sub-features. In order to aggregate multi-scale spatial structural information, the module sets three branches to learn attention weights. Among them, the first two branches decompose the global information into one-dimensional features in the X and Y directions for encoding, and use the shared 1×1 convolution to obtain new features. The features in the two directions are respectively subjected to the Sigmoid activation function to obtain the final weights, and are weighted in combination with the grouping features of the original input. The third branch uses 3×3 convolution to further extract multi-scale features. Finally, the outputs of the two parallel branches of pooling and convolution are fused using the cross-space learning structure. By transferring information between features of different scales, the module can use the correlation between features of different scales to enhance the consistency and richness of feature representation. While automatically learning the weights of each channel, the three branches accurately embed spatial information into the channel, which is conducive to improving the detection accuracy of the network.

[0088] In some embodiments, the first image feature with the smallest scale is input into the weighted fusion network using the backbone network to output the second image feature, including:

[0089] In the process of performing an upsampling operation on the first image feature with the smallest scale using the first fusion network of the weighted fusion network, the third image features obtained by each upsampling operation are fused with the first image features with the same scale to obtain fourth image features of different scales; in the process of performing a downsampling operation on the fourth image feature with the smallest scale using the second fusion network of the weighted fusion network, the fifth image features obtained by each downsampling operation are fused with the first image features and the fourth image features with the same scale to obtain second image features of different scales.

[0090] The weighted fusion network includes a first fusion network and a second fusion network. The network structures of the first fusion network and the second fusion network can be top-down feature pyramid structures (Feature Pyramid Networks, FPN for short) and bottom-up (Path Aggregation Network, PAN for short) structures, respectively. A jump connection is added between the backbone network and the second fusion network. The jump connection can be added between at least one network layer in the backbone network and the second fusion network. The first fusion network includes multiple upsampling network layers, and the second fusion network includes multiple downsampling layers. Avoid losing important features of the target when the number of network layers is deep. When performing feature fusion, set learnable weights to weight feature maps from different paths so that the network can adaptively learn more important features from small samples and improve detection accuracy.

[0091] Specifically, in the process of upsampling the first image feature with the smallest scale by the first fusion network, the third image feature obtained by each upsampling operation is fused with the first image feature with the same scale to obtain a fourth image feature of a different scale. In the process of downsampling the fourth image feature with the smallest scale by the second fusion network, the fifth image feature obtained by each downsampling operation is fused with the first image feature and the fourth image feature with the same scale to obtain a second image feature of a different scale.

[0092] It should be noted that only when there is a jump connection between the backbone network and the second fusion network, the second fusion network can fuse the corresponding first image feature with the fourth image feature and the fifth image feature of the same scale.

[0093] In some embodiments, before inputting the sample image into the target detection model and outputting the target bounding box map including the target object, the method further includes:

[0094] Get the training set corresponding to the target object;

[0095] The multiple original images in the training set are scaled according to preset ratios to obtain sample images corresponding to the multiple original images.

[0096] In the small sample target detection task, due to the small number of training and verification samples, the image color, target position and scale are relatively simple, which is prone to overfitting. Data enhancement can improve the diversity of samples through optical transformation, geometric transformation, data set expansion and other methods without increasing the amount of network calculation, and add a large number of rich features to network training. The YOLOv5s network uses a variety of data enhancement methods, such as HSV data enhancement and Mosaic data enhancement methods. HSV data enhancement refers to randomly adjusting the hue, saturation and brightness of the input image within a certain range to simulate different environmental effects such as lighting and background, and alleviate the overfitting problem caused by the single color of the training sample. The Mosaic data enhancement method randomly reads four images, randomly crops them, and splices them with a certain point in the output image as the center. The Mosaic data enhancement method can combine the targets in the image to alleviate the overfitting problem caused by the single target in the image.

[0097] Both HSV data augmentation and Mosaic data augmentation are random augmentation methods during the training process. In the case of small samples, relying solely on random changes during the training process will make the training unstable. Therefore, this paper takes the additional approach of data set expansion to scale the images in the training set, increasing the number of training images while alleviating the problem of a single target scale. The images in the training set are scaled proportionally, pixels are added around them to keep them consistent with the original image size, and they are added to the training set to expand the number of samples in the training set.

[0098] A target detection method proposed according to an embodiment of the present application is described below in conjunction with the accompanying drawings.

[0099] See also Figure 7 , the method specifically comprises the following steps:

[0100] S701: Input the image to be recognized into the target detection model, and output a target bounding box image including the target object.

[0101] S702: Input the target frame image into the trained recognition model, and output the type and position information of the object to be recognized in the image to be recognized.

[0102] The recognition model is trained using the above model training method.

[0103] The present application also provides a model training device, which is used to execute the model training method provided in any of the above embodiments. Figure 8 As shown, the device includes: an input module 801, a determination module 802 and an adjustment module 803.

[0104] An input module 801 is used to input a sample image into a target detection model and output a target bounding box image including a target object;

[0105] The input module 801 is further used to input the target frame image into a recognition model to obtain a predicted classification result and a predicted mask of the target frame image;

[0106] A determination module 802, configured to determine a loss function value of the recognition model based on the predicted classification result and the predicted mask;

[0107] The adjustment module 803 is used to adjust the model parameters of the recognition model based on the loss function value, and continue training until the preset training completion condition is met to obtain a trained recognition model.

[0108] The embodiment of the present application provides a model training device, including: inputting a sample image into a target detection model, outputting a target frame image including a target object; inputting the target frame image into a recognition model, obtaining a predicted classification result and a predicted mask of the target frame image; determining a loss function value of the recognition model based on the predicted classification result and the predicted mask; adjusting the model parameters of the recognition model based on the loss function value, and continuing training until a preset training completion condition is met to obtain a trained recognition model. The embodiment of the present application determines the loss function value of the recognition model based on the predicted classification result and the predicted mask, so that in the process of model training, attention is paid to the highly discriminative contour features between the classification and mask extraction targets, thereby improving the recognition accuracy of the recognition model.

[0109] In some embodiments, the input module 801 is specifically used for:

[0110] Acquire image features of the target frame image using a feature extraction network of a recognition model;

[0111] The image features are respectively input into a classification output network and a mask output network by using the feature extraction network, so that the classification output network outputs a predicted classification result of the target border image and the mask output network outputs a predicted mask of the target border image.

[0112] In some embodiments, the determination module 802 is specifically configured to:

[0113] Obtaining a label classification result and a label mask of the target border image;

[0114] Determine a first loss function value of the recognition model based on the label classification result and the predicted classification result;

[0115] Determining a second loss function value of the recognition model based on the label mask and the prediction mask;

[0116] A loss function value of the recognition model is determined based on the first loss function value and the second loss function value.

[0117] In some embodiments, the input module 801 is further specifically used for:

[0118] Inputting the sample image into the backbone network of the target detection model, and outputting first image features of different scales;

[0119] Using the backbone network, the first image feature with the smallest scale is input into a weighted fusion network, and second image features with different scales are output;

[0120] The second image features of different scales are input into the detection layer of the target detection model by using the weighted fusion network, and a target bounding box map of the target object is output.

[0121] In some embodiments, the input module 801 is further specifically used for:

[0122] The sample image is input into the backbone network of the target detection model. During the process of downsampling the sample image using multiple downsampling units of the backbone network, an attention mechanism is introduced into the preset downsampling units to obtain first image features of different scales.

[0123] In some embodiments, the input module 801 is further specifically configured to:

[0124] In the process of performing an upsampling operation on the first image feature with the smallest scale by using the first fusion network of the weighted fusion network, a third image feature obtained by each upsampling operation is fused with the first image feature with the same scale to obtain a fourth image feature with a different scale;

[0125] In the process of performing a downsampling operation on the fourth image feature with the smallest scale using the second fusion network of the weighted fusion network, the fifth image feature obtained by each downsampling operation is fused with the first image feature and the fourth image feature of the same scale to obtain a second image feature of a different scale.

[0126] In some embodiments, the above apparatus further includes an acquisition module and a scaling module:

[0127] The acquisition module is used to obtain the training set corresponding to the target object;

[0128] The scaling module is used to scale the multiple original images in the training set according to preset ratios to obtain sample images corresponding to the multiple original images.

[0129] The present application also provides a target detection device, which is used to execute the target detection method provided in any of the above embodiments. Fig. 9 As shown, the device includes: an input module 901, a determination module 902 and an adjustment module 903.

[0130] An input module 901 is used to input an image to be identified into a target detection model and output a target frame map including a target object;

[0131] The input module 901 is used to input the target frame image into a trained recognition model and output the type and location information of the object to be recognized in the image to be recognized. The recognition model is based on Figure 1 The model is trained using the model training method.

[0132] The present application also provides an electronic device to perform the above model training method. Fig.10 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. Fig.10 As shown, the electronic device 7 includes: a processor 700, a memory 701, a bus 702 and a communication interface 703, and the processor 700, the communication interface 703 and the memory 701 are connected via the bus 702; the memory 701 stores a computer program that can be run on the processor 700, and when the processor 700 runs the computer program, it executes the model training method or target detection method provided in any of the aforementioned embodiments of the present application.

[0133] The memory 701 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 703 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0134] The bus 702 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. Among them, the memory 701 is used to store programs, and the processor 700 executes the program after receiving the execution instruction. The model training method disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 700, or implemented by the processor 700.

[0135] The processor 700 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 700. The above processor 700 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 701, and the processor 700 reads the information in the memory 701 and completes the steps of the above method in combination with its hardware.

[0136] The electronic device provided in the embodiments of the present application and the model training method or target detection provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0137] The present application also provides a computer-readable storage medium corresponding to the model training method provided in the above embodiment. Fig.11 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, it executes the model training method or target detection method provided by any of the aforementioned embodiments.

[0138] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0139] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the model training method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0140] It should be noted that:

[0141] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0142] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0143] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: include: Input the sample image into the target detection model, and output the target bounding box map including the target object; Inputting the target frame image into a recognition model to obtain a predicted classification result and a predicted mask of the target frame image; Determining a loss function value of the recognition model based on the predicted classification result and the predicted mask; The model parameters of the recognition model are adjusted based on the loss function value, and the training is continued until a preset training completion condition is met to obtain a trained recognition model.

2. The method according to claim 1, characterized in that The step of inputting the target frame image into a recognition model and obtaining a prediction classification result and a prediction mask of the target frame image includes: Acquire image features of the target frame image using a feature extraction network of a recognition model; The image features are respectively input into a classification output network and a mask output network by using the feature extraction network, so that the classification output network outputs a predicted classification result of the target border image and the mask output network outputs a predicted mask of the target border image.

3. The method according to claim 1, characterized in that: Determining a loss function value of the recognition model based on the predicted classification result and the predicted mask includes: Obtaining a label classification result and a label mask of the target border image; Determine a first loss function value of the recognition model based on the label classification result and the predicted classification result; Determining a second loss function value of the recognition model based on the label mask and the prediction mask; A loss function value of the recognition model is determined based on the first loss function value and the second loss function value.

4. The method according to claim 1, characterized in that: The step of inputting a sample image into a target detection model and outputting a target bounding box map including a target object comprises: Inputting the sample image into the backbone network of the target detection model, and outputting first image features of different scales; Using the backbone network, the first image feature with the smallest scale is input into a weighted fusion network, and second image features with different scales are output; The second image features of different scales are input into the detection layer of the target detection model by using the weighted fusion network, and a target bounding box map of the target object is output.

5. The method according to claim 4, characterized in that The step of inputting the sample image into the backbone network of the target detection model and outputting the first image feature comprises: The sample image is input into the backbone network of the target detection model. During the process of downsampling the sample image using multiple downsampling units of the backbone network, an attention mechanism is introduced into the preset downsampling units to obtain first image features of different scales.

6. The method according to claim 4, characterized in that The step of using the backbone network to input the first image feature with the smallest scale into a weighted fusion network and outputting a second image feature comprises: In the process of performing an upsampling operation on the first image feature with the smallest scale by using the first fusion network of the weighted fusion network, a third image feature obtained by each upsampling operation is fused with the first image feature with the same scale to obtain a fourth image feature with a different scale; In the process of performing a downsampling operation on the fourth image feature with the smallest scale using the second fusion network of the weighted fusion network, the fifth image feature obtained by each downsampling operation is fused with the first image feature and the fourth image feature of the same scale to obtain a second image feature of a different scale.

7. The method according to claim 1, characterized in that Before inputting the sample image into the target detection model and outputting the target bounding box map including the target object, the method further includes: Get the training set corresponding to the target object; The multiple original images in the training set are scaled according to preset ratios to obtain sample images corresponding to the multiple original images.

8. A target detection method, characterized in that: include: Input the image to be identified into the target detection model, and output the target bounding box map including the target object; The target frame image is input into a trained recognition model, and the type and position information of the object to be identified in the image to be identified are output. The recognition model is trained using the model training method described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 8.

Images