Model training method, target detection method and device, equipment and product

Abstract

The invention relates to a model training method, a target detection method and device, equipment and a product, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a training image set, and performing feature extraction processing on a plurality of training images through an initial adapter network to obtain a first image feature; obtaining an initial target detection model; performing feature extraction processing on the plurality of training images by the head detection model to obtain second image features; determining a feature loss value based on the first image feature and the second image feature, and training the initial adapter network to obtain an adapter network; and constructing a target detection model according to the adapter network and the main body detection model. According to the invention, the adapter conversion decoding features are used as the input of the target detection model, a low-complexity adapter network can be trained at a low cost, a large-parameter target detection model does not need to be trained again, and the video memory occupation in the training process can be greatly reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a model training method, a target detection method, a model training device, a target detection device, an electronic device, and a computer program product. Background Art

[0002] At present, the goal of image coding based on machine learning is to further improve the compression rate of images. Its compressed feature representation can be decoded into specific pixel values ​​for human viewing, or directly used as input for image processing or computer vision models based on neural networks. Common machine vision compression algorithms include the Artificial Intelligence Image Compression Optimization (JPEG AI) algorithm and the domestic Data Coding for Machine Intelligence (DCM) algorithm.

[0003] At present, the target detection algorithm based on transformer not only surpasses the traditional faster region-based convolutional neural network (Faster Region-based Convolutional Neural Network, Faster R-CNN) model in large target detection, but also shows great advantages in small target detection and specific application scenarios. At the same time, the remote sensing image rotation target detection technology based on transformer has made significant progress, especially in model structure optimization, feature extraction strategy, and multi-scale and multi-directional target detection.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a model training method, a target detection method, a model training device, a target detection device, an electronic device, a computer-readable storage medium and a computer program product, thereby at least to a certain extent overcoming the problem that the relevant target detection model adopts a decoding feature-image reconstruction method and a network with an upsampling operator, resulting in the target detection process occupying high memory and high computing power.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present invention.

[0007] According to a first aspect of the present disclosure, a model training method is provided, comprising: obtaining a training image set and a pre-constructed initial adapter network, performing feature extraction processing on a plurality of training images in the training image set by means of the initial adapter network to obtain a first image feature; obtaining a pre-trained initial target detection model, wherein the initial target detection model comprises a head detection model and a body detection model; performing feature extraction processing on the plurality of training images by means of the head detection model to obtain a second image feature; determining a feature loss value based on the first image feature and the second image feature, and training the initial adapter network based on the feature loss value to obtain an adapter network; and constructing a target detection model according to the adapter network and the body detection model.

[0008] In an exemplary embodiment of the present disclosure, the performing feature extraction processing on the multiple training images in the training image set by the initial adapter network to obtain the first image feature includes: performing image enhancement processing on the multiple training images to obtain enhanced training images; acquiring an image codec, performing encoding and decoding processing on the enhanced training images by the image codec to obtain training image decoding features; and performing feature extraction processing on the training image decoding features by the initial adapter network to obtain the first image feature.

[0009] In an exemplary embodiment of the present disclosure, encoding and decoding the enhanced training image by the image codec to obtain the training image decoding features includes: performing image padding processing on the enhanced training image to obtain a padded training image; obtaining a downsampling multiple corresponding to the image codec; performing image compression processing on the padded training image based on the downsampling multiple to obtain the training image decoding features, wherein the decoding feature dimension of the training image decoding features is the same as the input feature dimension of the initial adapter network.

[0010] In an exemplary embodiment of the present disclosure, the head detection model performs feature extraction processing on the multiple training images to obtain a second image feature, including: obtaining a downsampling multiple corresponding to an image codec, and determining a model segmentation point of the initial target detection model based on the downsampling multiple; based on the model segmentation point, determining a head detection model and a body detection model of the initial target detection model; performing feature extraction processing on the training images through the head detection model to obtain a second image feature, wherein the image feature dimension of the second image feature is the same as the output feature dimension of the initial adapter network.

[0011] In an exemplary embodiment of the present disclosure, constructing a target detection model based on the adapter network and the subject detection model includes: obtaining an image codec; and serially processing the image codec, the adapter network, and the subject detection model to obtain the target detection model.

[0012] According to a second aspect of the present disclosure, a target detection method is provided, comprising: obtaining an image to be identified; obtaining a pre-trained target detection model, wherein the target detection model is trained based on the above-mentioned model training method, and the target detection model comprises an adapter network and a subject detection model; performing encoding and decoding processing on the image to be identified to obtain image decoding features; inputting the image decoding features into the adapter network to obtain adapter features; inputting the adapter features into the subject detection model to obtain a target detection category corresponding to the image to be identified.

[0013] According to a third aspect of the present disclosure, a model training device is provided, comprising: a first feature extraction module, used to obtain a training image set and a pre-constructed initial adapter network, and perform feature extraction processing on multiple training images in the training image set through the initial adapter network to obtain a first image feature; an initial model acquisition module, used to obtain a pre-trained initial target detection model, the initial target detection model includes a head detection model and a body detection model; a second feature extraction module, used to perform feature extraction processing on the multiple training images through the head detection model to obtain a second image feature; a network training module, used to determine a feature loss value based on the first image feature and the second image feature, and train the initial adapter network based on the feature loss value to obtain an adapter network; a detection model construction module, used to construct a target detection model based on the adapter network and the body detection model.

[0014] In an exemplary embodiment of the present disclosure, the first feature extraction module includes a first feature extraction unit, which is used to: perform image enhancement processing on the multiple training images to obtain enhanced training images; obtain an image codec, and perform encoding and decoding processing on the enhanced training images by the image codec to obtain training image decoding features; and perform feature extraction processing on the training image decoding features by the initial adapter network to obtain the first image features.

[0015] In an exemplary embodiment of the present disclosure, the first feature extraction unit includes a decoding feature extraction subunit, which is used to: perform image padding processing on the enhanced training image to obtain a padded training image; obtain a downsampling multiple corresponding to the image codec; perform image compression processing on the padded training image based on the downsampling multiple to obtain a decoding feature of the training image, wherein the decoding feature dimension of the decoding feature of the training image is the same as the input feature dimension of the initial adapter network.

[0016] In an exemplary embodiment of the present disclosure, the second feature extraction module includes a second feature extraction unit, which is used to: obtain a downsampling multiple corresponding to the image codec, and determine a model segmentation point of the initial target detection model based on the downsampling multiple; determine a head detection model and a body detection model of the initial target detection model based on the model segmentation point; perform feature extraction processing on the training image through the head detection model to obtain a second image feature, and the image feature dimension of the second image feature is the same as the output feature dimension of the initial adapter network.

[0017] In an exemplary embodiment of the present disclosure, the detection model construction module includes a detection model construction unit, which is used to: obtain an image codec; and perform serial processing on the image codec, the adapter network and the subject detection model to obtain the target detection model.

[0018] According to a fourth aspect of the present disclosure, a target detection device is provided, comprising: an image acquisition module, used to acquire an image to be identified; a model acquisition module, used to acquire a pre-trained target detection model, wherein the target detection model is trained based on the above-mentioned model training method, and the target detection model comprises an adapter network and a subject detection model; a decoding feature determination module, used to perform encoding and decoding processing on the image to be identified to obtain image decoding features; an adapter feature determination module, used to input the image decoding features into the adapter network to obtain adapter features; a detection and identification module, used to input the adapter features into the subject detection model to obtain a target detection category corresponding to the image to be identified.

[0019] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the model training method according to any one of the above items or the target detection method described above is implemented.

[0020] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the model training method according to any one of the above items or the target detection method described above is implemented.

[0021] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements any one of the above-mentioned model training methods or the above-mentioned target detection methods.

[0022] The technical solution provided by the present disclosure may have the following beneficial effects:

[0023] The model training method in the exemplary embodiment of the present disclosure, on the one hand, directly uses the decoding features of the image codec as the input of the target detection part model, and can train a low-complexity adapter network at a very small cost, without the need to retrain the target detection model with a large number of parameters and excellent performance, which can avoid the problems of high memory and high computing power in the training process; on the other hand, the adapter network is trained using the feature matching training method, avoiding the joint training of the adapter network with the subject detection model part, which can greatly reduce the video memory occupancy in the training process.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0026] Figure 1 A flowchart of a model training method according to an exemplary embodiment of the present disclosure is schematically shown;

[0027] Figure 2 The overall schematic diagram of the target detection system based on decoding features according to the exemplary embodiment of the present disclosure is schematically shown;

[0028] Figure 3 Schematically shows a simple adapter network basic unit structure diagram according to an exemplary embodiment of the present disclosure;

[0029] Figure 4 A flowchart of training an adapter based on feature matching according to an exemplary embodiment of the present disclosure is schematically shown;

[0030] Figure 5 A flowchart of a target detection method according to an exemplary embodiment of the present disclosure is schematically shown;

[0031] Figure 6 A system architecture design diagram of a target detection method according to an exemplary embodiment of the present disclosure is schematically shown;

[0032] Figure 7 Schematically shows a structural diagram of a self-designed adapter network according to an exemplary embodiment of the present disclosure;

[0033] Figure 8 A first recognition effect diagram of a target detection model for an image according to an exemplary embodiment of the present disclosure is schematically shown;

[0034] Fig. 9 A second recognition effect diagram of a target detection model for an image according to an exemplary embodiment of the present disclosure is schematically shown;

[0035] Fig.10 A block diagram of a model training device according to an exemplary embodiment of the present disclosure is schematically shown;

[0036] Fig.11 A block diagram of a target detection device according to an exemplary embodiment of the present disclosure is schematically shown;

[0037] Fig.12 A schematic diagram schematically illustrates a computer-readable storage medium according to an exemplary embodiment of the present disclosure;

[0038] Fig.13 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0039] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0040] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations, materials or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0042] Future research on Transformer-based object detection algorithms will continue to explore how to further improve the performance and efficiency of Transformer in object detection. However, high model complexity, large computing resource requirements, scale change and occlusion processing, and sparse domain feature extraction are still technical challenges that need to be solved. At present, open source large models are becoming more and more popular. It is very cost-effective to make full use of open source, pre-trained large models, rather than retraining new models for your own task scenarios.

[0043] In a related scheme, when the input of the original target detection model is an image, if a decoding feature-image reconstruction method is used, a network with an upsampling operator is usually used, such as a deep learning model for image super-resolution (Enhanced Deep Super-Resolution, EDSR). This process takes up high memory and computing power.

[0044] Based on this, in this example embodiment, a model training method is first provided. The model training method of the present disclosure can be implemented using a server, or the method described in the present disclosure can be implemented using a terminal device, wherein the terminal described in the present disclosure may include mobile terminals such as mobile phones, tablet computers, laptops, PDAs, personal digital assistants (PDAs), and fixed terminals such as desktop computers. Figure 1 The following is a schematic diagram of the model training method flow according to some embodiments of the present disclosure. Figure 1 , the model training method may include the following steps:

[0045] Step S110, obtaining a training image set and a pre-built initial adapter network, performing feature extraction processing on a plurality of training images in the training image set through the initial adapter network, and obtaining a first image feature.

[0046] Step S120, obtaining a pre-trained initial target detection model, where the initial target detection model includes a head detection model and a body detection model.

[0047] Step S130: The head detection model performs feature extraction processing on the multiple training images to obtain second image features.

[0048] Step S140, determining a feature loss value based on the first image feature and the second image feature, and training the initial adapter network based on the feature loss value to obtain an adapter network.

[0049] Step S150, constructing a target detection model according to the adapter network and the subject detection model.

[0050] According to the model training method in this example embodiment, on the one hand, by directly using the decoded features of the image codec as the input of the target detection part of the model, a low-complexity adapter network can be trained at a very low cost, and there is no need to retrain the target detection model with a large number of parameters and excellent performance, which can avoid the problems of high memory and high computing power in the training process; on the other hand, the adapter network is trained using the feature matching training method to avoid jointly training the adapter network with the subject detection model part, which can greatly reduce the video memory usage in the training process.

[0051] The model training method in this example embodiment will be further described below.

[0052] Step S110, obtaining a training image set and a pre-built initial adapter network, performing feature extraction processing on a plurality of training images in the training image set through the initial adapter network, and obtaining a first image feature.

[0053] According to some exemplary embodiments of the present disclosure, the training image set may be an image set consisting of a plurality of training images. The initial adapter network may be a model structure of a pre-built adapter. The first image feature may be an image feature obtained by extracting features from the plurality of training images by the initial adapter network.

[0054] This disclosure mainly includes three parts: Figure 2 , Figure 2 The overall schematic diagram of the construction of a target detection system based on decoding features according to an exemplary embodiment of the present disclosure is schematically shown. The construction of the target detection model mainly includes three parts: the construction of the adapter network, the adapter training, and the construction of the Transformer pre-trained target detection system based on decoding features. Among them, the construction part of the adapter network is mainly used to build the model structure of the adapter; the adapter training part is mainly used to set the basic general training method of the adapter network; the construction part of the Transformer pre-trained target detection system based on decoding features is mainly used to connect the trained network structures in series to obtain the target detection model.

[0055] The construction of the adapter network mainly involves the following contents: First, obtain a pre-generated training image set, and use the training image set containing multiple training images as the training data of the model. For example, the training image set disclosed in the present invention can be a dataset for image recognition (Common Objects in Context, COCO dataset). Then, obtain the pre-built initial adapter network, refer to Figure 3 , Figure 3 The schematic diagram shows a basic unit structure of a simple adapter network according to an exemplary embodiment of the present disclosure. The simplest adapter network structure can be composed of a convolution layer and an activation layer (ReLU layer). The initial adapter network is used to perform feature extraction processing on multiple training images respectively, and the initial adapter network outputs the first image feature. The first image feature determined by the above steps can be used as the data basis for model training.

[0056] In an exemplary embodiment of the present disclosure, for step S110, feature extraction processing is performed on multiple training images in a training image set by an initial adapter network to obtain a first image feature, including: image enhancement processing is performed on the multiple training images to obtain enhanced training images; an image codec is acquired, and the enhanced training images are encoded and decoded by the image codec to obtain training image decoding features; and feature extraction processing is performed on the training image decoding features by the initial adapter network to obtain the first image feature.

[0057] Among them, image enhancement processing can be a method of generating more samples by performing various transformations on the image so that the model training process has a sufficient number of samples. The enhanced training image can be an image obtained after image enhancement processing. The image codec can be a network structure for compressing and decompressing an image, and the image codec usually includes an encoder and a decoder. The training image decoding feature can be an image feature obtained after decoding the training image.

[0058] refer to Figure 4 , Figure 4The flowchart of the feature matching training adapter according to the exemplary embodiment of the present disclosure is schematically shown. First, the training image is preprocessed, and the preprocessing process includes data enhancement processing, such as the training image is subjected to data enhancement processing by the data enhancement method of the selected transformer target detection model to obtain an enhanced sample image. Common data enhancement methods include image rotation, flipping, cropping, and resizing. In this embodiment, a rotation, flipping and other methods in a deep learning model (Data-efficient Image Transformer, DETR) for target detection tasks are adopted, while the operations of converting the image into the corresponding tensor features, normalization, data filling, etc. still use the methods in the diffusion transformation model (Diffusion Transformer Model, DTM).

[0059] The adapter construction part is mainly responsible for building a network model based on deep learning. The main processing steps include: determining the image codec based on machine learning according to the needs of various business scenarios. Common ones include JPEG AI and domestic DCM codecs. The input of the initial adapter network is the decoding features of the image codec based on machine learning. In this disclosure, it is the decoding features of the training image obtained by encoding and decoding the enhanced training image using the image codec.

[0060] Then, the initial adapter network performs feature extraction processing on the training image decoding features to obtain the first image feature, which is recorded as feature 1. The first image feature obtained through the above steps can be used as the data basis for adapter network training.

[0061] In an exemplary embodiment of the present disclosure, the training image decoding features are obtained by the following steps: performing image padding processing on the enhanced training image to obtain a padded training image; obtaining a downsampling multiple corresponding to the image codec; performing image compression processing on the padded training image based on the downsampling multiple to obtain the training image decoding features, and the decoding feature dimension of the training image decoding features is the same as the input feature dimension of the initial adapter network.

[0062] The image padding process may be a padding process of padding the width and height of the training image to an integer multiple of 2 using an image codec. The padded training image may be an image obtained after the image padding process is performed on the training image. The downsampling multiple may refer to a value to which the size of the image is reduced to a fraction of the original size when the downsampling operation is performed on the training image. The decoded feature dimension may be a feature dimension of the image feature obtained after the training image is compressed and decoded.

[0063] Continue to refer Figure 4, image preprocessing of training images also includes image padding. For the obtained enhanced training images, the image codec can be used to normalize them and pad the height (height) and width (width) to integer multiples of 2. For example, suppose the dimension of an image is [3,h,w], where 3 means that the image is a three-channel color image. The codec usually normalizes the original image and pads h and w to integer multiples of 2. Suppose the dimension of the preprocessed image is [3,new_h,new_w]. After the image is decoded by the codec, the h and w of the features are integer multiples of 2.

[0064] Assume that the downsampling multiple corresponding to the image codec is n, that is, the original image is downsampled n times after preprocessing, and the dimension of the decoded feature obtained after downsampling compression is [channel, new_h / n, new_w / n], where channel is the number of channels of the decoded feature, which is generally an integer multiple of 2. In addition, the number of input channels of the adapter network = channel, that is, the decoded feature dimension of the training image decoded feature is the same as the input feature dimension of the initial adapter network. The training image is encoded and decoded by the image codec, and the decoded features output by the codec are used as the input of the initial adapter, so that the adapter network can convert the decoded features into the input of the target detection model.

[0065] Step S120, obtaining a pre-trained initial target detection model, where the initial target detection model includes a head detection model and a body detection model.

[0066] According to some exemplary embodiments of the present disclosure, the initial target detection model may be a pre-trained target detection model. The head detection model may be a network structure determined based on the segmentation point of the Transformer target detection model, and before the segmentation point, it may be represented by transformer_head. The body detection model may be a network structure determined based on the segmentation point of the Transformer target detection model, and after the segmentation point, it may be represented by transformer_body.

[0067] The initial target detection model in the present disclosure can select a trained network model, for example, an open source, publicly callable, Transformer-based target detection model with excellent prediction performance, such as DETR. The initial target detection model may include a target detection model head part and a main part. The target detection model head usually has a backbone network module based on a convolutional neural network, such as resnet50. The backbone module can usually downsample the h and w of the image layer by layer to reduce the input dimension of the transformer part.

[0068] Step S130: The head detection model performs feature extraction processing on the multiple training images to obtain second image features.

[0069] According to some exemplary embodiments of the present disclosure, the second image feature may be an image feature obtained by extracting features from a training image using a head detection model.

[0070] Continue to refer Figure 4 After image preprocessing of the training image, the head detection model (transformer_head part) is used to extract features of the training image and output the second image feature, which is recorded as feature2. For example, the transformer_head part may include a convolution layer (CONV layer), a backbone layer, an activation layer (ReLU layer), and a maximum pooling layer (MaxPooling layer).

[0071] In an exemplary embodiment of the present disclosure, for step S130, a head detection model performs feature extraction processing on multiple training images to obtain a second image feature, including: obtaining a downsampling multiple corresponding to the image codec, and determining a model segmentation point of the initial target detection model based on the downsampling multiple; based on the model segmentation point, determining a head detection model and a body detection model of the initial target detection model; performing feature extraction processing on the training images through the head detection model to obtain a second image feature, and the image feature dimension of the second image feature is the same as the output feature dimension of the initial adapter network.

[0072] The model segmentation point may be a layer of a network used to segment the model into a head detection model and a body detection model. The image feature dimension may be a feature dimension of an output image feature obtained by performing feature extraction processing on a training image by the head detection model.

[0073] For the pre-trained initial target detection model, such as the target detection model pre-trained based on Transformer, obtain the downsampling multiple n of the image codec, and find the segmentation point (model segmentation point) based on the Transformer target detection model according to the downsampling multiple n, that is, locate the layer of the network where the image is downsampled n times in the backbone, divide the network into two, and use the network layer before the segmentation point as the head detection model, denoted by transformer_head, and the network layer after the segmentation point as the body detection model, denoted by transformer_body.

[0074] Then, the head detection model is used to perform feature extraction on the training image to obtain the second image feature feature2, and the dimension of feature2 is [channel1, height / n of the input image, width / n of the input image], that is, the number of output channels of the adapter network is the same as channel1. By extracting the second image feature through the above steps, the feature loss value between the first image feature and the second image feature can be determined, and then the adapter network can be trained.

[0075] Step S140, determining a feature loss value based on the first image feature and the second image feature, and training the initial adapter network based on the feature loss value to obtain an adapter network.

[0076] According to some exemplary embodiments of the present disclosure, the feature loss value may be an indicator for measuring the difference between the first image feature and the second image feature. The adapter network, also known as the feature adaptation network, may be an adapter network structure obtained by training the initial adapter network based on the feature loss value.

[0077] The feature adaptation network in this disclosure is responsible for adjusting the feature dimensions obtained by image decoding to the feature dimensions required by the intelligent task network (based on the object detection model pre-trained by Transformer). The network structure of the adapter can be designed based on experience. Figure 4 ,The adapter training part mainly describes the training method of the adapter network, and the tensor dimensions of the obtained first image feature feature1 and the second image feature feature2 are completely consistent.

[0078] Since the feature dimension of the head detection model output is consistent with the output feature dimension of the adapter network, the present disclosure proposes a training method for aligning the two features for learning, specifically calculating the feature loss value between the first image feature feature1 and the second image feature feature2, and training the initial adapter network through the feature loss value, such as using a mean-square error (MSE) loss function to calculate the feature loss value, and training the initial adapter network. During the initial adapter network training process, the image codec and the head detection model are in a parameter freezing state, that is, only the parameters of the adapter network are adjusted during the training process. The specific training configuration parameters are shown in Table 1.

[0079] Table 1. Examples of training hyperparameter configurations.

[0080] Batch size 16 Epoch 50 Learning rate 1e-5 Optimizer SGD Loss function MSE

[0081] In addition, other training hyperparameters can be configured as shown in Table 1 and can be adjusted according to actual conditions during actual training, such as configuring the number of epochs to 20. Through the above training process, an adapter network that can achieve feature dimension adjustment is obtained.

[0082] Step S150, constructing a target detection model according to the adapter network and the subject detection model.

[0083] According to some exemplary embodiments of the present disclosure, the target detection model can be a Transformer pre-trained target detection model based on decoding features constructed based on the model training method of the present disclosure. After obtaining the adapter network, it can be processed in series with the main detection model to obtain the target detection model for subsequent target detection.

[0084] In an exemplary embodiment of the present disclosure, for step S150, a target detection model is constructed based on the adapter network and the subject detection model, including: obtaining an image codec; and serially processing the image codec, the adapter network and the subject detection model to obtain a target detection model.

[0085] The construction part of the Transformer pre-trained target detection system based on decoding features mainly connects the trained machine learning image codec, adapter network, and part of the transformer target detection model after the model segmentation point (i.e., the subject detection model) in series for forward prediction reasoning to obtain the target detection model. By connecting the image codec, adapter network, and subject detection model in series, the decoding features of the codec can be used as the input of the subject detection model, without the need to retrain the target detection model, thus avoiding the problem of high memory usage and computing power caused by retraining the target detection model with an upsampling operator.

[0086] It should be noted that the terms “first”, “second”, etc. used in the present disclosure are only used to distinguish different image extraction features and should not impose any limitations on the present disclosure.

[0087] In summary, the model training method disclosed in the present invention obtains a training image set and a pre-constructed initial adapter network, performs feature extraction processing on multiple training images in the training image set through the initial adapter network to obtain a first image feature; obtains a pre-trained initial target detection model, the initial target detection model includes a head detection model and a main body detection model; performs feature extraction processing on multiple training images through the head detection model to obtain a second image feature; determines a feature loss value based on the first image feature and the second image feature, trains the initial adapter network based on the feature loss value to obtain an adapter network; and constructs a target detection model based on the adapter network and the main body detection model. On the one hand, directly using the decoded features of the image codec as the input of the target detection part model, a low-complexity adapter network can be trained at a very low cost, and there is no need to retrain a target detection model with a large number of parameters and excellent performance, which can avoid the problem of high memory and high computing power in the training process; on the other hand, selecting the model segmentation point and using the feature matching training method to train the adapter network, avoiding the joint training of the main body detection model part with the adapter network, can greatly reduce the video memory usage in the training process.

[0088] In addition, a target detection method is also provided. The target detection method of the present disclosure can be implemented using a server, or the method described in the present disclosure can be implemented using a terminal device. Figure 5 A schematic diagram schematically illustrates a target detection method process according to some embodiments of the present disclosure.

[0089] refer to Figure 5 , the target detection method may include the following steps:

[0090] Step S510: obtaining an image to be recognized.

[0091] Step S520, obtaining a pre-trained target detection model, where the target detection model is trained based on a model training method, and the target detection model includes an adapter network and a subject detection model.

[0092] Step S530: perform encoding and decoding processing on the image to be identified to obtain image decoding features.

[0093] Step S540: input the image decoding features into the adapter network to obtain adapter features.

[0094] Step S550, input the adapter feature into the subject detection model to obtain the target detection category corresponding to the image to be identified.

[0095] According to the target detection method in this example embodiment, based on the decoded features of the image codec as input, a custom adapter network is trained and a pre-trained partial target detection model is used to complete target detection. The decoded features are directly converted using the adapter network as the input of the main detection model. During the target detection process, there is no need to reconstruct the image based on the decoded features to achieve the purpose of target detection, which can save the problem of high memory and high computing power in the target detection process.

[0096] refer to Figure 6 , Figure 6 The system architecture design diagram of the target detection method according to the exemplary embodiment of the present disclosure is schematically shown. When using the target detection model, the image is first preprocessed using a machine learning-based image codec, and the decoded features of the image are extracted from the preprocessed image through a machine learning-based codec as image decoding features; the image decoding features are input into the adapter network, and the adapter features are output; then, the adapter features are input into the transformer_body target detection model, and the bounding box (BBox) commonly used in target detection and the corresponding category prediction results are output.

[0097] Among them, the part before image encoding can be deployed on the terminal side. The user can compress the pictures collected on the terminal side and send them to the server side through a low-bandwidth network. The server side decodes the code stream generated by the encoding to obtain the corresponding decoding features, and finally returns the results of the decoding feature target recognition to the terminal side.

[0098] In specific target detection tasks, you can also select an image codec and build an adapter network based on the specific target detection requirements. For example, the image codec can use the DCM codec and design the network structure of the adapter network. Figure 7 , Figure 7 The structure diagram of a self-designed adapter network according to an exemplary embodiment of the present disclosure is schematically shown. Figure 7 The adapter network in can be composed of 4 groups of 3x3 convolutions and ReLU activation functions. The number of feature channels after each group of convolution operations is set as follows Figure 7 shown.

[0099] The input of the network is the DCM decoding feature, whose dimension is [batch_size, 128, original image preprocessing H / 4, original image preprocessing W / 4]; the dimension of the output adapter feature after the adapter is [batch_size, 64, original image preprocessing H / 4, original image preprocessing W / 4]. That is, in terms of dimension, the image decoding feature and the adapter feature are consistent in width and height.

[0100] The above target detection model is used to infer and predict the image to be identified. The recognition effect is as follows: Figure 8 and Fig. 9 As shown, reference Figure 8 , Figure 8 A first recognition effect diagram of a target detection model performing target detection on an image according to an exemplary embodiment of the present disclosure is schematically shown. Figure 8 The recognition effects of the original DETR and the target detection model disclosed in this disclosure are shown respectively. Figure 8 The predicted categories (partial data) based on the original DETR output are: cat, 1.00; remote control, 1.00; sofa, couch, 0.73. Figure 8 The predicted categories (partial data) based on the output of the above object detection model are: cat, 1.00; remote control, 1.00; sofa, couch, 0.98.

[0101] refer to Fig. 9 , Fig. 9 A second recognition effect diagram of a target detection model performing target detection on an image according to an exemplary embodiment of the present disclosure is schematically shown. Fig. 9 The recognition result of another image is provided, which also shows the recognition effect of the original DETR and the target detection model of the present disclosure. Fig. 9 The predicted categories (partial data) based on the original DETR output are: car, 0.99; person, 0.99; truck, 0.91. Fig. 9 The predicted categories (partial data) based on the output of the above target detection model are: car, 1.00; person, 1.00; truck, 0.97.

[0102] It should be noted that, although the steps of the method of the present invention are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0103] In addition, in this exemplary embodiment, a model training device is also provided. Fig.10 The model training device 1000 may include: a first feature extraction module 1010, an initial model acquisition module 1020, a second feature extraction module 1030, a network training module 1040 and a detection model construction module 1050.

[0104] Specifically, the first feature extraction module 1010 is used to obtain a training image set and a pre-constructed initial adapter network, and perform feature extraction processing on multiple training images in the training image set through the initial adapter network to obtain a first image feature; the initial model acquisition module 1020 is used to obtain a pre-trained initial target detection model, and the initial target detection model includes a head detection model and a body detection model; the second feature extraction module 1030 is used to perform feature extraction processing on multiple training images through the head detection model to obtain a second image feature; the network training module 1040 is used to determine a feature loss value based on the first image feature and the second image feature, and train the initial adapter network based on the feature loss value to obtain an adapter network; the detection model construction module 1050 is used to construct a target detection model based on the adapter network and the body detection model.

[0105] In an exemplary embodiment of the present disclosure, the first feature extraction module 1010 includes a first feature extraction unit, which is used to: perform image enhancement processing on multiple training images to obtain enhanced training images; obtain an image codec, and perform encoding and decoding processing on the enhanced training images through the image codec to obtain training image decoding features; perform feature extraction processing on the training image decoding features by the initial adapter network to obtain first image features.

[0106] In an exemplary embodiment of the present disclosure, the first feature extraction unit includes a decoding feature extraction subunit, which is used to: perform image padding processing on the enhanced training image to obtain a padded training image; obtain a downsampling multiple corresponding to the image codec; perform image compression processing on the padded training image based on the downsampling multiple to obtain a training image decoding feature, and the decoding feature dimension of the training image decoding feature is the same as the input feature dimension of the initial adapter network.

[0107] In an exemplary embodiment of the present disclosure, the second feature extraction module 1030 includes a second feature extraction unit, which is used to: obtain the downsampling multiple corresponding to the image codec, and determine the model segmentation point of the initial target detection model based on the downsampling multiple; determine the head detection model and the body detection model of the initial target detection model based on the model segmentation point; perform feature extraction processing on the training image through the head detection model to obtain a second image feature, and the image feature dimension of the second image feature is the same as the output feature dimension of the initial adapter network.

[0108] In an exemplary embodiment of the present disclosure, the detection model construction module 1050 includes a detection model construction unit for: acquiring an image codec; and serially processing the image codec, the adapter network, and the subject detection model to obtain a target detection model.

[0109] In addition, in this exemplary embodiment, a target detection device is also provided. Fig.11 The target detection device 1100 may include: an image acquisition module 1110, a model acquisition module 1120, a decoding feature determination module 1130, an adapter feature determination module 1140 and a detection and recognition module 1150.

[0110] Specifically, the image acquisition module 1110 is used to acquire the image to be identified; the model acquisition module 1120 is used to acquire a pre-trained target detection model, which is trained based on the above-mentioned model training method, and the target detection model includes an adapter network and a subject detection model; the decoding feature determination module 1130 is used to encode and decode the image to be identified to obtain image decoding features; the adapter feature determination module 1140 is used to input the image decoding features into the adapter network to obtain adapter features; the detection and identification module 1150 is used to input the adapter features into the subject detection model to obtain the target detection category corresponding to the image to be identified.

[0111] The specific details of the virtual modules of the above-mentioned model training devices and target detection devices have been described in detail in the corresponding model training methods and target detection methods. The undisclosed details can be found in the implementation method content of the method part, and will not be repeated here.

[0112] It should be noted that although several modules or units of the model training device and the target detection device are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0113] The exemplary embodiments of the present disclosure also provide a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the above-mentioned model training method and target detection method are implemented.

[0114] In one embodiment, the computer program product may be a tangible product containing the computer program, such as a computer-readable storage medium storing the computer program. Fig.12 , Fig.12The schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure is schematically shown. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, and other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid state drive (SSD), etc. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory (Nand Flash), etc.

[0115] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0116] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0117] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device may execute) the method steps of various exemplary embodiments of the present disclosure, such as the above-mentioned model training method and target detection method.

[0118] The exemplary embodiments of the present disclosure also provide an electronic device, which may include a processor and a memory. The memory stores executable instructions of the processor, such as a computer program. The processor executes the method steps of various exemplary embodiments of the present disclosure by executing the executable instructions. In addition, the electronic device may also include a display for displaying a graphical user interface.

[0119] Reference below Fig.13 , the electronic device is exemplarily described in the form of a general-purpose computing device. It should be understood that Fig.13The electronic device 1300 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0120] like Fig.13 As shown, the electronic device 1300 may include: a processor 1310 , a memory 1320 , a bus 1330 , an I / O (input / output) interface 1340 , a network adapter 1350 , and a display 1360 .

[0121] The memory 1320 may include a volatile memory, such as a RAM 1321, a cache unit 1322, and may also include a non-volatile memory, such as a ROM 1323. The memory 1320 may also include one or more program modules 1324, such program modules 1324 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof may include the implementation of a network environment. For example, the program module 1324 may include each module in the above-mentioned device.

[0122] The processor 1310 may include one or more processing units. For example, the processor 1310 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0123] The processor 1310 may be used to execute executable instructions stored in the memory 1320, such as executing the above-mentioned model training method or target detection method.

[0124] The bus 1330 is used to realize the connection between different components of the electronic device 1300, and may include a data bus, an address bus, and a control bus.

[0125] The electronic device 1300 can communicate with one or more external devices 1400 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 1340 .

[0126] The electronic device 1300 can communicate with one or more networks through the network adapter 1350. For example, the network adapter 1350 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 1350 can communicate with other modules of the electronic device 1300 through the bus 1330.

[0127] The electronic device 1300 may display a graphical user interface through the display 1360 , such as a recognition result display interface that displays image target detection results.

[0128] although Fig.13 Not shown, other hardware and / or software modules may also be provided in the electronic device 1300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0129] As can be seen from the above, the technical solution of the present disclosure can be implemented as a method, an apparatus, a system, a computer program product, a storage medium, an electronic device, etc. Those skilled in the art can understand that various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, such as being respectively referred to as a "circuit", "module" or "system".

[0130] It should be understood that the present disclosure is not limited to the specific method steps or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. Those skilled in the art will easily think of other embodiments based on the specific embodiments provided by the present disclosure. Therefore, the specific embodiments provided by the present disclosure are only exemplary, and the scope and spirit of the present disclosure are indicated by the claims, and any variations, uses or adaptive changes of the present disclosure should be covered, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the technical field that are not disclosed in the present disclosure.

Claims

1. A model training method, characterized in that: include: Acquire a training image set and a pre-built initial adapter network, and perform feature extraction processing on a plurality of training images in the training image set through the initial adapter network to obtain a first image feature; Acquire a pre-trained initial target detection model, wherein the initial target detection model includes a head detection model and a body detection model; The head detection model performs feature extraction processing on the multiple training images to obtain second image features; determining a feature loss value based on the first image feature and the second image feature, and training the initial adapter network based on the feature loss value to obtain an adapter network; A target detection model is constructed based on the adapter network and the subject detection model.

2. The method according to claim 1, characterized in that The step of performing feature extraction processing on a plurality of training images in the training image set by the initial adapter network to obtain a first image feature comprises: Performing image enhancement processing on the multiple training images to obtain enhanced training images; Acquire an image codec, and perform encoding and decoding processing on the enhanced training image by using the image codec to obtain a decoding feature of the training image; The initial adapter network performs feature extraction processing on the training image decoding features to obtain the first image features.

3. The method according to claim 2, characterized in that The encoding and decoding processing of the enhanced training image by the image codec to obtain the training image decoding features includes: Performing image filling processing on the enhanced training image to obtain a filled training image; Obtaining a downsampling multiple corresponding to the image codec; The padded training image is subjected to image compression processing based on the downsampling multiple to obtain the training image decoding feature, wherein the decoding feature dimension of the training image decoding feature is the same as the input feature dimension of the initial adapter network.

4. The method according to claim 1, characterized in that: The step of performing feature extraction processing on the plurality of training images by the head detection model to obtain second image features includes: Obtaining a downsampling multiple corresponding to an image codec, and determining a model segmentation point of the initial object detection model based on the downsampling multiple; Based on the model segmentation points, determining a head detection model and a body detection model of the initial target detection model; The training image is subjected to feature extraction processing by the head detection model to obtain a second image feature, wherein the image feature dimension of the second image feature is the same as the output feature dimension of the initial adapter network.

5. The method according to any one of claims 1 to 4, characterized in that: The step of constructing a target detection model according to the adapter network and the subject detection model includes: Get the image codec; The image codec, the adapter network and the subject detection model are connected in series to obtain the object detection model.

6. A target detection method, characterized in that: include: Obtain an image to be recognized; Obtain a pre-trained target detection model, wherein the target detection model is trained based on the model training method according to any one of claims 1 to 5, and the target detection model includes an adapter network and a subject detection model; Performing encoding and decoding processing on the image to be identified to obtain image decoding features; Inputting the image decoding feature into the adapter network to obtain an adapter feature; The adapter feature is input into the subject detection model to obtain the target detection category corresponding to the image to be identified.

7. A model training device, characterized in that: include: A first feature extraction module is used to obtain a training image set and a pre-built initial adapter network, and perform feature extraction processing on a plurality of training images in the training image set through the initial adapter network to obtain a first image feature; An initial model acquisition module, used to acquire a pre-trained initial target detection model, wherein the initial target detection model includes a head detection model and a body detection model; A second feature extraction module, configured to perform feature extraction processing on the plurality of training images by using the head detection model to obtain second image features; a network training module, configured to determine a feature loss value based on the first image feature and the second image feature, and train the initial adapter network based on the feature loss value to obtain an adapter network; The detection model building module is used to build a target detection model based on the adapter network and the main detection model.

8. A target detection device, characterized in that: include: An image acquisition module, used for acquiring an image to be identified; A model acquisition module, used to acquire a pre-trained target detection model, wherein the target detection model is trained based on the model training method according to any one of claims 1 to 5, and the target detection model includes an adapter network and a subject detection model; A decoding feature determination module, used for performing encoding and decoding processing on the image to be identified to obtain image decoding features; an adapter feature determination module, configured to input the image decoding feature into the adapter network to obtain an adapter feature; The detection and recognition module is used to input the adapter features into the subject detection model to obtain the target detection category corresponding to the image to be recognized.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the model training method as described in any one of claims 1 to 5, or to implement the target detection method as described in claim 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the model training method as described in any one of claims 1 to 5, or implements the target detection method as described in claim 6.

Images