Image object detection method, device, medium and equipment
By introducing a soft-supervised subnet in a single-stage object detection algorithm, the semantic gradient of the classification subnet is used to soft-supervise the position prediction results of the regression subnet, the problem of feature misalignment is solved, and the accuracy of image object detection is improved and efficiency.
Patent Information
- Application Number
- CN202311740451.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-15
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-12-15
AI Technical Summary
The single-stage object detection algorithm has a problem of feature aberration in image object detection, resulting in a low accuracy of prediction results.
The network architecture adopts a single-stage object detection algorithm, including backbone network, classification subnetwork and regression subnetwork, and introduces soft-supervised subnetwork into the output results of the regression subnetwork, and uses the semantic gradient back-passed regression subnetworks to the regression subnetwork. Soft supervision of the position prediction results ensures that the position prediction results focus more on semantic information.
The regression accuracy of image object detection results is improved, the problem of feature misalignment is avoided, and the calculation efficiency of single-stage object detection algorithm is maintained.
Smart Images

Figure CN117710334B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of target detection technology, and in particular, relates to an image object detection method, device, medium and equipment. Background Art
[0002] The single-stage target detection algorithm is widely used in the positioning and classification of image objects. The algorithm directly predicts the results of feature points in an end-to-end manner, but there may be problems with feature alignment, which leads to relatively low accuracy of the prediction results. Summary of the invention
[0003] The embodiments of the present application provide an image object detection method, apparatus, medium and device, which can improve the accuracy of image object detection by a single-stage target detection algorithm at least to a certain extent.
[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.
[0005] According to a first aspect of an embodiment of the present application, a method for detecting an object in an image is provided. The method uses a single-stage target detection algorithm to perform object detection on a target image. The network architecture of the single-stage target detection algorithm includes a backbone network, a classification subnetwork, and a regression subnetwork. The network architecture also includes a soft supervision subnetwork, and the soft supervision subnetwork is connected to the classification subnetwork and the regression subnetwork, respectively. The method includes:
[0006] Performing feature extraction on the target image through the backbone network to obtain a feature map;
[0007] Inputting the feature map into the classification subnetwork and the regression subnetwork to perform category prediction and position prediction respectively;
[0008] For the feature map input to the regression sub-network, the feature map is processed by the soft supervision sub-network and then sent to the classification sub-network, so as to soft-supervise the position prediction result output by the regression sub-network based on the semantic gradient returned by the classification sub-network;
[0009] According to the category prediction result of the classification subnetwork and the position prediction result of the regression subnetwork, the object detection result of the target image is output.
[0010] In some embodiments of the present application, based on the above-mentioned solution, the soft supervision sub-network includes a feature adaptation module and a soft supervision module, and the feature map is processed by the soft supervision sub-network and sent to the classification sub-network, including:
[0011] The feature map is subjected to deformable convolution sampling by the feature adaptation module, and offset compensation is performed on each obtained sampling point to generate a target feature;
[0012] The target feature is weighted by the soft supervision module to obtain a target weight value, and the target weight value is sent to the classification subnetwork, where the target weight value is used to weight the category prediction result of the classification subnetwork.
[0013] In some embodiments of the present application, based on the above solution, the feature adaptation module is used to:
[0014] Perform deformable convolution sampling on each feature point in the feature map to obtain a sampling point set for each feature point;
[0015] An offset calculation is performed on each sampling point in each sampling point set, and a bilinear interpolation method is used to perform offset superposition on each sampling point to obtain the target feature.
[0016] In some embodiments of the present application, based on the aforementioned scheme, the soft supervision module includes a first convolutional layer Conv, a normalization layer GN, an activation function layer ReLU, a second convolutional layer Conv and an activation function layer Sigmod, which are connected in sequence.
[0017] In some embodiments of the present application, based on the aforementioned solution, the soft supervision sub-network further includes a data matching module, and the data matching module is used to perform data matching processing on the target features.
[0018] In some embodiments of the present application, based on the aforementioned solution, the soft supervision of the position prediction result output by the regression subnetwork based on the semantic gradient returned by the classification subnetwork includes:
[0019] Based on the semantic gradient returned by the classification subnetwork, obtaining a modified pre-selected box corresponding to the target feature;
[0020] The basic pre-selected box of the regression sub-network is corrected by the corrected pre-selected box to perform soft supervision on the position prediction result of the regression sub-network.
[0021] In some embodiments of the present application, based on the above solution, the outputting the object detection result of the target image includes:
[0022] The object detection result of the target image is outputted using a non-maximum suppression method.
[0023] According to a second aspect of an embodiment of the present application, an image object detection device is provided, wherein the device uses a single-stage target detection algorithm to perform object detection on a target image, wherein the network architecture of the single-stage target detection algorithm includes a backbone network, a classification subnetwork, and a regression subnetwork, wherein the network architecture also includes a soft supervision subnetwork, wherein the soft supervision subnetwork is connected to the classification subnetwork and the regression subnetwork, respectively, and wherein the device includes:
[0024] An extraction unit, used for performing feature extraction on the target image through the backbone network to obtain a feature map;
[0025] A prediction unit, used for inputting the feature map into the classification subnetwork and the regression subnetwork for category prediction and position prediction respectively;
[0026] A supervision unit, for processing the feature map input to the regression sub-network through the soft supervision sub-network and then sending the feature map to the classification sub-network, so as to soft-supervise the position prediction result output by the regression sub-network based on the semantic gradient returned by the classification sub-network;
[0027] An output unit is used to output the object detection result of the target image according to the category prediction result of the classification subnetwork and the position prediction result of the regression subnetwork.
[0028] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one computer program instruction is stored. The at least one computer program instruction is loaded and executed by a processor to implement the operations performed by any method described in the first aspect.
[0029] According to a fourth aspect of an embodiment of the present application, an electronic device is provided, comprising one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the operations performed by any of the methods described in the first aspect.
[0030] One or more technical solutions provided by the embodiments of the present invention achieve at least the following technical effects or advantages:
[0031] The image object of the present invention adopts a single-stage target detection algorithm. For the feature map input to the regression subnetwork, the feature map is processed by the soft supervision subnetwork and then sent to the classification subnetwork, so as to perform soft supervision on the position prediction result output by the semantic gradient regression subnetwork based on the classification subnetwork, so that the position prediction result output by the regression subnetwork is more focused on the semantic information, avoiding the problem of feature misalignment in single-stage target detection and improving the accuracy of the regression of image object detection results.
[0032] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0034] Figure 1 A flowchart of an image object detection method according to an embodiment of the present application;
[0035] Figure 2 This is a network architecture diagram of a single-stage target detection algorithm according to an embodiment of the present application;
[0036] Figure 3 A structural diagram of a soft monitoring module according to an embodiment of the present application;
[0037] Figure 4 A structural diagram of an image object detection device according to an embodiment of the present application;
[0038] Figure 5 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION
[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0040] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.
[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0042] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0043] It should also be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those shown or described.
[0044] Object detection algorithms are used to locate the bounding boxes of objects in images and classify them. With the development of convolutional neural networks (CNNs), object detection algorithms based on CNNs have significantly improved performance and are widely used in technical fields such as autonomous driving, robot navigation, and infrared imaging.
[0045] It should be noted that the target detection algorithm based on convolutional neural network includes single-stage target detection algorithm and multi-stage target detection algorithm. Among them, the multi-stage target detection algorithm adopts a network framework of multiple regressions. First, a pre-selected box is generated according to the feature points. Secondly, the high-quality pre-selected box obtained by screening is sampled and feature information is aggregated. Then, the pre-selected box is fine-tuned through the downstream network using the feature information, and finally the final target object category and bounding box are predicted. The multi-stage target detection algorithm has the advantage of high prediction result accuracy, but there are problems of complex calculation process and low efficiency. The single-stage target detection algorithm directly predicts the result of the feature points in an end-to-end manner, which has the advantage of high calculation efficiency, but there is a problem of low prediction result accuracy. The reason for the above defects is that there is a lack of a process of sampling and aggregating feature information through the pre-selected box, which makes it impossible to align the features, and then leads to low prediction result accuracy.
[0046] Therefore, in order to overcome the technical defects of the single-stage target detection algorithm and the technical defects of the complex calculation process and low efficiency of the multi-stage target detection algorithm, the embodiment of the present application proposes an image object detection method, which performs soft supervision on the position prediction results output by the semantic gradient regression subnetwork returned by the classification subnetwork without increasing the computational burden, so that the position prediction results output by the regression subnetwork are more focused on semantic information, avoiding the problem of feature misalignment in single-stage target detection, and improving the accuracy of image object detection result regression. The image object detection method will be described in detail below.
[0047] See also Figure 1 and Figure 2 , Figure 1 A flowchart of an image object detection method according to an embodiment of the present application is shown. Figure 2 A network architecture diagram of a single-stage target detection algorithm according to an embodiment of the present application is shown.
[0048] like Figure 1 and Figure 2 As shown, according to a first aspect of an embodiment of the present application, an image object detection method is provided, wherein the method uses a single-stage target detection algorithm to perform object detection on a target image, wherein the network architecture of the single-stage target detection algorithm includes a backbone network, a classification subnetwork, and a regression subnetwork, wherein the network architecture also includes a soft supervision subnetwork, wherein the soft supervision subnetwork is connected to the classification subnetwork and the regression subnetwork, respectively, and the method is implemented, but is not limited to, by steps S101 to S104:
[0049] Step S101. Extracting features of the target image through the backbone network to obtain a feature map;
[0050] In step S101, the target image refers to an image obtained by scaling an initial image acquired by an image acquisition device, wherein the image acquisition device may be a camera or other device, and the size of the target image after scaling may be 640×480.
[0051] In step S101, the backbone network adopts the ResNet50 network. In some embodiments, the ResNet50 network can extract the features of the target image through multiple convolutional layers and pooling layers, and use residual blocks to solve the problem of gradient disappearance in deep networks. For example: each residual block includes two convolutional layers, wherein the first convolutional layer is used to reduce the size of the feature map, and the second convolutional layer is used to increase the depth of the feature. It can be understood that in the process of feature extraction, ResNet50 gradually extracts high-level features of the image through multiple residual blocks, so that more complex image content can be recognized. In practical applications, the ResNet50 model that has been pre-trained on a large data set can be used for feature extraction, and these features can be used in subsequent classification subnetworks and regression subnetworks. It can be seen that the ResNet50 network is an efficient feature extraction model. It uses residual blocks to solve the problem of gradient disappearance in deep networks, and can effectively extract high-level features of complex images, thereby playing an important role in image object detection.
[0052] It can be understood that after the ResNet50 network performs feature extraction on the target image, the obtained feature map can be a high-dimensional matrix, such as a 64×64×256 high-dimensional matrix.
[0053] Step S102: Input the feature map into the classification subnetwork and the regression subnetwork to perform category prediction and position prediction respectively;
[0054] like Figure 2 As shown in the diagram, in the network architecture using a single-stage target detection algorithm, a classification branch and a regression branch (i.e., a classification subnetwork and a regression subnetwork) are generally included. In the classification subnetwork, multiple Conv-GN-ReLU (convolution-normalization-activation function) layers and convolution activation layers are provided, such as two sequentially connected Conv-GN-ReLU layers with a convolution kernel size of 3×3, and a Conv-sigmoid layer with a convolution kernel size of 3×3. The feature map is subjected to category recognition, i.e., semantic segmentation, through the above modules, thereby identifying the object category in the target image. The specific algorithm principle is the existing principle and will not be described in detail here. In the regression subnetwork, multiple Conv-GN-ReLU (convolution-normalization-activation function) layers, conv convolution layers, and pts2bbox minimum circumscribed rectangular transform layers are set. The Conv-GN-ReLU (convolution-normalization-activation function) layer and the conv convolution layer are used to sample the feature points of the feature map input to the regression branch to obtain the sampling point set P corresponding to each feature point. I , and for the sampling point set P I After multiple convolutions, the sampling point set P is transformed intoI Convert to pre-selected box It can be understood that the principle of position prediction of the feature map by the regression sub-network is the existing principle and will not be repeated here.
[0055] Step S103. For the feature map input to the regression sub-network, the feature map is processed by the soft supervision sub-network and then sent to the classification sub-network, so as to perform soft supervision on the position prediction result output by the regression sub-network based on the semantic gradient returned by the classification sub-network;
[0056] like Figure 2 As shown in the network architecture diagram using a single-stage target detection algorithm, the embodiment of the present application adds a soft supervision subnetwork on the basis of the classification subnetwork and the regression subnetwork, and connects the classification subnetwork and the regression subnetwork respectively through the soft supervision subnetwork to realize the cascade between the classification subnetwork and the regression subnetwork. The specific network structure and algorithm principle of the soft supervision subnetwork will be described in detail below.
[0057] In some embodiments of step S103, based on the above scheme, the soft supervision sub-network includes a feature adaptation module and a soft supervision module, and the feature map is processed by the soft supervision sub-network and then sent to the classification sub-network, including:
[0058] Step S1031. Perform deformable convolution sampling on the feature map through the feature adaptation module, and perform offset compensation on each obtained sampling point to generate a target feature;
[0059] In some embodiments of step S103, based on the above solution, the feature adaptation module is used to:
[0060] Perform deformable convolution sampling on each feature point in the feature map to obtain a sampling point set for each feature point;
[0061] An offset calculation is performed on each sampling point in each sampling point set, and a bilinear interpolation method is used to perform offset superposition on each sampling point to obtain the target feature.
[0062] Specifically, the feature adaptation module adopts a Dconv (deconvolution)-GN-ReLU layer with N sampling points, which is used to perform deformable convolution sampling on each feature point of the feature map input to the regression subnetwork, where N represents the number of sampling points corresponding to each feature point. In the embodiment of the present application, the number of sampling points corresponding to each feature point can be 9 points. The offset of each sampling point, that is, the corresponding position of the sampling point in the feature map, can be obtained through the feature point and semantic information. Since the data obtained by the offset calculation may include decimals, but when the decimal is corresponding to the feature map, the decimal has no position in the image, a bilinear interpolation method is used to offset and superimpose each sampling point to obtain the target feature, for example, the target feature is calculated by the weights around the offset calculation result of the sampling point.
[0063] Step S1032. The target feature is weighted by the soft supervision module to obtain a target weight value, and the target weight value is sent to the classification subnetwork, where the target weight value is used to weight the category prediction result of the classification subnetwork.
[0064] In some embodiments of step S1032, based on the aforementioned scheme, the soft supervision module includes a first convolutional layer Conv, a normalization layer GN, an activation function layer ReLU, a second convolutional layer Conv and an activation function layer Sigmod which are connected in sequence.
[0065] like Figure 3 As shown, this is a structural diagram of the soft-supervision module of an embodiment of the present application. By weighting the target features through the first convolution layer Conv, the normalization layer GN, the activation function layer ReLU, the second convolution layer Conv and the activation function layer Sigmod in sequence, the weighted weight of the position prediction of the regression branch for the category prediction of the classification subnetwork can be obtained, and the category prediction of the classification subnetwork is weighted by the weight to supervise the category prediction of the classification subnetwork, so that the category prediction can pay attention to the location information of the object.
[0066] In some embodiments of step S103, based on the above-mentioned solution, the soft supervision sub-network further includes a data matching module, and the data matching module is used to perform data matching processing on the target features.
[0067] It can be understood that in order to ensure the matching between data during the data calculation process, such as the matching between data dimensions, the soft supervision subnetwork is also provided with a data matching module, and the data matching module may include a softmax function layer and an integral function layer to process and dimensionally match the data.
[0068] In some embodiments of step S103, based on the above scheme, the soft supervision of the position prediction result output by the regression sub-network based on the semantic gradient returned by the classification sub-network includes:
[0069] Step S31. Based on the semantic gradient returned by the classification subnetwork, a modified pre-selected box corresponding to the target feature is obtained;
[0070] In step S31, it can be understood that since the classification subnetwork and the regression subnetwork are cascaded through the soft supervision subnetwork, when the classification subnetwork performs semantic gradient feedback, the semantic gradient is fed back to the regression subnetwork through the soft supervision subnetwork, specifically, through the soft supervision module.
[0071] Step S32: Correct the basic pre-selected box of the regression sub-network through the corrected pre-selected box to perform soft supervision on the position prediction result of the regression sub-network.
[0072] It is understandable that due to the basic pre-selection box in the prior art and the final output bounding box All by image tag Supervised training, but this method will force the edge-sensitive sampling point set P I Approximately the extreme value of gt-bboxes, due to the lack of strong supervision, the sampling point set P I It cannot adapt to the extreme values of its specified object, resulting in poor quality of edge sampling points, that is, inconsistent sampling points are always generated, resulting in irrelevant features, which in turn deteriorates downstream tasks. Therefore, the sampling point set P I The accurate positioning of is of great significance for feature adaptation.
[0073] From the above, we can know that the soft supervision module calculates the weights of the target features corresponding to the feature map, and uses the weights to weight the category predictions of the classification subnetwork to supervise the category predictions of the classification subnetwork, so that the category predictions can focus on the location information of the object; then after the classification subnetwork obtains the influence of the position weights on the semantic segmentation, when the semantic gradient is returned, for example, the BP (Back Propagation) algorithm can be used to inform the regression subnetwork of which positions can focus more on the semantic information, thereby achieving soft supervision of the position regression box prediction of the regression subnetwork. Specifically, after the target features corresponding to the feature map are weighted and sent to the classification subnetwork, the semantic gradients returned by the classification subnetwork are fine-tuned with the target features to obtain a corrected pre-selected box, such as Figure 2 In By modifying the pre-selected box Basic pre-selection box of regression sub-network Fine-tune the basic pre-selection box The final output bounding box Supervised training is performed to make the bounding box regression more focused on semantic information, thus achieving the basic pre-selection box Soft supervision avoids direct supervision of the basic pre-selected box The problem of poor quality of edge sampling points caused by supervised training.
[0074] Combined with the above description, the following is a further explanation of how to use the soft supervision sub-network to realize the soft supervision of the classification sub-network on the regression sub-network through a specific calculation formula:
[0075]
[0076] in, Represents the sampling set corresponding to each feature point of the feature map, Conv3 represents the Conv layer with a convolution kernel size of 3×3, ConvModule3 represents the Conv-GN-ReLU layer with a convolution kernel size of 3×3, Represents the feature map input to the regression subnetwork, Indicates the correction pre-selection box, DConvModule N represents a Dconv (deconvolution)-GN-ReLU layer with N sampling points, represents the pre-selected result of the classification sub-network, represents the feature map input to the classification subnetwork, represents the basic pre-selected box, pts2box represents the minimum bounding rectangle transformation layer, represents the loss function of the regression subnetwork, represents the position prediction result of the regression sub-network, represents the loss function of the classification subnetwork, Represents the category prediction result of the classification subnetwork.
[0077] Step S104: output the object detection result of the target image according to the category prediction result of the classification subnetwork and the position prediction result of the regression subnetwork.
[0078] In some embodiments of step S104, based on the above solution, outputting the object detection result of the target image includes:
[0079] The object detection result of the target image is outputted using a non-maximum suppression method.
[0080] It can be understood that the algorithm principle of the above non-maximum suppression method is an existing principle and will not be repeated here.
[0081] Based on the above disclosed content, the image object of the embodiment of the present application adopts a single-stage target detection algorithm. For the feature map input to the regression subnetwork, the feature map is processed by the soft supervision subnetwork and then sent to the classification subnetwork, so as to perform soft supervision on the position prediction result output by the semantic gradient regression subnetwork based on the classification subnetwork, so that the position prediction result output by the regression subnetwork is more focused on the semantic information, avoiding the problem of feature misalignment in single-stage target detection, and improving the accuracy of the regression of image object detection results. At the same time, the embodiment of the present application does not increase the computational burden, and improves the accuracy of the single-stage target detection algorithm for image object detection while satisfying the advantage of high efficiency of the single-stage target detection algorithm for image object detection.
[0082] The following describes an embodiment of the device of the present application, which can be used to execute the method in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the above embodiment of the method of the present application.
[0083] See also Figure 4 , shows a structural diagram of an image object detection device according to an embodiment of the present application.
[0084] like Figure 4 As shown, according to the second aspect of the embodiment of the present application, an image object detection device 200 is provided, wherein the device uses a single-stage target detection algorithm to perform object detection on a target image, wherein the network architecture of the single-stage target detection algorithm includes a backbone network, a classification subnetwork, and a regression subnetwork, wherein the network architecture also includes a soft supervision subnetwork, wherein the soft supervision subnetwork is connected to the classification subnetwork and the regression subnetwork, respectively, and the device 200 includes:
[0085] An extraction unit 201 is used to extract features of the target image through the backbone network to obtain a feature map;
[0086] A prediction unit 202, configured to input the feature map into the classification subnetwork and the regression subnetwork for category prediction and position prediction respectively;
[0087] A supervision unit 203 is used for processing the feature map input to the regression sub-network through the soft supervision sub-network and then sending the feature map to the classification sub-network, so as to perform soft supervision on the position prediction result output by the regression sub-network based on the semantic gradient returned by the classification sub-network;
[0088] The output unit 204 is used to output the object detection result of the target image according to the category prediction result of the classification subnetwork and the position prediction result of the regression subnetwork.
[0089] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one computer program instruction is stored. The at least one computer program instruction is loaded and executed by a processor to implement the operations performed by any method described in the first aspect.
[0090] The computer readable storage medium may be a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the computer readable storage medium of the present application is not limited thereto. In the present application, the readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0091] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).
[0092] See also Figure 5 , which is a schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application.
[0093] According to a fourth aspect of an embodiment of the present application, an electronic device is provided, comprising one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the operations performed by any of the methods described in the first aspect.
[0094] like Figure 5 As shown, the electronic device 400 is in the form of a general computing device. The components of the electronic device 400 may include but are not limited to: at least one processing unit 410, at least one storage unit 420, and a bus 430 connecting different system components (including the storage unit 420 and the processing unit 410).
[0095] The storage unit stores program codes, which can be executed by the processing unit 410, so that the processing unit 410 executes the steps described in the above “Example Method” section of this specification according to various exemplary implementations of the present application.
[0096] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 421 and / or a cache memory unit 422 , and may further include a read-only memory unit (ROM) 423 .
[0097] The storage unit 420 may also include a program / utility 424 having a set (at least one) of program modules 425, such program modules 425 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0098] Bus 430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0099] The electronic device 400 may also communicate with one or more external devices 500 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 400, and / or communicate with any device that enables the electronic device 400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 450. Furthermore, the electronic device 400 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 460. As shown, the network adapter 460 communicates with other modules of the electronic device 400 via a bus 430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0100] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium. Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hard wiring, or a combination of any of these. In addition, each functional unit may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit.
[0101] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0102] The units described as separate components may or may not be physically separated, and the components of the control device may or may not be physical units, that is, they may be located in one place or distributed in multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.
[0104] The above description is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.
Claims
1. A method for detecting an object in an image, wherein the method uses a single-stage target detection algorithm to detect an object in a target image, wherein the network architecture of the single-stage target detection algorithm includes a backbone network, a classification subnetwork, and a regression subnetwork. It is characterized in that The network architecture further includes a soft supervision subnetwork, and the soft supervision subnetwork is connected to the classification subnetwork and the regression subnetwork respectively. The method includes: Performing feature extraction on the target image through the backbone network to obtain a feature map; Inputting the feature map into the classification subnetwork and the regression subnetwork to perform category prediction and position prediction respectively; For the feature map input to the regression sub-network, the feature map is processed by the soft supervision sub-network and then sent to the classification sub-network, so as to perform soft supervision on the position prediction result output by the regression sub-network based on the semantic gradient returned by the classification sub-network; Outputting an object detection result of the target image according to the category prediction result of the classification subnetwork and the position prediction result of the regression subnetwork; The soft supervision sub-network includes a feature adaptation module and a soft supervision module, and the feature map is processed by the soft supervision sub-network and then sent to the classification sub-network, including: The feature map is subjected to deformable convolution sampling by the feature adaptation module, and offset compensation is performed on each obtained sampling point to generate a target feature; The target feature is weighted by the soft supervision module to obtain a target weight value, and the target weight value is sent to the classification subnetwork, where the target weight value is used to weight the category prediction result of the classification subnetwork.
2. The method according to claim 1, It is characterized in that The feature adaptation module is used to: Perform deformable convolution sampling on each feature point in the feature map to obtain a sampling point set for each feature point; An offset calculation is performed on each sampling point in each sampling point set, and a bilinear interpolation method is used to perform offset superposition on each sampling point to obtain the target feature.
3. The method according to claim 1, It is characterized in that The soft supervision module includes a first convolution layer Conv, a normalization layer GN, an activation function layer ReLU, a second convolution layer Conv and an activation function layer Sigmod which are connected in sequence.
4. The method according to claim 1, It is characterized in that The soft supervision sub-network also includes a data matching module, and the data matching module is used to perform data matching processing on the target features.
5. The method according to claim 1, It is characterized in that The soft supervision of the position prediction result output by the regression subnetwork based on the semantic gradient returned by the classification subnetwork includes: Based on the semantic gradient returned by the classification subnetwork, obtaining a modified pre-selected box corresponding to the target feature; The basic pre-selected box of the regression sub-network is corrected by the corrected pre-selected box to perform soft supervision on the position prediction result of the regression sub-network.
6. The method according to claim 1, It is characterized in that The outputting the object detection result of the target image includes: The object detection result of the target image is outputted using a non-maximum suppression method.
7. An image object detection device, the device uses a single-stage target detection algorithm to detect objects in a target image, the network architecture of the single-stage target detection algorithm includes a backbone network, a classification subnetwork and a regression subnetwork, It is characterized in that The network architecture further includes a soft supervision subnetwork, which is connected to the classification subnetwork and the regression subnetwork respectively. The device includes: An extraction unit, used for performing feature extraction on the target image through the backbone network to obtain a feature map; A prediction unit, used for inputting the feature map into the classification subnetwork and the regression subnetwork for category prediction and position prediction respectively; A supervision unit, for processing the feature map input to the regression sub-network through the soft supervision sub-network and then sending the feature map to the classification sub-network, so as to soft-supervise the position prediction result output by the regression sub-network based on the semantic gradient returned by the classification sub-network; An output unit, used to output the object detection result of the target image according to the category prediction result of the classification subnetwork and the position prediction result of the regression subnetwork; The soft supervision sub-network includes a feature adaptation module and a soft supervision module, and the feature map is processed by the soft supervision sub-network and then sent to the classification sub-network, including: The feature map is subjected to deformable convolution sampling by the feature adaptation module, and offset compensation is performed on each obtained sampling point to generate a target feature; The target feature is weighted by the soft supervision module to obtain a target weight value, and the target weight value is sent to the classification subnetwork, where the target weight value is used to weight the category prediction result of the classification subnetwork.
8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores at least one computer program instruction, and the at least one computer program instruction is loaded and executed by a processor to implement the operations performed by any one of the methods according to claims 1-6.
9. An electronic device, It is characterized in that The method comprises one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the operations performed by the method according to any one of claims 1 to 6.
