Method, device and medium for object detection based on image and category attention

Abstract

The present invention provides a method, device and medium for target detection based on image and category attention, including: constructing a neural network model; constructing an image set; obtaining the category quantity distribution probability of each category in the image through the category of the object in the image and the number of objects contained in each category; obtaining the area distribution probability of each category in the image through the area occupied by each category; obtaining the object quantity distribution probability of the image through the number of objects in the image and the number of objects in the image set; obtaining the image attention weight according to the object quantity distribution probability and the area distribution probability of the image; constructing a loss function model according to the category quantity distribution probability of each category in the image and the image attention weight; training the neural network model and the loss function model through the image set; and using the trained neural network model and the loss function model to perform target detection on a new image. The above method and device improve the detection accuracy without increasing any training parameters.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to a method, device and medium for target detection based on image and category attention. Background Art

[0002] In recent years, deep learning-based object detection has made great progress. The latest object detection networks have improved the basic detection models in various aspects, and most of the improvements have increased the computational complexity and complexity of the network. In object detection networks, there are also problems of extremely unbalanced object categories and huge differences in the number of difficult and easy images. These two problems work together to prevent the model from focusing on important categories and images during training, thereby misleading the network and damaging detection accuracy. Object category imbalance means that different object types in the object detection dataset have different numbers of instances. And difficult and easy image imbalance means that in common object detection datasets, a large number of images are simple images, and simple images contribute less to network training.

[0003] The deep learning attention mechanism mainly includes position attention and channel attention. The position attention method assigns different weights to different feature map positions, so that it can focus on important areas in the image. The channel attention method considers that different feature map channels have different importance and assigns different weights to different channels. These two common attention methods are feature map-level and do not take image and object categories into consideration. Therefore, people began to consider expanding the attention mechanism.

[0004] In 2020, Peng proposed a method to implement an image attention mechanism using a deep learning classification network. This method uses an additional classification network to calculate the importance of a certain area in the image or the entire image, so as to focus on complex areas and images when training the detection network. Although this method has a significant improvement in accuracy, its training process is complicated and requires the use of a classification network to process the same image. In addition, in order to obtain more accurate regional importance, they introduced a dynamically updated network branch, which resulted in additional parameters and computation, so the network training time and inference time also increased compared to the baseline algorithm.

[0005] Currently, the more common object detection models are usually complex to implement and introduce a large number of computational parameters compared to the benchmark models, which slows down the training and inference speeds. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide an image and category attention-based target detection method, system, electronic device and medium for improving the target detection accuracy without increasing any training parameters.

[0007] According to one aspect of the present invention, there is provided a method for object detection based on image and category attention, comprising:

[0008] Constructing a neural network model, wherein the neural network model is used to detect the classification probability of objects in the image and the predicted value of the positioning offset of their positions;

[0009] Constructing an image set, wherein the image set is used to store images for which object categories and positioning offsets have been determined;

[0010] Obtaining a probability distribution of the number of objects in each category of the image according to the categories of the objects in the image and the number of objects in each category;

[0011] Obtaining the area distribution probability of each category in the image through the area occupied by each category;

[0012] The probability of the number of objects in the image is obtained by the number of objects in the image and the number of objects in the image set;

[0013] Obtaining an attention weight of the image according to a probability of object quantity distribution and an area distribution probability of the image;

[0014] Construct a loss function model based on the probability of the number of categories in each category in the image and the attention weight of the image;

[0015] The neural network model and loss function model are trained using the image set;

[0016] The trained neural network model and loss function model are used to perform target detection on the new image to obtain the classification probability of the object in the new image and the predicted value of the positioning offset of its position.

[0017] Optionally, it also includes:

[0018] A first balancing factor is introduced into the probability distribution of the number of categories of each category in the image to obtain the attention weight of each category in the image;

[0019] Among them, the loss function model is constructed according to the attention weight of each category in the image and the attention weight of the image.

[0020] Optionally, the step of obtaining the attention weight of each category of the image further includes:

[0021] The category containing more objects has a larger attention weight.

[0022] Optionally, the step of obtaining the attention weight of each category of the image further includes:

[0023] The categories containing more objects have smaller attention weights.

[0024] Optionally, it also includes:

[0025] A second balancing factor is introduced to obtain the attention weight of the image according to the distribution probability of the number of objects in the image and the area distribution probability of each category.

[0026] Optionally, the neural network model includes one or more of RetinaNet, RCNN, Mask RCNN, SSD and YOLO.

[0027] Optionally, it also includes:

[0028] The contribution degree is introduced into the attention weight of the image to balance the contribution degree of the distribution probability of the number of objects in the image and the area distribution probability of each category.

[0029] According to another aspect of the present invention, there is provided an object detection system based on image and category attention, comprising:

[0030] A model building module, which builds a neural network model, wherein the neural network model is used to detect the classification probability of objects in the image and the predicted value of the positioning offset of their positions;

[0031] An image set construction module is used to construct an image set, wherein the image set is used to store images of which the object category and the positioning offset have been determined;

[0032] A category quantity distribution probability obtaining module obtains the category quantity distribution probability of each category in the image according to the category of the objects in the image and the number of objects contained in each category;

[0033] An area distribution probability obtaining module, which obtains the area distribution probability of each category in the image through the area occupied by each category;

[0034] The object quantity distribution probability acquisition module obtains the object quantity distribution probability of the image through the number of objects in the image and the number of objects in the image set;

[0035] An image attention acquisition module obtains the attention weight of the image according to the number distribution probability and area distribution probability of objects in the image;

[0036] The loss function model building module builds the loss function model according to the probability of the number of categories of each category in the image and the attention weight of the image;

[0037] The training module trains the neural network model and loss function model through the image set;

[0038] The target detection module uses the trained neural network model and loss function model to perform target detection on the new image to obtain the classification probability of the object in the new image and the predicted value of the positioning offset of its position.

[0039] In addition, the present invention further provides an electronic device, the electronic device comprising:

[0040] at least one processor; and,

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned image and category-based attention target detection method.

[0043] In addition, the present invention also provides a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the above-mentioned image and category-based attention target detection method is implemented.

[0044] The image and category-based attention target detection method, system, electronic device and medium described in the present invention extend the attention mechanism in deep learning to the image and object category level, greatly improving the final target detection accuracy while ensuring that the amount of calculation does not increase. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] By referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more apparent and easy to understand. In the accompanying drawings:

[0046] Figure 1 A flowchart of an object detection method based on image and category attention provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of modules of an object detection system based on image and category attention provided by another embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the internal structure of an electronic device for implementing an image and category-based attention target detection method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0050] The present invention provides an object detection method based on image and category attention. Figure 1FIG. 1 is a flow chart of a method for object detection based on image and category attention provided by an embodiment of the present invention. The method can be executed by a device, and the device can be implemented by software and / or hardware.

[0051] In this embodiment, the object detection method based on image and category attention includes:

[0052] Step S1, constructing a neural network model, wherein the neural network model is used to detect the classification probability of an object in an image and the predicted value of the positioning offset of its position;

[0053] Step S2, constructing an image set, wherein the image set is used to store images whose object categories and positioning offsets have been determined;

[0054] Step S3, obtaining the category quantity distribution probability of each category in the image according to the category of the objects in the image and the number of objects contained in each category. For example, in image I i ,{1≤i≤s}, the target object category is {l1,l2,…,l m}, I i The categories included are {l1,l2,…,l m The number of objects in the i1 ,n i2 ,n i3 ,…,n im ; For I i , the probability of the number of categories is

[0055]

[0056] Step S4, obtaining the area distribution probability of each category in the image according to the area occupied by each category, for example, i The object categories included are {l1,l2,…,l m The area of ​​} is S i1 ,S i2 ,S i3 ,…,S im , for I i , and its category area distribution probability is

[0057]

[0058] Among them, S ij For image I i The total area of ​​objects of type j in , S is the total area of ​​the image. Optionally, when training the neural network structure, the images of the same size are scaled to the same size even if they are different, for example, S = 1024x1024;

[0059] Step S5, obtaining the probability of the number of objects in the image through the number of objects in the image and the number of objects in the image set, for example,

[0060] p i =N i / N

[0061] Among them, N i is the image i i The total number of annotated objects in the image set, N is the total number of annotated objects in the image set;

[0062] Step S6, obtaining the attention weight of the image according to the object quantity distribution probability and area distribution probability of the image, for example, obtaining the attention weight of the image by combining the object quantity distribution probability and the area distribution probability exponential and logarithm. Optionally, obtaining the attention weight of the image by the following formula:

[0063]

[0064] Step S7, constructing a loss function model according to the probability of the number of categories of each category in the image and the attention weight of the image. For example, the loss function model is constructed by the following formula:

[0065]

[0066] Among them, p ic =(p i1 ,p i2 ,p i3 ,...,p im ), o is the index of the predicted box of the target object output by the neural network structure, cls io is the classification loss of the prediction box output by the neural network structure, loc io It is the localization loss of the predicted box output by the neural network structure.

[0067] Step S8, training the neural network model and the loss function model through the image set;

[0068] Step S9, using the trained neural network model and loss function model to perform target detection on the new image, and obtain the classification probability of the object in the new image and the predicted value of the positioning offset of its position.

[0069] In one embodiment, in step S3, it also includes:

[0070] The first balancing factor is introduced into the probability of the category quantity distribution of each category in the image to obtain the attention weight of each category in the image.

[0071] In one embodiment, the step of obtaining the attention weight of each category of the image further includes:

[0072] The more objects a category contains, the greater its attention weight. For example, the first balancing factor σ1 is introduced into the probability distribution of the number of categories in each category of the image, and the attention weight of each category in the image is obtained according to the following formula:

[0073]

[0074] w ic =(w i1 ,w i2 ,w i3 ,...,w im ).

[0075] In one embodiment, the step of obtaining the attention weight of each category of the image further includes:

[0076] The category with more objects has smaller attention weight, for example,

[0077] The first balancing factor is introduced into the probability distribution of the number of categories in each category of the image, and the attention weight of each category in the image is obtained according to the following formula:

[0078]

[0079] w ic =(w i1 ,w i2 ,w i3 ,...,w im ).

[0080] In one embodiment, in step S7, a loss function model is constructed based on the attention weight of each category in the image and the attention weight of the image.

[0081]

[0082] In one embodiment, loss contribution is introduced into the loss function model to balance the contribution of classification loss and positioning loss.

[0083]

[0084] Among them, δ is the loss contribution.

[0085] In one embodiment, in step S6, it further includes:

[0086] The second balancing factor σ2 is introduced to obtain the attention weight of the image according to the probability of the number of objects in the image and the probability of the area distribution of each category.

[0087]

[0088] In one embodiment, in step S6, it further includes:

[0089] Introduce the contribution degree γ in the attention weight of the image to balance the contribution degree of the distribution probability of the number of objects in the image and the area distribution probability of each category

[0090]

[0091] In one embodiment, step S8 includes:

[0092] Obtain the reference frame and prediction frame of the image in the image set through the neural network structure;

[0093] Obtain intermediate variables based on the positioning offset of the prediction box and the reference box

[0094]

[0095] in, Represents image i i The probability (0 or 1) that the true category corresponding to the predicted box o is category j, c ioj Output the probability of the object contained in the prediction box o in the image i being category j for the neural network structure, q ioj is an intermediate variable;

[0096] Obtaining classification loss through intermediate variables

[0097] cls io =-(1-q io ) τ log(q io )

[0098] Among them, q io is an m-dimensional (number of categories) vector, q ioj is one of the values. The above formula processes a single value to obtain each value in the vector. After that, the vector is used for calculation. τ is a hyperparameter, which can be set or obtained through training.

[0099] The positioning loss of the prediction box is obtained by the following formula

[0100]

[0101] Among them, t io =(t x ,t y ,t w ,t h ,t θ ) is the center coordinate, width, height and orientation angle of the prediction box output by the neural network structure. is the true value of the positioning offset of the center coordinates, width, height and orientation angle of the predicted box, locio For image I i The positioning loss of the prediction box o (each prediction box has a positioning loss and a classification loss, both of which are m-dimensional);

[0102] The loss function model is trained through an image set, and the loss function model is solved and evaluated through the minimum loss function.

[0103] The above-mentioned target detection method based on image and category attention introduces a category attention method based on the distribution of object category quantity and an image attention method based on the distribution of object quantity and area distribution to address the object category imbalance problem and the extreme imbalance problem of difficult and easy image samples faced during the training of the detection network (neural network model), so as to better increase the detection accuracy of the target detection model without increasing any training parameters.

[0104] In one embodiment, the method for object detection based on image and category attention includes:

[0105] First, category attention based on the number distribution of object categories, in detail:

[0106] When the detection network (neural network) is trained, the number of various object categories in each picture is counted, and then the category quantity distribution is obtained. The quantity distribution is obtained by dividing the number of objects of each category in a picture by the total number of objects in the picture. In the quantity distribution part, there are two methods to assign appropriate weights to object types. The first method is to assign larger weights to categories with relatively small numbers. At this time, it is considered that increasing the weights of categories with small numbers can reduce the accuracy differences of the model on different categories. The second method is to assign larger weights to categories with large numbers. At this time, it is considered that categories with large numbers are easy to learn. In addition, in order to reduce the differences in the importance of different object categories, the present invention also introduces balance factors σ1 and σ2 (the first balance factor and the second balance factor are set to the same value to reduce the parameters that need to be searched). When the balance factor is infinite, the performance of the detection network is the same as the baseline network.

[0107] Specifically, for the image I in the training data set (image set), i ,{1≤i≤s}, the target object category is {l1,l2,…,l m}, I i The categories included are {l1,l2,…,l m The number of objects in the i1 ,n i2 ,n i3 ,…,n im , for I i , the probability of the number of categories is

[0108]

[0109] In order to prevent the neural network structure from focusing too much on certain categories of objects, a balance factor is introduced. Under the balance factor σ, image I i The attention weight of category j in is:

[0110]

[0111] In the above-mentioned method of obtaining the attention weights of categories, the categories with fewer numbers have greater weights. There is also a method of assigning weights to different object categories in different images, that is, to give more weights to the categories with more numbers:

[0112]

[0113] w ic =(w i1 ,w i2 ,w i3 ,...,w im )

[0114] in, That is to say, w ic is an m-dimensional vector representing image I i The corresponding class weight vector, m is the number of classes in the dataset.

[0115] Second, the image attention method based on the number distribution and area distribution of objects, in detail:

[0116] In the image attention part, the object quantity distribution and area distribution are combined. The object quantity distribution is the ratio of the number of objects in an image to the total number of objects in the entire data set. In addition, the present invention also considers the area distribution of objects, because the present invention believes that if the area occupied by an object in an image is larger, then the image has greater importance.

[0117] Specifically, for the image I in the training dataset i ,{1≤i≤s}, the object category is {l1,l2,…,l m}, I i The object categories included are {l1,l2,…,l m The area of ​​} is S i1 ,S i2 ,S i3 ,…,S im , for I i , and its category area distribution probability is

[0118]

[0119] Combining the quantity distribution and category area distribution together, and under the action of the balancing factor σ, the attention weight of image i is:

[0120]

[0121] Among them, γ>0 is used to balance the contribution of the two distributions to the image weight. In implementation, it can be simply set to 1. i is image I i Where, N is the total number of labeled objects in the data set. The present invention enables the detection network to focus on important images during training.

[0122] For the network to image I i The classification probability prediction value and positioning offset prediction value of each predicted box o obtained by processing are c io ∈R m (c io is an m-dimensional vector, where each value represents the probability that the prediction box belongs to a certain category, and m is the number of object categories), t io =(t x ,t y ,t w ,t h ,t θ )(5-dimensional vector, each value of which is the offset value after encoding, offset value + anchor box (i.e. reference box, usually, when performing target detection, a large number of reference boxes are laid on the picture, and then, only the deviation between the reference box and the prediction box is predicted, rather than directly obtaining the prediction box, so that the network can learn more easily how to better detect the prediction box of the object after processing). The true value of the classification label is (m-dimensional vector, one-hot encoding method, that is, only one value in the vector is 1, and the rest are 0), the actual value of the positioning offset is Where x, y, w, h, and θ represent the center coordinates, width, height, and orientation angle of the box, respectively.

[0123] t x =(xx a ) / w a ,t y =(yy a ) / h a

[0124] t w =log(w / w a ),t h =log(h / h a )

[0125] t θ =tan(θ-θ a )

[0126] In the above formula, the subscript a represents the anchor box (reference box), such as w a is the width of the reference box (the position, width, height and angle parameters of the reference box have different values ​​in different target detection models), and the other values ​​are similar. gt ,y gt ,w gt ,h gt ,θ gt )(gt, ground truth) can get the real offset value You can also use the predicted offset value t io Get the predicted box (x p ,y p ,w p ,h p ,θ p )(p,prediction).

[0127] Third, construct the loss function, in detail:

[0128] For image I i For a certain prediction box o in , if the true label corresponding to category j is 1, that is Then the intermediate variable q ioj From the following formula:

[0129]

[0130] in, The classification loss is obtained by the following formula:

[0131] cls io =-(1-q io ) τ log(q io )( τ is a hyperparameter, usually 2)

[0132] In the above formula, cls io Representative image I i The classification loss of the predicted box o is an m-dimensional vector, where m is the number of categories.

[0133] The positioning loss is obtained according to the following formula:

[0134]

[0135] In the above formula, loc io Representative image I i The positioning loss of the predicted box o is a 5-dimensional vector. It is summed and compressed into a scalar, and then its value is copied m-1 times, and finally expanded to an m-dimensional vector, and loc is also used.io To represent this m-dimensional vector.

[0136] We get the image I i The classification loss of the predicted box o in Positioning loss Use the following formula to get the overall loss function. Where δ is used to balance the contribution of classification loss and positioning loss to the total loss. i Is a scalar representing the image weight. ic Is an m-dimensional vector representing the category weight of the image.

[0137]

[0138] In one embodiment, under the same parameters, computational complexity, and training and reasoning time constraints, a test is performed on a large-scale target detection dataset DOTA, and the average precision mAP is used as an evaluation indicator to compare the target detection method based on image and category attention of the present invention with the target detection method using RetinaNet (without using quantity distribution and area distribution). The first method of the present invention is to assign a greater weight to a category with a relatively small number, and the second method of the present invention is to assign a greater weight to a category with a large number. The first method of the present invention has an improvement of 2.11 points (from 59.86% to 61.97%, at this time the balance factor σ is 800 / N, where N is the total number of labeled objects in the dataset) in mAP relative to the RetinaNet method, and the second method of the present invention has an improvement of 1.84 points (from 59.86% to 61.70%, at this time the balance factor σ is 900 / N) in mAP relative to the RetinaNet method.

[0139] like Figure 2 , which is a functional module diagram of the target detection system based on image and category attention of the present invention.

[0140] The target detection system 100 based on image and category attention of the present invention can be installed in an electronic device. According to the functions to be implemented, in one embodiment, the template-based corpus automatic expansion device can include a model construction module 10, an image set construction module 20, a category quantity distribution probability acquisition module 30, an area distribution probability acquisition module 40, an object quantity distribution probability acquisition module 50, an image attention acquisition module 60, a loss function model construction module 70, a training module 80 and a target detection module 90. The module described in the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0141] In this embodiment, the functions of each module / unit are as follows:

[0142] Image and category based attention based object detection systems include:

[0143] A model building module 10 is used to build a neural network model, wherein the neural network model is used to detect the classification probability of an object in an image and the predicted value of the positioning offset of its position;

[0144] An image set construction module 20 is used to construct an image set, wherein the image set is used to store images of which the object category and the positioning offset have been determined;

[0145] The quantity distribution probability obtaining module 30 obtains the category quantity distribution probability of each category in the image according to the category of the objects in the image and the number of objects contained in each category;

[0146] An area distribution probability obtaining module 40 obtains the area distribution probability of each category in the image according to the area occupied by each category;

[0147] An object quantity distribution probability obtaining module 50 obtains the object quantity distribution probability of the image according to the number of objects in the image and the number of objects in the image set;

[0148] An image attention obtaining module 60, which obtains the attention weight of the image according to the object quantity distribution probability and area distribution probability of the image;

[0149] A loss function model building module 70 builds a loss function model according to the probability of the number of categories of each category in the image and the attention weight of the image;

[0150] A training module 80, training a neural network model and a loss function model through an image set;

[0151] The target detection module 90 uses the trained neural network model and loss function model to perform target detection on the new image to obtain the classification probability of the object in the new image and the predicted value of the positioning offset of its position.

[0152] In one embodiment, the quantity distribution probability obtaining module 30 includes:

[0153] An object category obtaining unit, which obtains the category of the object in the image;

[0154] A first quantity obtaining unit, which obtains the quantity of objects contained in each category;

[0155] The quantity distribution probability obtaining unit obtains the category quantity distribution probability of each category in the image according to the following formula:

[0156]

[0157] Optionally, the method further includes: a first category attention weight obtaining unit, which introduces a first balancing factor into the category quantity distribution probability of each category in the image, and obtains the attention weight of each category in the image according to the following formula:

[0158]

[0159] w ic =(w i1 ,w i2 ,w i3 ,...,w im );

[0160] In addition, optionally, it also includes: a second attention weight obtaining unit, which introduces a third balance factor into the probability of the number of categories of each category in the image, and obtains the attention weight of each category in the image according to the following formula:

[0161]

[0162] w ic =(w i1 ,w i2 ,w i3 ,...,w im )

[0163] Among them, in the first attention weight acquisition unit, the first balance factor in the attention weight of the category with a small number and large weight is set to σ1; in the second attention weight acquisition unit, the third balance factor in the attention weight of the category with a large number and large weight is set to σ3, and the second balance factor of the attention weight of the image is set to σ2. When the category attention weight of the first attention weight acquisition unit is used for target monitoring, σ1=σ2, and when the category attention weight of the second attention weight acquisition unit is used for target monitoring, σ3=σ2.

[0164] In one embodiment, the image attention acquisition module 60 includes:

[0165] The first image attention acquisition unit introduces a second balancing factor, and obtains the attention weight of the image according to the distribution probability of the number of objects in the image and the area distribution probability of each category through the following formula:

[0166]

[0167] Optionally, it also includes: a second image attention obtaining unit, which introduces the contribution degree to obtain the attention weight of the image according to the number distribution probability and area distribution probability of the image through the following formula:

[0168]

[0169] In one embodiment, the loss function model building module 70 builds the loss function model by the following formula:

[0170]

[0171] In one embodiment, the training module 80 includes:

[0172] The prediction box and reference box acquisition unit obtains the reference box and prediction box of the image in the image set through a neural network structure;

[0173] The intermediate variable acquisition unit obtains the intermediate variable according to the positioning offset of the prediction box and the reference box.

[0174]

[0175] Classification loss acquisition unit, classification loss cls is obtained through intermediate variables io =-(1-q io ) τ log(q io );

[0176] The positioning loss obtaining unit obtains the positioning loss of the prediction box through the following formula

[0177]

[0178] The training unit trains the following loss function model through the image training set, and solves and evaluates the loss function model through the minimum loss function

[0179]

[0180] like Figure 3 , which is a schematic diagram of the structure of an electronic device for implementing the target detection method based on image and category attention according to the present invention.

[0181] The electronic device 200 may include a memory 210 , a processor 220 , and a bus, and may also include a computer program stored in the memory 210 and executable on the processor 220 , such as an image and category-based attention target detection program 211 .

[0182] The memory 210 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. The memory 210 may be an internal storage unit of the electronic device 200 in some embodiments, such as a mobile hard disk of the electronic device 200. The memory 210 may also be an external storage device of the electronic device 200 in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 200. Further, the memory 210 may also include both an internal storage unit of the electronic device 200 and an external storage device. The memory 210 may not only be used to store application software and various types of data installed in the electronic device 200, such as the code of the template-based corpus automatic expansion program, but also be used to temporarily store data that has been output or is to be output.

[0183] The processor 220 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 220 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 210 (such as target detection programs based on image and category attention, etc.), and calls data stored in the memory 210 to execute various functions of the electronic device 200 and process data.

[0184] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 210 and at least one processor 220, etc.

[0185] Figure 3 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3The structure shown does not constitute a limitation on the electronic device 200 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0186] For example, although not shown, the electronic device 200 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 220 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device 200 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0187] Furthermore, the electronic device 200 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 200 and other electronic devices.

[0188] Optionally, the electronic device 200 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 200 and to display a visual user interface.

[0189] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0190] The image and category-based attention target detection program 211 stored in the memory 210 of the electronic device 200 is a combination of multiple instructions, and when running in the processor 220, it can achieve:

[0191] Constructing a neural network model, wherein the neural network model is used to detect the classification probability of objects in the image and the predicted value of the positioning offset of their positions;

[0192] Constructing an image set, wherein the image set is used to store images for which object categories and positioning offsets have been determined;

[0193] Obtaining a probability distribution of the number of objects in each category of the image according to the categories of the objects in the image and the number of objects in each category;

[0194] Obtaining the area distribution probability of each category in the image through the area occupied by each category;

[0195] The probability of the number of objects in the image is obtained by the number of objects in the image and the number of objects in the image set;

[0196] Obtaining an attention weight of the image according to a probability of object quantity distribution and an area distribution probability of the image;

[0197] Construct a loss function model based on the probability of the number of categories in each category in the image and the attention weight of the image;

[0198] The neural network model and loss function model are trained using the image set;

[0199] The trained neural network model and loss function model are used to perform target detection on the new image to obtain the classification probability of the object in the new image and the predicted value of the positioning offset of its position.

[0200] Specifically, the specific implementation method of the processor 220 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0201] Furthermore, if the module / unit integrated in the electronic device 200 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0202] In addition, an embodiment of the present application further provides a computer-readable storage medium, which may be non-volatile or volatile, and includes a computer program, which implements the following operations when executed by a processor:

[0203] Constructing a neural network model, wherein the neural network model is used to detect the classification probability of objects in the image and the predicted value of the positioning offset of their positions;

[0204] Constructing an image set, wherein the image set is used to store images for which object categories and positioning offsets have been determined;

[0205] Obtaining a probability distribution of the number of objects in each category of the image according to the categories of the objects in the image and the number of objects in each category;

[0206] Obtaining the area distribution probability of each category in the image through the area occupied by each category;

[0207] The probability of the number of objects in the image is obtained by the number of objects in the image and the number of objects in the image set;

[0208] Obtaining an attention weight of the image according to a probability of object quantity distribution and an area distribution probability of the image;

[0209] Construct a loss function model based on the probability of the number of categories in each category in the image and the attention weight of the image;

[0210] The neural network model and loss function model are trained using the image set;

[0211] The trained neural network model and loss function model are used to perform target detection on the new image to obtain the classification probability of the object in the new image and the predicted value of the positioning offset of its position.

[0212] The specific implementation of the computer-readable storage medium of the present invention is roughly the same as the specific implementation of the above-mentioned image and category-based attention target detection method, device, and electronic device, and will not be repeated here.

[0213] In the above embodiments, the neural network model includes but is not limited to RetinaNet, RCNN, MaskRCNN, SSD and YOLO.

[0214] The target detection method, device, and electronic device based on image and category attention of the present invention introduce category attention into target detection, use the number distribution and area distribution of objects to calculate the importance of an image, and use the number distribution of object categories to calculate the category importance, thereby achieving a better increase in the detection accuracy of the target detection model without increasing any training parameters.

[0215] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0216] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0217] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0218] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0219] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0220] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for object detection based on image and category attention, characterized in that include: Constructing a neural network model, wherein the neural network model is used to detect the classification probability of objects in the image and the predicted value of the positioning offset of their positions; Constructing an image set, wherein the image set is used to store images for which object categories and positioning offsets have been determined; The probability of the number of objects in each category in the image is obtained by the categories of the objects in the image and the number of objects in each category, and a first balancing factor is introduced into the probability of the number of objects in each category in the image. , the attention weight of each category in the image is obtained according to the following formula: ;in, is the probability of the number of categories; the area distribution probability of each category in the image is obtained by the area occupied by each category; the probability of the number of objects in the image is obtained by the number of objects in the image and the number of objects in the image set; the second balance factor is introduced , the attention weight of the image is obtained according to the distribution probability of the number of objects in the image and the area distribution probability of each category: ;in, is an image The total number of labeled objects in is the total number of annotated objects in the image set, The area distribution probability of the category is obtained; a loss function model is constructed according to the attention weight of each category in the image and the attention weight of the image; the neural network model and the loss function model are trained through the image set; the trained neural network model and the loss function model are used to detect the target in the new image to obtain the classification probability of the object in the new image and the predicted value of the positioning offset of its position.

2. The method for object detection based on image and category attention according to claim 1, characterized in that: The step of obtaining the attention weight of each category of the image also includes: the category containing more objects has a greater attention weight.

3. The method for object detection based on image and category attention according to claim 1, characterized in that: The step of obtaining the attention weight of each category of the image also includes: the category containing more objects has a smaller attention weight.

4. The method for object detection based on image and category attention according to claim 1, characterized in that: The neural network model includes and One or more of .

5. The method for object detection based on image and category attention according to claim 1, characterized in that: Also includes: The contribution degree is introduced into the attention weight of the image to balance the contribution degree of the distribution probability of the number of objects in the image and the area distribution probability of each category.

6. An image and category-based attention-based object detection system, characterized in that The method comprises: a model building module, which builds a neural network model, wherein the neural network model is used to detect the classification probability of an object in an image and the predicted value of the positioning offset of its position; an image set building module, which builds an image set, wherein the image set is used to store images whose object categories and positioning offsets have been determined; and a category quantity distribution probability acquisition module, which acquires the category quantity distribution probability of each category in the image through the category of the object in the image and the number of objects contained in each category, and introduces a first balancing factor into the category quantity distribution probability of each category in the image. , the attention weight of each category in the image is obtained according to the following formula: ;in, is the probability of the number of categories distributed; an area distribution probability obtaining module, which obtains the area distribution probability of each category in the image through the area occupied by each category; The object quantity distribution probability acquisition module obtains the object quantity distribution probability of the image through the number of objects in the image and the number of objects in the image set; Image attention acquisition module, introducing the second balancing factor , the attention weight of the image is obtained according to the distribution probability of the number of objects in the image and the area distribution probability of each category: ; is an image The total number of labeled objects in is the total number of annotated objects in the image set, The loss function model construction module is used to construct the loss function model according to the attention weight of each category in the image and the attention weight of the image; the training module is used to train the neural network model and the loss function model through the image set; the target detection module is used to perform target detection on the new image using the trained neural network model and the loss function model to obtain the classification probability of the object in the new image and the predicted value of the positioning offset of its position.

7. An electronic device, characterized in that: The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image and category-based attention target detection method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for object detection based on image and category attention is implemented as described in any one of claims 1 to 5.

Images