Image classification method, device, apparatus and storage medium

Abstract

Embodiments of the present application disclose an image classification method, device and equipment, and a storage medium. The method comprises: obtaining a to-be-processed image; outputting a predicted feature map based on the to-be-processed image by an image classification model modified based on a target detection model, wherein the target detection model is a model for predicting a target detection frame based on an anchor frame; the image classification model is a model obtained by modifying a prior frame for increasing the proportion of an object based on the anchor frame; the predicted feature map comprises predicted information obtained based on the prior frame; and determining a multi-label category of the to-be-processed image based on the predicted feature map.

Description

TECHNICAL FIELD

[0001] The present application relates to, but is not limited to, the technical field of computer vision, and in particular to an image classification method and device, equipment and a storage medium. BACKGROUND

[0002] When deploying an algorithm into a real production environment, there is often a problem of inconsistency between the distribution of the detected object and the training data (Out-of-Distribution Detection, OOD). Most previous studies have focused on the OOD problem in multi-classification tasks, while in multi-label classification tasks, the OOD problem has been less studied. Most existing studies on the OOD problem in multi-label classification tasks are to expand the data to improve the algorithm indicators, which not only leads to long training time, but also does not achieve ideal results. SUMMARY

[0003] Therefore, the embodiments of the present application provide at least an image classification method, device, equipment and storage medium.

[0004] The technical solutions of the embodiments of the present application are implemented as follows:

[0005] In a first aspect, the embodiments of the present application provide an image classification method, which comprises: acquiring a to-be-processed image; outputting a predicted feature map based on the to-be-processed image by an image classification model modified based on a target detection model, wherein the target detection model is a model for predicting a target detection frame based on an anchor frame; the image classification model is a model obtained by modifying a prior frame for increasing the proportion of an object based on the anchor frame; the predicted feature map comprises predicted information predicted based on the prior frame; and determining a multi-label category of the to-be-processed image based on the predicted feature map.

[0006] In a second aspect, the embodiments of the present application provide an image classification device, which comprises: a first acquisition module configured to acquire a to-be-processed image; an output module configured to output a predicted feature map based on the to-be-processed image by an image classification model modified based on a target detection model, wherein the target detection model is a model for predicting a target detection frame based on an anchor frame; the image classification model is a model obtained by modifying a prior frame for increasing the proportion of an object based on the anchor frame; the predicted feature map comprises predicted information predicted based on the prior frame; and a first determination module configured to determine a multi-label category of the to-be-processed image based on the predicted feature map.

[0007] In a third aspect, the embodiments of the present application provide a computer device, which comprises a memory and a processor, and the memory stores a computer program capable of running on the processor, and the processor implements part or all of the steps in the above method when executing the program.

[0008] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement some or all of the steps of the above method.

[0009] In the embodiments of the present application, the anchor box in the target detection model is modified into a prior box for increasing the proportion of the object, and an image classification model is obtained. Then, the image classification model outputs a predicted feature map, and further obtains the multi-label categories of the image to be processed. On the one hand, since the prior box is obtained based on the anchor box, the structure of the anchor box itself can be utilized, and the proportion of the object in the prior box is increased, so that the prior box can more accurately represent the position of the object, thereby improving the accuracy of model training. On the other hand, since the target detection model itself can distinguish the object of interest and the irrelevant object in the image containing different categories and multiple objects, modifying the target detection model into the image classification model can well utilize the inherent ability of the target detection model, realize the multi-label classification of the image, and solve the OOD problem in the multi-label classification task.

[0010] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the technical solutions of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the technical solutions of the present application.

[0012] Figure 1 An implementation flowchart of an image classification method provided by the embodiments of the present application is shown in the figure;

[0013] Figure 2 An implementation flowchart of a forming method of an image classification model provided by the embodiments of the present application is shown in the figure;

[0014] Figure 3 An implementation flowchart of a training method of an image classification model provided by the embodiments of the present application is shown in the figure;

[0015] Figure 4 A schematic diagram of a YOLO v3 model provided by the embodiments of the present application is shown in the figure;

[0016] Figure 5A A schematic diagram of an anchor box provided by the embodiments of the present application is shown in the figure;

[0017] Figure 5B A schematic diagram of a prior box provided by the embodiments of the present application is shown in the figure;

[0018] Figure 6A schematic diagram of a composition structure of an image classification device provided by an embodiment of the present application is shown in the figure.

[0019] Figure 7 A schematic diagram of a hardware entity of a computer device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0020] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further described in detail below in combination with the drawings and embodiments, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by a person of ordinary skill in the art without making any creative effort fall within the scope of protection of the present application.

[0021] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0022] The terms "first / second / third" involved only distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first / second / third" can interchange the specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the present application and are not intended to limit the present application.

[0024] An image classification method is provided by an embodiment of the present application, which can be executed by a processor of a computer device. The computer device can be a server, a notebook computer, a tablet computer, a desktop computer, a smart television, a set-top box, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable game device) and the like with data processing capability. Figure 1 An implementation flowchart of an image classification method provided by an embodiment of the present application is shown in the figure, as shown in the figure, the method comprises the following steps S101 to S103: Figure 1

[0025] Step S101: obtaining an image to be processed;

[0026] ​Here, the to-be-processed image refers to any image that needs to be classified. For example, an image of a cat, an image of a dog, and the like. The to-be-processed image can be two-dimensional or three-dimensional, such as a grayscale image or a Red Green Blue (RGB) image, and the like. The size of the to-be-processed image can be any size, for example, in the case of a two-dimensional to-be-processed image, the size can be 1024*1024.

[0027] Step S102: based on the to-be-processed image, output a predicted feature map by an image classification model modified based on a target detection model, wherein the target detection model is a model for predicting a target detection box by an anchor box; the image classification model is a model obtained by modifying the anchor box to a prior box for increasing the proportion of the object; and the predicted feature map includes predicted information obtained based on the prior box.

[0028] Here, the target detection model refers to a model for detecting a target to obtain a detection box, a category, and a confidence. The target detection model can predict a target detection box by an anchor box, such as Faster RCNN, YOLO v3, and the like; or directly output a predicted target detection box, such as YOLO v1, and the like. The embodiments of the present application are applicable to the target detection model that predicts a target detection box by an anchor box.

[0029] The image classification model is a model obtained by modifying the anchor box of the target detection model to a prior box for increasing the proportion of the object. The prior box can be obtained by modifying the anchor box. In some embodiments, the method of obtaining the prior box can include reducing the width and length of the anchor box by the same proportion to obtain the prior box. For example, the width and length of the anchor box are reduced to 1 / 2, 2 / 3, and the like of the previous ones, so that the proportion of the object in the prior box is increased compared to the proportion of the object in the original anchor box, that is, the proportion of the background is reduced.

[0030] The predicted feature map refers to a feature map including predicted information of the to-be-processed image obtained by processing the to-be-processed image. In some embodiments, the predicted information can include the category score, the confidence, and the detection box information (such as the offset of the target detection box) of the object; in some embodiments, since the target of the image classification model is image classification, the predicted information can only include the category score and the confidence, and does not include the detection box information.

[0031] The prior box is used for labeling sample data before model training. By changing the prior box, the position of the target object in the predicted feature map can be changed, and thus the predicted information predicted by the predicted feature map can be changed. Since the proportion of the object in the prior box is increased and the proportion of the background is reduced, the position of the target object can be more accurately labeled, and thus the model finally trained is more accurate.

[0032] In some embodiments, the implementation of step S102 can include the following steps S1021 and S1022:

[0033] Step S1021: outputting a feature map by a feature extraction module in the image classification model based on the image to be processed;

[0034] Here, the feature extraction module is used to extract features from the image to be processed to obtain a feature map. According to different types of target detection models, the feature map can be a feature map of one size (for example, the target detection model is FasterRCNN), or a feature map of multiple sizes (for example, the target detection model is YOLO v3).

[0035] Step S1022: outputting the predicted feature map by a predictor in the image classification model based on the feature map.

[0036] Here, the predictor is used to output a predicted feature map according to the feature map. In some embodiments, the predictor can be a 1*1 convolution, and the predictor is trained to output prediction information.

[0037] Step S103: determining the multi-label categories of the image to be processed based on the predicted feature map.

[0038] Here, since the predicted feature map includes prediction information predicted based on the prior box, and the prediction information includes category scores and confidence, the prediction score of each category can be obtained by the category scores and the confidence, and the category with a prediction score higher than a threshold is determined as the multi-label category of the image to be processed.

[0039] Correspondingly, in some embodiments, the implementation of step S103 can include the following steps S1031 and S1032:

[0040] Step S1031: determining the prediction score of each category based on the category scores and the confidence in the predicted feature map;

[0041] Here, the predicted feature map generally includes three dimensions of length, height and channel, wherein the dimension of the channel dimension includes the number of categories and the confidence. For example, the image classification model can distinguish 80 categories, and for a prior box, the dimension of the channel dimension is at least 80+1, wherein 80 is the number of categories and 1 is the confidence. Multiplying the confidence by each category score can obtain the prediction score of each category.

[0042] Step S1032: determining the multi-label categories of the image to be processed based on the prediction score of each category.

[0043] Here, the implementation of step S1032 can include determining a class whose prediction score is higher than a threshold as a multi-label class of the image to be processed.

[0044] In the embodiments of the present application, the anchor box in the target detection model is modified into a prior box for increasing the proportion of the object, to obtain an image classification model; and then a prediction feature map is output by using the image classification model, and then a multi-label class of the image to be processed is obtained. On the one hand, since the prior box is obtained based on the anchor box, the structure of the anchor box itself can be utilized, and the proportion of the object in the prior box is increased, so that the prior box can more accurately represent the position of the object, to improve the accuracy of model training. On the other hand, since the target detection model itself can distinguish the object of interest and the irrelevant object for the image containing different classes and multiple objects, the internal ability of the target detection model can be well utilized by modifying the target detection model into the image classification model, to realize the multi-label classification of the image, and to solve the OOD problem in the multi-label classification task.

[0045] In some embodiments, as shown in Figure 2 The forming method of the image classification model can include the following steps S201 to S203:

[0046] Step S201: obtaining an anchor box in the target detection model;

[0047] Here, the target detection model is taken as YOLO v3 for illustration.

[0048] YOLO v3 can output three feature maps of different scales, which are 13*13, 26*26 and 52*52. The anchor box corresponding to the 13*13 feature map is: 116*90, 156*198 and 373*326; the anchor box corresponding to the 26*26 feature map is: 30*61, 62*45 and 59*119; and the anchor box corresponding to the 52*52 feature map is: 10*13, 16*30 and 33*23.

[0049] Step S202: determining a center point of the anchor box;

[0050] Step S203: reducing the length and width of the anchor box to half of the original length and width respectively with the center point as the center, to obtain the prior box.

[0051] In the embodiments of the present application, the length and width of the anchor box are reduced to half of the original length and width to obtain the prior box, so that the proportion of the object in the prior box is increased and the proportion of the background is reduced. In addition, since the prior box is obtained based on the anchor box, the structure of the anchor box itself can be utilized, and the proportion of the object in the prior box is increased, so that the prior box can more accurately represent the position of the object, to improve the accuracy of model training.

[0052] In some embodiments, the loss function of the target detection model comprises a classification loss, a confidence loss and a bounding box loss, and the prediction information comprises a class score and a confidence, as shown in Figure 3 As shown, the image classification model is trained in the following method, comprising the following steps S301 to S304:

[0053] Step S301: obtaining a training sample set;

[0054] Here, the training sample set refers to a set of image samples used to train the image classification model.

[0055] Step S302: determining, based on the training sample set, a class score and a confidence predicted based on the prior bounding box in each training sample output by the image classification model;

[0056] That is, the image classification model here does not output bounding box information.

[0057] In some embodiments, the image classification model can be realized to output only the class score and the confidence without outputting the bounding box information by modifying the dimension of the output channel dimension of the predictor, i.e., removing the 4-dimensional data of the bounding box information.

[0058] In some embodiments, the target detection model can be a YOLO model, the image classification model can be a model obtained by modifying the YOLO model based on the anchor box to increase the proportion of the object, and the predictor in the YOLO model is used to remove the 4-dimensional data of the bounding box in the channel dimension of the output prediction feature map to obtain the model. The following takes the image classification model as an example to be modified from the YOLO model for detailed description:

[0059] For the YOLO model, the output dimension of the predictor is N x N x [3x(4+1+C)], wherein N x N represents the grid number of the output prediction feature map, 3 represents that there are 3 anchor boxes for each grid, 4 represents the position offset (x, y, w, h) of each bounding box, 1 represents the confidence of each bounding box, and C represents the class of the bounding box.

[0060] The modified YOLO model, i.e., the image classification model, modifies the output dimension of the predictor to N x N x [3x(1+C)], i.e., removes the 4-dimensional data of the bounding box, and the remaining dimensions represent that only the confidence and the class score are output for each grid.

[0061] Step S303: determining, based on the class score and the confidence predicted based on the prior bounding box in each training sample, a loss function of the image classification model;

[0062] Here, the loss function is based on the class score and the confidence, i.e., the loss function does not include the bounding box loss.

[0063] In some embodiments, the implementation of step S303 can include the following steps S3031 and S3032:

[0064] Step S3031: based on the class score and the confidence predicted based on the prior box in each training sample, determine the classification loss and the confidence loss of the image classification model.

[0065] Here, the calculation method of the classification loss and the confidence loss can refer to related technologies.

[0066] Step S3032: determine the sum of the classification loss and the confidence loss, and obtain the loss function of the image classification model.

[0067] That is, L_total = L_obj + L_cls.

[0068] Wherein, L_total represents the loss function of the image classification model, L_obj represents the confidence loss, and L_cls represents the class loss.

[0069] Step S304: based on the loss function, train the image classification model to convergence.

[0070] In the embodiments of the present application, the class score and the confidence predicted based on the prior box in each training sample output by the image classification model are used to determine the loss function of the image classification model, and then the image classification model is trained to convergence. Since the prediction information output by the image classification model does not include the bounding box information, the loss function of the image classification model does not include the bounding box loss. Compared with the method of training the image classification model by including the bounding box loss, the training method provided in the embodiments of the present application can make the model focus more on the training of the classification task, thereby improving the accuracy of the model; and can reduce the size of the model, facilitating the training.

[0071] The target detection algorithm can be regarded as an extension of the multi-label problem, which not only needs to detect multiple categories of objects in the image, but also needs to predict the position of the object in the image, and can distinguish the interested object from the irrelevant object in the image containing different categories and multiple objects, which is an inherent ability of the target detection model.

[0072] At present, the target detection generally adopts the scheme of YOLO series. Taking YOLO v3 as an example, the problem to be solved is: how to convert YOLO v3 into a model suitable for multi-label classification problem, only using the ability of YOLO v3 itself, without relying on the increase of input data.

[0073] YOLO v3 is a general object detection algorithm model, which can well judge the objects of interest and irrelevant objects in an image. Using this characteristic, the OOD problem in the multi-label classification task can be effectively solved.

[0074] The embodiment of the application mainly modifies YOLO v3 locally, so that it can be directly used in the multi-label classification task. The changes include: 1) remove the detection box information; 2) use local grid (i.e. the above prior box) in the detection box confidence calculation process, and the calculation method of the class probability is consistent with YOLO v3.

[0075] Figure 4 The structure of YOLO v3 is shown. As can be seen, YOLO v3 includes residual modules, detection layers, up-sampling layers and other omitted layers, and includes two feature processing methods of splicing and adding between layers to form new features. The entire structure, the input image passes through Darknet-53 (without full connection layer, not shown in the figure), and then passes through the detection layer to output three size feature maps, which can be understood by referring to the structure of YOLO v3. Among them, the first feature map is down-sampled by 32 times, the second feature map is down-sampled by 16 times, and the third feature map is down-sampled by 8 times. The single number in the figure represents the serial number of the network layer.

[0076] The output dimension of each scale feature map is N x N x [3 x (4 + 1 + C)], wherein N x N represents the grid number of the output feature map, 3 represents that each grid has 3 anchor boxes, 4 represents the position offset information (x, y, w, h) of each detection box, 1 represents the confidence of each detection box, and C represents the class of the detection box. The loss function of YOLO v3 includes three parts: one is the detection box loss L_bbox, the second is the confidence loss L_obj, and the third is the class loss L_cls.

[0077] Figure 5A For Figure 4 The enlarged view of the smallest feature map (i.e. scale 1, size 13*13) in the middle. Among them, the small black box 502 represents the grid, which is 13 in the horizontal and vertical directions, and the white box 501 represents the rectangular box of the object, which corresponds to the black grid (i.e. the small black box 502). During inference, 3 rectangular boxes, confidence and class scores corresponding to the rectangular box are predicted for each grid.

[0078] The embodiment of the application migrates YOLO v3 to the multi-label classification task, and the inherent ability of YOLO v3 can solve the OOD problem in the multi-label classification task. Therefore, the embodiment of the application makes some modifications based on the basic framework of YOLO v3.

[0079] First, for the detection layer, YOLO v3 can be used to predict the detection box, corresponding to L_bbox. However, the target task in the embodiment of the present application is multi-label classification, only the class-related information is needed, and the detection box information is not needed. Therefore, the output dimension of the detection layer (i.e. the predictor) of YOLO v3 is modified to N x N x [3x(1+C)], and the 4-dimensional data of the detection box is removed, and the remaining dimensions represent that each grid only outputs the confidence and class score.

[0080] Second, for the detection box confidence, the judgment logic used by YOLO v3 to represent the confidence of each detection box is mainly based on the intersection over union (IoU) to distinguish positive and negative samples. YOLO v3 detects objects through each grid, however, each object is not limited to being predicted by only one grid; at the same time, since the outline of the object is not necessarily a rectangle, there may be some grids that contain other unrelated objects, causing the detection of other objects when detecting. Therefore, for the grids occupied by the object, the embodiment of the present application uses the definition of "local grid" to replace the anchor box in the original YOLO v3, which comprehensively considers the prediction of objects in multiple grids and the condition that no other unrelated objects appear in the grid.

[0081] Referring to Figure 5B To understand "local grid", the white box 501 represents the complete bounding box of the object (i.e. the original anchor box), the black small box 502 represents the center grid of the complete bounding box, and the white box 503 represents half of the complete bounding box (i.e. the prior box). The embodiment of the present application uses the white box 503 to replace the white box 501 originally used by YOLO v3. The white box 503 can make the occupied grid contain only the object itself, and reduce the possibility of other unrelated objects appearing in the grid.

[0082] Finally, for the class score, YOLO v3 is used to represent the class score of each detection box, i.e. the probability of belonging to each class. Since each grid may be responsible for detecting the same object, each grid is assigned multiple classes, i.e. the class score of each grid represents the probability of the grid belonging to each class. The calculation method of the probability is consistent with the definition of YOLO v3, and sigmoid is used to output the probability of each class.

[0083] After the above modifications and follow-ups, the loss function is defined as:

[0084] L_total = L_obj + L_cls;

[0085] L_total represents the total loss, L_obj represents the confidence loss of the object, and L_cls represents the class loss of the object. The difference from YOLO v3 is that the detection box loss is less.

[0086] Based on the above improvements, the feature map output by YOLO v3 does not contain the bounding box information, only the confidence and class score of each object. At the same time, the operation of local grid is adopted, so that the network can better distinguish the objects of interest and irrelevant objects.

[0087] In the multi-label classification task, the final output is also the score and confidence of the picture belonging to each class. The output of the embodiment of the present application is consistent with it, so the YOLO v3 can be migrated to the multi-label classification task by the embodiment of the present application. Since YOLO v3 inherently has the ability to distinguish objects of interest and irrelevant objects, it helps to solve the OOD problem in the multi-label classification task.

[0088] Based on the foregoing embodiments, the embodiment of the present application provides an image classification device, which comprises various modules and units included in each module, and can be realized by a processor in a computer device. Of course, it can also be realized by a specific logic circuit. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).

[0089] Figure 6 The composition structure diagram of the image classification device provided by the embodiment of the present application is shown in Figure 6 As shown in the figure, the image classification device 600 comprises a first acquisition module 610, an output module 620 and a first determination module 630, wherein:

[0090] The first acquisition module 610 is configured to acquire a to-be-processed image.

[0091] The output module 620 is configured to output a predicted feature map by an image classification model modified based on a target detection model based on the to-be-processed image, wherein the target detection model is a model for predicting a target detection box based on an anchor box; the image classification model is a model obtained by modifying a prior box for increasing the proportion of an object based on the anchor box; and the predicted feature map comprises predicted information predicted based on the prior box.

[0092] The first determination module 630 is configured to determine a multi-label class of the to-be-processed image based on the predicted feature map.

[0093] In some embodiments, the apparatus further comprises a second obtaining module configured to obtain an anchor box in the target detection model; a second determining module configured to determine a center point of the anchor box; and a reducing module configured to reduce a length and a width of the anchor box by half respectively with the center point as a center to obtain the prior box.

[0094] In some embodiments, the loss function of the target detection model comprises a classification loss, a confidence loss and a detection box loss, the prediction information comprises a class score and a confidence, and the apparatus further comprises a third obtaining module configured to obtain a training sample set; a third determining module configured to determine, based on the training sample set, the class score and the confidence predicted based on the prior box in each training sample output by the image classification model; a fourth determining module configured to determine, based on the class score and the confidence predicted based on the prior box in each training sample, the loss function of the image classification model; and a training module configured to train the image classification model based on the loss function until convergence.

[0095] In some embodiments, the fourth determining module comprises a first determining submodule configured to determine, based on the class score and the confidence predicted based on the prior box in each training sample, the classification loss and the confidence loss of the image classification model; and a second determining submodule configured to determine a sum of the classification loss and the confidence loss to obtain the loss function of the image classification model.

[0096] In some embodiments, the first determining module 630 comprises a third determining submodule configured to determine, based on the class score and the confidence in the prediction feature map, a prediction score of each class; and a fourth determining submodule configured to determine, based on the prediction score of each class, a multi-label class of the image to be processed.

[0097] In some embodiments, the output module 620 comprises a first output submodule configured to output, based on the image to be processed, a feature map by a feature extraction module in the image classification model; and a second output submodule configured to output, based on the feature map, the prediction feature map by a predictor in the image classification model.

[0098] In some embodiments, the target detection model is a YOLO model, the image classification model is a prior box obtained by modifying the anchor box of the YOLO model to increase the proportion of an object, and the predictor in the YOLO model is used to output a model obtained by removing 4-dimensional data of a detection box from a channel dimension of the prediction feature map.

[0099] The descriptions of the above device embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects as the method embodiments. In some embodiments, the device provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0100] It should be noted that, in the embodiments of the present application, if the image classification method described above is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various program code storage media. Thus, the embodiments of the present application are not limited to any particular hardware, software or firmware, or any combination of hardware, software and firmware.

[0101] The embodiments of the present application provide a computer device, including a memory and a processor, the memory stores a computer program capable of running on the processor, and the processor implements part or all of the steps in the above method when executing the program.

[0102] The embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement part or all of the steps in the above method. The computer readable storage medium can be transitory or non-transitory.

[0103] The embodiments of the present application provide a computer program, which includes computer readable code, and when the computer readable code is running in a computer device, a processor in the computer device executes part or all of the steps in the above method.

[0104] The embodiment of the present application provides a computer program product, which comprises a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, part or all of the steps in the above method are implemented. The computer program product can be implemented by hardware, software or a combination thereof. In some embodiments, the computer program product is embodied as a computer storage medium. In other embodiments, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0105] It should be noted that the above description of various embodiments tends to emphasize the differences between various embodiments, and the same or similar parts can be referred to each other. The above description of the device, storage medium, computer program and computer program product embodiments is similar to the description of the method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the device, storage medium, computer program and computer program product embodiments of the present application, please refer to the description of the method embodiments.

[0106] It should be noted that, Figure 7 A hardware entity of a computer device in the embodiment of the present application is shown in FIG. 7, which comprises a processor 701, a communication interface 702 and a memory 703, wherein: Figure 7 The hardware entity of the computer device 700 comprises a processor 701, a communication interface 702 and a memory 703, wherein:

[0107] The processor 701 generally controls the overall operation of the computer device 700.

[0108] The communication interface 702 can enable the computer device to communicate with other terminals or servers through a network.

[0109] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed by the processor 701 and modules in the computer device 700 (for example, image data, audio data, voice communication data and video communication data), which can be realized by FLASH or RAM. The processor 701, the communication interface 702 and the memory 703 can transmit data through the bus 704.

[0110] It should be understood that every feature and combination of features that is described above in relation to one embodiment is applicable to at least one other embodiment, unless specifically stated otherwise. It should also be understood that every embodiment described above can be combined with any other embodiment unless specifically stated otherwise.

[0111] It should be noted that, as used in this application, the terms "comprises" or "comprising," or the like are used in the sense of "including" and not of "consisting only of," such that the process, method, article, or apparatus that includes elements in addition to those listed after such a term in this application are still within the scope of that process, method, article, or apparatus. Where the term "comprises" is used in the form "comprising" or "comprises", unless otherwise stated, it is to be interpreted as meaning that it includes not only the stated elements but also elements that are inherent in the process, method, article, or apparatus that make up the generic non- limiting field related to the process, method, article, or apparatus.

[0112] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed components can be through some interfaces, indirect coupling or communication connection between devices or units, which can be electrical, mechanical or other forms.

[0113] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; they can be located in one place or distributed on multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0114] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0115] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, the foregoing program can be stored in a computer readable storage medium, and the program executes the steps of the method embodiments when executed; and the foregoing storage medium includes a mobile storage device, a read only memory (ROM), a magnetic disc or an optical disc, and various storage medium capable of storing program codes.

[0116] Alternatively, the integrated units of the present application can be stored in a computer readable storage medium if the integrated units are realized in the form of software function modules and sold or used as independent products. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes a mobile storage device, a ROM, a magnetic disc or an optical disc, and various storage medium capable of storing program codes.

[0117] The above is only an embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application.

Claims

1. An image classification method, characterized by, The method comprises the following steps: acquiring an image to be processed; outputting a predicted feature map by an image classification model modified based on a target detection model based on the image to be processed, wherein the target detection model is a model for predicting a target detection frame by an anchor frame; the image classification model is a model obtained by modifying a prior frame for increasing the proportion of an object based on the anchor frame, and a predictor in the image classification model is used to output a channel dimension of the predicted feature map without 4-dimensional data of a detection frame; the predicted feature map comprises predicted information predicted based on the prior frame; determining a multi-label category of the image to be processed based on the predicted feature map; wherein the forming method of the image classification model comprises: acquiring an anchor frame in the target detection model; determining a center point of the anchor frame; reducing the length and width of the anchor frame to half of the original length and width respectively with the center point as the center to obtain the prior frame.

2. The method of claim 1, wherein, The loss function of the target detection model comprises a classification loss, a confidence loss and a detection frame loss, the predicted information comprises a category score and a confidence, and the image classification model is trained according to the following method, comprising: acquiring a training sample set; determining the category score and the confidence predicted based on the prior frame in each training sample output by the image classification model based on the training sample set; determining the loss function of the image classification model based on the category score and the confidence predicted based on the prior frame in each training sample; training the image classification model based on the loss function until convergence.

3. The method of claim 2, wherein, The method for determining the loss function of the image classification model based on the category score and the confidence predicted based on the prior frame in each training sample comprises: determining the classification loss and the confidence loss of the image classification model based on the category score and the confidence predicted based on the prior frame in each training sample; determining the sum of the classification loss and the confidence loss to obtain the loss function of the image classification model.

4. The method of claim 2, wherein, The method for determining the multi-label category of the image to be processed based on the predicted feature map comprises: determining a prediction score of each category based on the category score and the confidence in the predicted feature map; determining the multi-label category of the image to be processed based on the prediction score of each category.

5. The method of claim 1, wherein, The method for outputting a predicted feature map by an image classification model modified based on a target detection model based on the image to be processed comprises: outputting a feature map by a feature extraction module in the image classification model based on the image to be processed; outputting the predicted feature map by a predictor in the image classification model based on the feature map.

6. The method of claim 5, wherein, The target detection model is a YOLO model.

7. An image classification apparatus characterized by comprising: The method comprises the following steps: a first acquiring module is configured to acquire an image to be processed; An output module is configured to output a predicted feature map based on the image to be processed by an image classification model modified based on a target detection model, wherein the target detection model is a model for predicting a target detection frame based on an anchor frame; the image classification model is a model obtained by modifying the anchor frame to a prior frame for increasing the proportion of an object, and removing 4-dimensional data of the detection frame from a channel dimension of a predictor in the image classification model to obtain the model for outputting the predicted feature map; and the predicted feature map comprises predicted information obtained based on the prior frame. A first determination module is configured to determine a multi-label category of the image to be processed based on the predicted feature map. A second acquisition module is configured to acquire the anchor frame in the target detection model. A second determination module is configured to determine a center point of the anchor frame. A reduction module is configured to reduce a length and a width of the anchor frame to half of the original length and width respectively with the center point as a center to obtain the prior frame.

8. A computer device comprising a memory and a processor, the memory storing a computer program capable of running on the processor, characterized in that, The processor executes the program to implement the steps in the method of any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps in the method of any one of claims 1 to 6.

Images