Multi-label target detection method and device, computer equipment and storage medium

Abstract

The invention discloses a multi-label target detection method and device, computer equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining a to-be-detected image; the to-be-detected image is input into a multi-label target detection model, and the multi-label target detection model comprises a basic network layer, a regression prediction layer and a classification prediction layer; in the basic network layer, carrying out image feature extraction on a to-be-detected image to obtain a feature map; in the regression prediction layer, performing regression prediction on the boundary of each to-be-predicted target based on the feature map to obtain bounding box information; in the classification prediction layer, carrying out blocking processing on the to-be-detected image based on the bounding box information to obtain a plurality of class blocks; and obtaining a category prediction result output by each category block to obtain a multi-label target detection result. The invention also relates to the technical field of block chains, and the to-be-detected image is stored on a block chain network. According to the method, the detection precision and the multi-label detection capability of the model in a specific task are remarkably improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a multi-label target detection method, device, computer equipment and storage medium. Background Art

[0002] Object detection is a core task in the field of computer vision, which aims to automatically identify the location and category of target objects from images or videos.

[0003] With the rapid development of real-time target detection technology, taking YOLO (You Only Look Once) as an example, a fast target detection algorithm, the YOLO series of real-time target detection technologies have become a universal and efficient solution recognized by the industry, and are widely used in natural scene target recognition such as people, cars, and animals. However, in specific fields such as vehicle self-service damage assessment, the direct application of these YOLO series target detection technologies still faces challenges. For example, YOLO target detection is often designed based on the assumption of class equality, which simplifies the complex hierarchical relationship and severity differences between vehicle damage (such as abrasions, scratches, etc.) and repair solutions (such as painting, replacement, etc.). This simplified processing limits the accuracy of the model in fine classification and assessment of damage degree, and it is difficult to meet the high requirements of accuracy and comprehensiveness of damage assessment business. In addition, YOLO target detection currently only supports single-label target detection and cannot be directly applied to multi-label task scenarios. For example, in the vehicle self-service damage assessment business scenario, it is usually necessary to predict the vehicle's damage, components and repair solutions at the same time. The current common solution is to train three detection models to detect these three types of information respectively, but this will lead to a complex model structure, high resource consumption, and complex post-processing.

[0004] Therefore, improving the accuracy and multi-label detection capability of YOLO target detection in specific tasks has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a multi-label target detection method, apparatus, computer equipment and storage medium to improve the accuracy and multi-label detection capability of YOLO target detection in specific tasks.

[0006] In order to solve the above technical problems, the present application embodiment provides a multi-label target detection method, which adopts the following technical solution:

[0007] A multi-label target detection method, comprising:

[0008] Receive a target detection instruction and obtain an image to be detected, wherein the image to be detected contains a number of targets to be predicted;

[0009] Inputting the image to be detected into a pre-trained multi-label target detection model, wherein the multi-label target detection model includes a basic network layer, a regression prediction layer, and a classification prediction layer;

[0010] In the basic network layer, image features are extracted from the image to be detected to obtain a feature map;

[0011] In the regression prediction layer, the boundaries of each target to be predicted are regressed and predicted based on the feature map to obtain the bounding box information of each target to be predicted;

[0012] In the classification prediction layer, the image to be detected is divided into blocks based on the bounding box information of each target to be predicted to obtain several category blocks;

[0013] Get the category prediction results output by each category block to obtain the multi-label target detection results.

[0014] Furthermore, the basic network layer is built based on the YOLO framework. In the basic network layer, the steps of extracting image features from the image to be detected and obtaining the feature map specifically include:

[0015] Perform local convolution operation on the image to be detected to obtain local features of the image;

[0016] Perform nonlinear transformation on local features of the image to obtain nonlinear image features;

[0017] Pooling operation is performed on nonlinear image features to obtain feature maps.

[0018] Furthermore, in the regression prediction layer, the step of performing regression prediction on the boundary of each target to be predicted based on the feature map to obtain the bounding box information of each target to be predicted specifically includes:

[0019] Divide the feature map into grids to obtain several grid units, each of which contains a target to be predicted.

[0020] Determine an anchor frame that matches the grid unit from a preset anchor frame library to obtain a matching anchor frame;

[0021] Based on the matching anchor box of the grid unit, the bounding box corresponding to the grid unit is determined by regression prediction;

[0022] Get the bounding box information corresponding to each grid unit, and get the bounding box information of each target to be predicted.

[0023] Furthermore, based on the matching anchor box of the grid unit, the step of determining the bounding box corresponding to the grid unit by regression prediction specifically includes:

[0024] Obtain information of the grid unit, obtain grid unit information, and obtain information of the matching anchor box, obtain matching anchor box information;

[0025] Calculate the center coordinate offset value, width scaling factor and height scaling factor between the grid cell and the matching anchor box based on the grid cell information and the matching anchor box information;

[0026] Predict the center point of the bounding box based on the center coordinate offset value;

[0027] Predicting a bounding box width and a bounding box height based on a width scaling factor and a height scaling factor;

[0028] Determines the bounding box based on the bounding box center point, bounding box width, and bounding box height.

[0029] Further, after the step of determining the bounding box based on the bounding box center point, the bounding box width and the bounding box height, the method further includes:

[0030] Calculate the confidence value of the bounding box to obtain a first confidence value;

[0031] Comparing the first confidence value with a preset regression prediction confidence threshold;

[0032] A bounding box whose first confidence value is greater than the regression prediction confidence threshold is determined as a bounding box corresponding to the grid unit.

[0033] Furthermore, in the classification prediction layer, the image to be detected is divided into blocks based on the bounding box information of each target to be predicted to obtain a number of category blocks, specifically including:

[0034] Based on the center point of the bounding box, the width of the bounding box and the height of the bounding box, the image region is divided in the image to be detected to obtain a plurality of image sub-regions;

[0035] Treat each image sub-region as a category block.

[0036] Furthermore, the step of obtaining the category prediction results output by each category block to obtain the multi-label target detection results specifically includes:

[0037] Get the category prediction results output by each category block;

[0038] Calculate the confidence value of the category prediction result output by each category block to obtain a second confidence value;

[0039] Comparing the second confidence value with a preset classification prediction confidence threshold;

[0040] Determine a category block whose second confidence value is greater than a classification prediction confidence threshold, and obtain a target category block;

[0041] Get the category prediction result output by the target category block and obtain the multi-label target detection result.

[0042] In order to solve the above technical problems, the embodiment of the present application also provides a multi-label target detection device, which adopts the following technical solution:

[0043] A multi-label target detection device, comprising:

[0044] An image acquisition module is used to receive a target detection instruction and acquire an image to be detected, wherein the image to be detected contains a number of targets to be predicted;

[0045] An image input module is used to input the image to be detected into a pre-trained multi-label target detection model, wherein the multi-label target detection model includes a basic network layer, a regression prediction layer, and a classification prediction layer;

[0046] A feature extraction module is used to extract image features of the image to be detected in the basic network layer to obtain a feature map;

[0047] A regression prediction module is used to perform regression prediction on the boundaries of each target to be predicted based on the feature map in the regression prediction layer to obtain the boundary box information of each target to be predicted;

[0048] A classification prediction module is used to divide the image to be detected into blocks based on the bounding box information of each target to be predicted in the classification prediction layer to obtain a number of category blocks;

[0049] The prediction result module is used to obtain the category prediction results output by each category block and obtain the multi-label target detection results.

[0050] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0051] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the multi-label target detection method as described in any one of the above items are implemented.

[0052] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0053] A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the multi-label target detection method as described in any one of the above.

[0054] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0055] The present application discloses a multi-label target detection method, device, computer equipment and storage medium, which belongs to the field of artificial intelligence technology. The present application adds a regression prediction layer and adjusts the classification prediction layer structure through a multi-label target detection model built on the YOLO target detection framework, inputs the image to be detected into a pre-trained multi-label target detection model, uses the basic network layer to extract image features, and the regression prediction layer predicts the bounding box of the target. The classification prediction layer performs block processing based on the bounding box and outputs the prediction results of each category block, thereby realizing multi-label target detection. The present application significantly improves the detection accuracy and multi-label detection capability of the model in specific tasks; among them, the regression prediction layer can more accurately regress the bounding box information of the target and reduce the detection error; the classification prediction layer enhances the model's classification capability for multiple targets in complex scenes through block processing, supports the simultaneous detection of multiple targets and assigns multiple labels to each target, thereby effectively responding to multi-label task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 An exemplary system architecture diagram is shown in which the present application can be applied;

[0058] Figure 2 A flowchart of an embodiment of a multi-label target detection method according to the present application is shown;

[0059] Figure 3 A schematic diagram of the structure of an embodiment of a multi-label target detection device according to the present application is shown;

[0060] Figure 4 A schematic structural diagram of an embodiment of a computer device according to the present application is shown. DETAILED DESCRIPTION

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0062] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0063] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0064] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0065] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0066] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0067] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0068] It should be noted that the multi-label target detection method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the multi-label target detection device is generally arranged in the server / terminal device.

[0069] It should be understood that Figure 1 The numbers of terminal devices, networks and servers in the embodiment are merely illustrative. The above system may have any number of terminal devices, networks and servers according to implementation requirements.

[0070] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for multi-label target detection according to the present application. The multi-label target detection method comprises the following steps:

[0071] S201, receiving a target detection instruction and acquiring an image to be detected, wherein the image to be detected contains a number of targets to be predicted.

[0072] Specifically, the system first receives external instructions, starts the detection process, and obtains images containing multiple targets to be predicted. These targets can be objects of different categories, or different instances of the same object, so that the pre-trained multi-label detection model can be used to detect the targets in the image and identify the locations and categories of all targets.

[0073] Taking vehicle damage and corresponding repair plans as an example, the system receives a vehicle damage detection instruction and prepares to analyze the damaged parts and damage conditions of the vehicle. Through a camera or uploaded pictures, the system obtains an image of the vehicle. The image may contain multiple damaged parts, such as doors, wheels, bumpers, etc., as well as the damage conditions of each damaged part.

[0074] S202, inputting the image to be detected into a pre-trained multi-label target detection model, wherein the multi-label target detection model includes a basic network layer, a regression prediction layer, and a classification prediction layer;

[0075] In an embodiment of the present application, a multi-label target detection model is built based on the YOLO target detection framework. YOLO (You Only Look Once) is a fast and accurate target detection framework. Its core idea is to convert the object detection task into a regression problem. Through a convolutional neural network (CNN) directly reasoning on the image, the target detection problem is expressed as a regression problem. The bounding boxes and categories of all targets in the image can be predicted through a single forward propagation. This single processing method enables YOLO to achieve real-time processing speed, which is very suitable for applications such as video analysis and real-time monitoring.

[0076] In a specific embodiment, the multi-label target detection model is built on the basis of the YOLOv7 version framework, and a regression prediction layer is added, and the classification prediction layer structure is adjusted, and the classification prediction layer is transformed into a category prediction based on a category block to realize the multi-label detection function. Among them, the regression prediction layer is responsible for predicting the bounding box information of the target, including the center point coordinates, width and height, or offset and scale, etc.; the classification prediction layer is responsible for predicting the category probability of the target in each bounding box. For multi-label classification, each category of the target in the bounding box has an independent probability output.

[0077] In the loss calculation stage of multi-label target detection model training, the base network layer loss L in the yolov7 framework is retained obj , bounding box loss L box , regression prediction layer loss L cls , regression prediction layer loss L reg , the calculation formula of the loss function Loss of the final multi-label target detection model is as follows:

[0078] Loss = αL obj +βL box +γL cls +δL reg

[0079] Among them, α, β, γ, and δ are hyperparameters used to adjust the balance between the terms in the loss function.

[0080] Specifically, after obtaining the image to be detected, the system will adjust the image to the fixed size required by the YOLO network, such as 416x416 or 608x608, and then input the adjusted image to be detected into the pre-trained multi-label target detection model.

[0081] S203: In the basic network layer, image features are extracted from the image to be detected to obtain a feature map.

[0082] Specifically, in the basic network layer, the system performs convolution operations, feature nonlinear transformation operations, and pooling operations on the image to be detected. After these convolution, nonlinear transformation, and pooling operations, the system outputs a series of feature maps that contain feature information of different scales and levels in the image.

[0083] S204: In the regression prediction layer, regression prediction is performed on the boundary of each target to be predicted based on the feature map to obtain the boundary box information of each target to be predicted.

[0084] Specifically, in the regression prediction layer, the system first divides the input feature map into multiple grids, and each grid unit is responsible for detecting whether the center point of the target in the image falls within the grid unit. Then, the bounding box regression prediction is implemented based on the anchor box mechanism (Anchor Box) in the YOLOv7 detection framework. The anchor box mechanism defines multiple preset anchor boxes (AnchorBoxes) for regression prediction of the bounding box of the target. Each grid unit will refer to multiple anchor boxes during the detection process and regress the actual target box corresponding to each anchor box.

[0085] S205 , in the classification prediction layer, the image to be detected is divided into blocks based on the bounding box information of each target to be predicted to obtain a plurality of category blocks.

[0086] Specifically, block processing refers to using the previously predicted bounding box information to divide the image into different areas (blocks), which correspond to candidate targets of different categories. Each category block can be understood as an area surrounding a certain bounding box, which will be further classified. In the classification prediction layer, the system performs block processing on the image to be detected based on the bounding box information of each target to be predicted, and obtains several category blocks.

[0087] S206, obtaining the category prediction results output by each category block to obtain a multi-label target detection result.

[0088] Specifically, the YOLO framework usually predicts the category distribution for each bounding box, and the classifier will make classification predictions for each category block and output a category probability distribution, which indicates the probability that this category block (region) belongs to each category. For the entire image, the system will combine the prediction results of all category blocks to form the final multi-label detection result. For some targets, there may be multiple prediction results (multiple category blocks). The final output is based on the classification results of all category blocks. By comprehensively considering factors such as confidence and category probability, the final target detection result is generated.

[0089] In the above embodiments, the present application significantly improves the detection accuracy and multi-label detection capability of the model in specific tasks; among them, the regression prediction layer can more accurately regress the bounding box information of the target and reduce detection errors; the classification prediction layer enhances the model's classification capability for multiple targets in complex scenes through block processing, supports simultaneous detection of multiple targets and assigns multiple labels to each target, thereby effectively responding to multi-label task requirements.

[0090] Furthermore, the basic network layer is built based on the YOLO framework. In the basic network layer, the steps of extracting image features from the image to be detected and obtaining the feature map specifically include:

[0091] Perform local convolution operation on the image to be detected to obtain local features of the image;

[0092] Perform nonlinear transformation on local features of the image to obtain nonlinear image features;

[0093] Pooling operation is performed on nonlinear image features to obtain feature maps.

[0094] In the above embodiments, the base network layer is usually composed of a series of convolutional layers, activation layers (such as ReLU or LeakyReLU) and pooling layers. First, the image is processed by a convolutional layer, which uses multiple convolutional kernels (filters) to scan the image to extract features. These convolutional kernels can detect different image features, such as edges, textures, etc. After the convolutional layer, an activation function (such as Leaky ReLU) is applied to introduce nonlinear characteristic transformations so that the network can learn more complex features. After the convolution operation, a pooling layer (such as maximum pooling or average pooling) is added to downsample and reduce the size of the feature map, thereby reducing the amount of calculation and controlling overfitting. After these convolution, activation and pooling operations, the base network layer outputs a series of feature maps, which contain feature information of different scales and levels in the image, and are the basis for subsequent target detection and classification.

[0095] Through the above steps, in the basic network layer of the YOLO framework, the process of image feature extraction of the image to be detected is completed through a series of convolution operations, activation function application and pooling layers, and the final feature map captures the key information in the image.

[0096] Furthermore, in the regression prediction layer, the step of performing regression prediction on the boundary of each target to be predicted based on the feature map to obtain the bounding box information of each target to be predicted specifically includes:

[0097] Divide the feature map into grids to obtain several grid units, each of which contains a target to be predicted.

[0098] Determine an anchor frame that matches the grid unit from a preset anchor frame library to obtain a matching anchor frame;

[0099] Based on the matching anchor box of the grid unit, the bounding box corresponding to the grid unit is determined by regression prediction;

[0100] Get the bounding box information corresponding to each grid unit, and get the bounding box information of each target to be predicted.

[0101] In the above embodiment, the system divides the input feature map into grid units of the same size, wherein each grid unit contains a target to be predicted, and each grid unit is responsible for detecting whether the center point of the target in the image falls within the grid unit. Different versions of YOLO use different grid sizes. For example, YOLOv3 uses feature maps of three scales: 13×13, 26×26, and 52×52. Then, bounding box regression prediction is implemented based on the anchor box mechanism. The anchor box mechanism defines multiple preset anchor boxes for regressing the bounding box of the predicted target. Each grid unit will refer to multiple anchor boxes during the detection process and regress the actual bounding box corresponding to each anchor box. Finally, the bounding box information corresponding to each grid unit is obtained to obtain the bounding box information of each target to be predicted.

[0102] Through the above steps, the regression prediction layer directly predicts the bounding box of the target through the grid cells and anchor box mechanism on the feature map. Each grid cell regresses the predicted center position, size and confidence of the target, and combines the preset size of the anchor box to generate the final bounding box. After post-processing non-maximum suppression, the precise bounding box of each target is obtained.

[0103] Furthermore, based on the matching anchor box of the grid unit, the step of determining the bounding box corresponding to the grid unit by regression prediction specifically includes:

[0104] Obtain information of the grid unit, obtain grid unit information, and obtain information of the matching anchor box, obtain matching anchor box information;

[0105] Calculate the center coordinate offset value, width scaling factor and height scaling factor between the grid cell and the matching anchor box based on the grid cell information and the matching anchor box information;

[0106] Predict the center point of the bounding box based on the center coordinate offset value;

[0107] Predicting a bounding box width and a bounding box height based on a width scaling factor and a height scaling factor;

[0108] Determines the bounding box based on the bounding box center point, bounding box width, and bounding box height.

[0109] In the above embodiment, for each anchor box, YOLO predicts the following three key parameters to regress the bounding box of the target, including the center coordinate offset value, the width scaling factor, and the height scaling factor. Among them, the center coordinate offset value represents the offset of the center of the bounding box relative to the upper left corner of the grid unit, which is usually limited to between 0 and 1 by the Sigmoid function. The width scaling factor and the height scaling factor are used to predict the width and height of the target bounding box, respectively. The model predicts the scaling ratio of the width and height relative to the anchor box, which is usually achieved through exponential transformation.

[0110] In target detection, first, the system needs to predict the precise position of the target, that is, the center point of the bounding box. The center coordinate offset value represents the offset of the target bounding box center relative to the upper left corner of the grid unit. This is done through a regression prediction, and the offset value is limited to the grid unit through the Sigmoid function to ensure that the predicted bounding box is within the valid area of ​​the image. Then, the system determines the width scaling factor and height scaling factor by calculating the width ratio and height ratio of the grid unit and the matching anchor box. By calculating the width and height scaling factors, the model can adjust the size of the anchor box to make it more consistent with the size of the actual target.

[0111] Finally, the system predicts the center point of the bounding box based on the center coordinate offset value, and predicts the bounding box width and bounding box height based on the width scaling factor and the height scaling factor.

[0112] Among them, the formulas for the predicted bounding box center point (bx, by) are as follows:

[0113] bx=σ(tx)+cx

[0114] by=σ(ty)+cy

[0115] Where tx is the offset value of the center coordinate x-axis, ty is the offset value of the center coordinate y-axis, σ(tx) and σ(ty) are the offsets of tx and ty after Sigmoid activation, respectively, and cx and cy are the horizontal and vertical coordinates of the upper left corner of the grid unit.

[0116] The formulas for the predicted bounding box width and height (bw, bh) are as follows:

[0117] bw=pw+e tw

[0118] bh=ph+e th

[0119] Where pw and ph are the width and height respectively, tw and th are the width scaling factor and height scaling factor respectively.

[0120] Through the above steps, YOLO's regression prediction layer predicts the bounding box of the target through the grid cells and anchor box mechanism on the feature map. Each grid cell regresses the predicted center position, size, and confidence of the target, and combines the preset size of the anchor box to generate the final bounding box.

[0121] Further, after the step of determining the bounding box based on the bounding box center point, the bounding box width and the bounding box height, the method further includes:

[0122] Calculate the confidence value of the bounding box to obtain a first confidence value;

[0123] Comparing the first confidence value with a preset regression prediction confidence threshold;

[0124] A bounding box whose first confidence value is greater than the regression prediction confidence threshold is determined as a bounding box corresponding to the grid unit.

[0125] In the above embodiment, for each bounding box, YOLO also predicts the confidence score of whether the bounding box contains the target, and the confidence is the IoU (intersection over union) between the bounding box and the actual target multiplied by the probability that the target exists in the area. Specifically, the system calculates the confidence value of each bounding box to obtain a first confidence value, compares each first confidence value with a preset regression prediction confidence threshold, and determines the bounding box whose first confidence value is greater than the regression prediction confidence threshold as the bounding box corresponding to the grid unit.

[0126] In addition, the system uses the non-maximum suppression (NMS) method to remove overlapping bounding boxes and retain only the boxes with the highest confidence, which can reduce redundant detection and improve the accuracy of bounding boxes.

[0127] Through the above steps, the confidence of the bounding box is calculated and compared with the preset threshold, and the bounding box with high confidence is effectively screened out as the final recognition result of the grid unit, which significantly improves the accuracy of target detection, reduces false detection and missed detection, and optimizes detection efficiency and performance.

[0128] Furthermore, in the classification prediction layer, the image to be detected is divided into blocks based on the bounding box information of each target to be predicted to obtain a number of category blocks, specifically including:

[0129] Based on the center point of the bounding box, the width of the bounding box and the height of the bounding box, the image region is divided in the image to be detected to obtain a plurality of image sub-regions;

[0130] Treat each image sub-region as a category block.

[0131] In the above embodiment, in the previous regression prediction layer, the system has obtained the bounding box information of each target, including the center point, width, height, etc. These bounding boxes roughly locate the position of the target in the image. Then, the system uses the previously predicted bounding box information to divide the image into different areas, which correspond to candidate targets of different categories. Each category block is generated based on the detected bounding box information, that is, based on the center coordinates, width and height of each bounding box, a sub-area in the image is extracted, and this sub-area becomes a category block. For example, for a bounding box, the center point (bx, by) of the box and the width bw and height bh of the box will be used to extract the image content of the area, and this part of the image will be used as a category block.

[0132] Each category block actually corresponds to a candidate category area. These areas will be sent to the classifier for classification to determine which category the target in each block belongs to. The YOLO framework usually uses the results of bounding box prediction combined with the global information of the feature map for classification prediction.

[0133] Through the above steps, the image is divided into multiple category blocks, which represent areas that may contain objects. Then, the classifier is used to make classification predictions for each category block, determine the target category in each block, use feature maps and bounding box information to determine the category of the target, and combine the context information of the entire image to output the final multi-label detection result.

[0134] Furthermore, the step of obtaining the category prediction results output by each category block to obtain the multi-label target detection results specifically includes:

[0135] Get the category prediction results output by each category block;

[0136] Calculate the confidence value of the category prediction result output by each category block to obtain a second confidence value;

[0137] Comparing the second confidence value with a preset classification prediction confidence threshold;

[0138] Determine a category block whose second confidence value is greater than a classification prediction confidence threshold, and obtain a target category block;

[0139] Get the category prediction result output by the target category block and obtain the multi-label target detection result.

[0140] In the above embodiment, first, the system predicts each category block through a classifier to obtain its corresponding category distribution. For example, the classifier will output a probability vector, indicating the probability that the category block belongs to each category. Then, in the multi-label detection task, a single category block may correspond to multiple labels, and each label will have a confidence value. The confidence value indicates the confidence of the model in the category prediction result. Usually, the confidence value can be calculated by the maximum probability of the predicted category or other measurement methods. Next, the system will compare the confidence value of each category block with the classification prediction confidence threshold, and filter out the category blocks with confidence values ​​higher than the threshold, and consider these blocks to contain detection results with higher confidence. Through confidence comparison, it is determined that those category blocks with confidence values ​​greater than the threshold are target category blocks, and these category blocks are considered to be valid detection results in the detection task. Finally, the system uses the category prediction results of the filtered target category blocks as the final multi-label target detection results, retaining only those predictions with sufficiently high confidence values ​​exceeding the threshold, and outputting the final results of multi-label target detection.

[0141] Through the above steps, by calculating the confidence and performing threshold screening, the system can filter out low-confidence detection results, thereby improving detection accuracy and reducing false detections. Through multi-label confidence determination, the system can accurately detect multiple categories of targets in complex scenes and perform multi-label classification on each target.

[0142] In the above embodiment, the present application discloses a multi-label target detection method, which belongs to the field of artificial intelligence technology. The present application adds a regression prediction layer and adjusts the classification prediction layer structure through a multi-label target detection model built based on the YOLO target detection framework, inputs the image to be detected into a pre-trained multi-label target detection model, uses the basic network layer to extract image features, and the regression prediction layer predicts the bounding box of the target. The classification prediction layer performs block processing based on the bounding box and outputs the prediction results of each category block, thereby realizing multi-label target detection. The present application significantly improves the detection accuracy and multi-label detection capability of the model in specific tasks; among them, the regression prediction layer can more accurately regress the bounding box information of the target and reduce the detection error; the classification prediction layer enhances the model's classification capability for multiple targets in complex scenes through block processing, supports the simultaneous detection of multiple targets and assigns multiple labels to each target, thereby effectively responding to multi-label task requirements.

[0143] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure) can receive instructions or obtain data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0144] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned image information to be detected, the above-mentioned image information to be detected can also be stored in a node of a blockchain.

[0145] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0146] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0147] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0148] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0149] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0150] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a multi-label target detection device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0151] like Figure 3 As shown, the multi-label target detection device 300 described in this embodiment includes:

[0152] The image acquisition module 301 is used to receive a target detection instruction and acquire an image to be detected, wherein the image to be detected contains a number of targets to be predicted;

[0153] An image input module 302 is used to input the image to be detected into a pre-trained multi-label target detection model, wherein the multi-label target detection model includes a basic network layer, a regression prediction layer, and a classification prediction layer;

[0154] A feature extraction module 303 is used to extract image features of the image to be detected in the basic network layer to obtain a feature map;

[0155] A regression prediction module 304 is used to perform regression prediction on the boundary of each target to be predicted based on the feature map in the regression prediction layer to obtain the boundary box information of each target to be predicted;

[0156] The classification prediction module 305 is used to perform block processing on the image to be detected based on the bounding box information of each target to be predicted in the classification prediction layer to obtain a plurality of category blocks;

[0157] The prediction result module 306 is used to obtain the category prediction results output by each category block to obtain the multi-label target detection results.

[0158] Furthermore, the basic network layer is built based on the YOLO framework, and the feature extraction module 303 is specifically used for:

[0159] A local convolution operation is performed on the image to be detected to obtain local features of the image; a feature nonlinear transformation is performed on the local features of the image to obtain nonlinear image features; a pooling operation is performed on the nonlinear image features to obtain a feature map.

[0160] Furthermore, the regression prediction module 304 is specifically used for:

[0161] The feature map is grid-divided to obtain several grid units, each of which contains a target to be predicted; an anchor box matching the grid unit is determined from a preset anchor box library to obtain a matching anchor box; based on the matching anchor box of the grid unit, a bounding box corresponding to the grid unit is determined by regression prediction; the bounding box information corresponding to each grid unit is obtained to obtain the bounding box information of each target to be predicted.

[0162] Furthermore, the regression prediction module 304 is further used to:

[0163] Obtain information of the grid unit to obtain the grid unit information, and obtain information of the matching anchor box to obtain the matching anchor box information; calculate the center coordinate offset value, width scaling factor and height scaling factor between the grid unit and the matching anchor box based on the grid unit information and the matching anchor box information; predict the center point of the bounding box based on the center coordinate offset value; predict the bounding box width and the bounding box height based on the width scaling factor and the height scaling factor; determine the bounding box based on the bounding box center point, the bounding box width and the bounding box height.

[0164] Furthermore, the regression prediction module 304 is also used for:

[0165] Calculate the confidence value of the bounding box to obtain a first confidence value; compare the first confidence value with a preset regression prediction confidence threshold; and determine the bounding box whose first confidence value is greater than the regression prediction confidence threshold as the bounding box corresponding to the grid unit.

[0166] Furthermore, the classification prediction module 305 is specifically used for:

[0167] Based on the center point of the bounding box, the width of the bounding box and the height of the bounding box, the image region is divided in the image to be detected to obtain a plurality of image sub-regions;

[0168] Treat each image sub-region as a category block.

[0169] Furthermore, the prediction result module 306 is specifically used for:

[0170] Get the category prediction results output by each category block;

[0171] Calculate the confidence value of the category prediction result output by each category block to obtain a second confidence value;

[0172] Comparing the second confidence value with a preset classification prediction confidence threshold;

[0173] Determine a category block whose second confidence value is greater than a classification prediction confidence threshold, and obtain a target category block;

[0174] Get the category prediction result output by the target category block and obtain the multi-label target detection result.

[0175] In the above embodiment, the present application discloses a multi-label target detection device, which belongs to the field of artificial intelligence technology. The present application adds a regression prediction layer and adjusts the classification prediction layer structure through a multi-label target detection model built based on the YOLO target detection framework, inputs the image to be detected into a pre-trained multi-label target detection model, uses the basic network layer to extract image features, and the regression prediction layer predicts the bounding box of the target. The classification prediction layer performs block processing based on the bounding box and outputs the prediction results of each category block, thereby realizing multi-label target detection. The present application significantly improves the detection accuracy and multi-label detection capability of the model in specific tasks; among them, the regression prediction layer can more accurately regress the bounding box information of the target and reduce detection errors; the classification prediction layer enhances the model's classification capability for multiple targets in complex scenes through block processing, supports simultaneous detection of multiple targets and assigns multiple labels to each target, thereby effectively responding to multi-label task requirements.

[0176] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0177] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0178] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0179] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a multi-label target detection method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0180] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the multi-label target detection method.

[0181] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0182] In the above embodiment, the present application discloses a computer device, which belongs to the field of artificial intelligence technology. The present application adds a regression prediction layer and adjusts the classification prediction layer structure through a multi-label target detection model built based on the YOLO target detection framework, inputs the image to be detected into the pre-trained multi-label target detection model, uses the basic network layer to extract image features, and the regression prediction layer predicts the bounding box of the target. The classification prediction layer performs block processing based on the bounding box and outputs the prediction results of each category block, thereby realizing multi-label target detection. The present application significantly improves the detection accuracy and multi-label detection capability of the model in specific tasks; among them, the regression prediction layer can more accurately regress the bounding box information of the target and reduce the detection error; the classification prediction layer enhances the model's classification capability for multiple targets in complex scenes through block processing, supports the simultaneous detection of multiple targets and assigns multiple labels to each target, thereby effectively responding to multi-label task requirements.

[0183] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the multi-label target detection method as described above.

[0184] In the above embodiment, the present application discloses a computer-readable storage medium, which belongs to the field of artificial intelligence technology. The present application adds a regression prediction layer and adjusts the classification prediction layer structure through a multi-label target detection model built on the YOLO target detection framework, inputs the image to be detected into the pre-trained multi-label target detection model, uses the basic network layer to extract image features, and the regression prediction layer predicts the bounding box of the target. The classification prediction layer performs block processing based on the bounding box and outputs the prediction results of each category block, thereby realizing multi-label target detection. The present application significantly improves the detection accuracy and multi-label detection capability of the model in specific tasks; among them, the regression prediction layer can more accurately regress the bounding box information of the target and reduce the detection error; the classification prediction layer enhances the model's classification capability for multiple targets in complex scenes through block processing, supports the simultaneous detection of multiple targets and assigns multiple labels to each target, thereby effectively responding to multi-label task requirements.

[0185] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0186] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0187] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A multi-label target detection method, characterized in that: include: Receiving a target detection instruction and acquiring an image to be detected, wherein the image to be detected contains a plurality of targets to be predicted; Inputting the image to be detected into a pre-trained multi-label target detection model, wherein the multi-label target detection model includes a basic network layer, a regression prediction layer, and a classification prediction layer; In the basic network layer, image features are extracted from the image to be detected to obtain a feature map; In the regression prediction layer, regression prediction is performed on the boundary of each of the objects to be predicted based on the feature map to obtain the boundary box information of each of the objects to be predicted; In the classification prediction layer, the image to be detected is divided into blocks based on the bounding box information of each target to be predicted to obtain a plurality of category blocks; The category prediction results output by each of the category blocks are obtained to obtain a multi-label target detection result.

2. The multi-label target detection method according to claim 1, characterized in that: The basic network layer is built based on the YOLO framework. In the basic network layer, the step of extracting image features from the image to be detected to obtain a feature map specifically includes: Performing a local convolution operation on the image to be detected to obtain local features of the image; Performing feature nonlinear transformation on the local features of the image to obtain nonlinear image features; A pooling operation is performed on the nonlinear image features to obtain the feature map.

3. The multi-label target detection method according to claim 1, characterized in that: The step of performing regression prediction on the boundaries of each target to be predicted based on the feature map in the regression prediction layer to obtain the boundary box information of each target to be predicted specifically includes: Gridding the feature map to obtain a plurality of grid units, wherein each of the grid units contains one target to be predicted; Determine an anchor frame matching the grid unit from a preset anchor frame library to obtain a matching anchor frame; Based on the matching anchor box of the grid unit, determining the bounding box corresponding to the grid unit by regression prediction; The bounding box information corresponding to each of the grid units is obtained to obtain the bounding box information of each of the targets to be predicted.

4. The multi-label target detection method according to claim 3, characterized in that: The step of determining the bounding box corresponding to the grid unit by regression prediction based on the matching anchor box of the grid unit specifically includes: Acquire the information of the grid unit to obtain the grid unit information, and acquire the information of the matching anchor box to obtain the matching anchor box information; Calculate a center coordinate offset value, a width scaling factor, and a height scaling factor between the grid unit and the matching anchor frame based on the grid unit information and the matching anchor frame information; Predicting a center point of a bounding box based on the center coordinate offset value; Predicting a bounding box width and a bounding box height based on a width scaling factor and a height scaling factor; The bounding box is determined based on the bounding box center point, the bounding box width, and the bounding box height.

5. The multi-label target detection method according to claim 4, characterized in that: After the step of determining the bounding box based on the bounding box center point, the bounding box width and the bounding box height, the step further includes: Calculating a confidence value of the bounding box to obtain a first confidence value; Comparing the first confidence value with a preset regression prediction confidence threshold; A bounding box whose first confidence value is greater than the regression prediction confidence threshold is determined as a bounding box corresponding to the grid unit.

6. The multi-label target detection method according to claim 4, characterized in that: The step of performing block processing on the image to be detected based on the bounding box information of each target to be predicted in the classification prediction layer to obtain a plurality of category blocks specifically includes: Based on the center point of the bounding box, the width of the bounding box and the height of the bounding box, an image region is divided in the image to be detected to obtain a plurality of image sub-regions; Each of the image sub-regions is regarded as a category block.

7. The multi-label target detection method according to claim 1, characterized in that: The step of obtaining the category prediction results output by each of the category blocks to obtain the multi-label target detection results specifically includes: Obtaining the category prediction results output by each of the category blocks; Calculating the confidence value of the category prediction result output by each of the category blocks to obtain a second confidence value; Comparing the second confidence value with a preset classification prediction confidence threshold; Determine a category block whose second confidence value is greater than the classification prediction confidence threshold, and obtain a target category block; Obtain the category prediction result output by the target category block to obtain the multi-label target detection result.

8. A multi-label target detection device, characterized in that: include: An image acquisition module is used to receive a target detection instruction and acquire an image to be detected, wherein the image to be detected contains a number of targets to be predicted; An image input module, used to input the image to be detected into a pre-trained multi-label target detection model, wherein the multi-label target detection model includes a basic network layer, a regression prediction layer and a classification prediction layer; A feature extraction module, used to extract image features of the image to be detected in the basic network layer to obtain a feature map; A regression prediction module, used to perform regression prediction on the boundaries of each of the targets to be predicted based on the feature map in the regression prediction layer to obtain boundary box information of each of the targets to be predicted; A classification prediction module, used for performing block processing on the image to be detected based on the bounding box information of each target to be predicted in the classification prediction layer to obtain a plurality of category blocks; The prediction result module is used to obtain the category prediction results output by each category block to obtain a multi-label target detection result.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the multi-label target detection method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the multi-label target detection method according to any one of claims 1 to 7 are implemented.

Images