Object detection method and device and medium

Abstract

The invention provides an object detection method and device and a medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a target image and a target detection model, and the target detection model comprises a detection module and a prediction module; inputting the target image into a detection module for feature extraction, performing feature processing based on an attention mechanism on the obtained image features to obtain attention features, and performing classification prediction and bounding box regression based on the attention features to obtain respective class logic information and frame position information of a plurality of candidate frames; and inputting the attention features, the class logic information and the frame position information into a prediction module, and performing de-duplication prediction on the plurality of candidate frames for a target object in the target image to obtain an object detection result. The object detection result is used for indicating the position of the prediction frame corresponding to the target object predicted by the prediction module in the plurality of candidate frames and the object category of the prediction frame. The object detection accuracy can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an object detection method, device and medium. BACKGROUND

[0002] A target detection model is an artificial intelligence algorithm applied to object detection in an image, aiming to predict a set of target objects and their categories and bounding boxes from the image. Existing target detection models usually generate multiple redundant and nearly repeated prediction results for each target object in the image, and use Non-Maximum Suppression (NMS) to eliminate the most overlapping predictions in inference, but this approach tends to retain the candidate box with the highest confidence, ignoring the importance of overlapping regions, which is not conducive to object detection of densely packed and mutually overlapping objects, and the detection effect is poor. SUMMARY

[0003] The present application provides an object detection method, device and medium, which can significantly improve the information expression accuracy and comprehensiveness of object detection.

[0004] In one aspect, the present application provides an object detection method, which comprises:

[0005] obtaining a target image and a target detection model, the target detection model comprising a detection module and a prediction module;

[0006] inputting the target image into the detection module for feature extraction, performing feature processing based on an attention mechanism on the obtained image features to obtain attention features, and performing classification prediction and bounding box regression based on the attention features to obtain class logic information and box position information of each of the multiple candidate boxes;

[0007] inputting the attention features, the class logic information and the box position information into the prediction module, performing deduplication prediction of the multiple candidate boxes for target objects in the target image to obtain an object detection result, the object detection result being used to indicate the position of a prediction box corresponding to the target object in the multiple candidate boxes and the object category of the prediction box predicted by the prediction module.

[0008] In another aspect, an object detection device is provided, comprising:

[0009] an obtaining module configured to obtain a target image and a target detection model, the target detection model comprising a detection module and a prediction module;

[0010] The detection module is configured to input the target image into the detection module to extract features, perform feature processing on the obtained image features based on an attention mechanism to obtain attention features, and perform classification prediction and bounding box regression based on the attention features to obtain class logic information and box position information of each of the plurality of candidate boxes.

[0011] The prediction module is configured to input the attention features, the class logic information, and the box position information into the prediction module, perform deduplication prediction on the plurality of candidate boxes for a target object in the target image to obtain an object detection result, and the object detection result is used to indicate a position of a prediction box corresponding to the target object in the plurality of candidate boxes and an object category of the prediction box predicted by the prediction module.

[0012] In another aspect, a computer device is provided, which includes a processor and a memory, and the memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the object detection method as described above.

[0013] In another aspect, a computer readable storage medium is provided, which stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the object detection method as described above.

[0014] In another aspect, a server is provided, which includes a processor and a memory, and the memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the object detection method as described above.

[0015] In another aspect, a terminal is provided, which includes a processor and a memory, and the memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the object detection method as described above.

[0016] In another aspect, a computer program product or computer program is provided, which includes computer instructions executed by a processor to implement the object detection method as described above.

[0017] The object detection method, device, equipment, storage medium, server, terminal, computer program, and computer program product provided in the application have the following technical effects:

[0018] The target detection model used in the technical solution of the present application comprises a detection module and a prediction module. The detection module performs feature processing and preliminary prediction on a target image based on an attention mechanism to obtain a plurality of candidate boxes and corresponding attention features, class logic information and box position information. An additional prediction module is used to perform de-duplication prediction based on the attention features, the class logic information and the box position information, taking the attention features, the class logic information and the box position information as inputs, combining the class logic information and the box position information, to determine the prediction box and the category of the target object from the plurality of candidate boxes. The scheme uses the strong relationship modeling capability of the attention mechanism to regard object detection as a set prediction problem, avoids the operation mode of manual customization rules such as NMS post-processing, simultaneously performs de-duplication prediction based on the learnable prediction module, and does not need to perform one-to-one binary matching, thereby improving the object detection capability of the overlapping region and the target dense image.

[0019] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0021] Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application;

[0022] Figure 2 is a flowchart of an object detection method provided by an embodiment of the present application;

[0023] Figure 3 is a structural block diagram of a target detection model provided by an embodiment of the present application;

[0024] Figure 4 is a flowchart of a training method of a target detection model provided by an embodiment of the present application;

[0025] Figure 5 is a structural and training framework diagram of a target detection model provided by an embodiment of the present application;

[0026] Figure 6 is a flowchart of construction of a candidate prediction set of a label assigner provided by an embodiment of the present application;

[0027] Figure 7 is a group of effect comparison diagrams provided by an embodiment of the present application;

[0028] Figure 8 is another set of effect comparison graphs provided by an embodiment of the present application.

[0029] Figure 9 is a framework schematic diagram of an object detection device provided by an embodiment of the present application.

[0030] Figure 10 is a hardware structure block diagram of an electronic device for executing an object detection method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0032] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or sub-modules does not necessarily have to be limited to those steps or sub-modules clearly listed, but can include other steps or sub-modules not clearly listed or inherent to these processes, methods, products or devices.

[0033] Please refer to Figure 1 , Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application, as Figure 1 shown, the application environment can at least include a server 01.

[0034] The server 01 in the embodiments of the present application can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0035] Specifically, the server 01 involved above can include a physical device, which can specifically include a network communication submodule, a processor, a memory, and the like, or can include software running in the physical device, which can specifically include an application program and the like.

[0036] In the embodiments of the present application, the server 01 can be configured to obtain a target image and a target detection model, the target detection model including a detection module and a prediction module, input the target image into the detection module for feature extraction, perform feature processing based on an attention mechanism on the obtained image features to obtain attention features, and perform classification prediction and boundary box regression based on the attention features to obtain class logic information and frame position information of each of a plurality of candidate boxes; and input the attention features, the class logic information, and the frame position information into the prediction module to perform deduplication prediction on the plurality of candidate boxes for target objects in the target image to obtain an object detection result. Specifically, the server 01 can also be configured to provide a training service based on an initial detection model to obtain the target detection model.

[0037] In addition, it can be understood that Figure 1 The application environment shown is only an application environment of the object detection method, and the application environment can include more or fewer nodes, which are not limited in the present application. For example, a terminal can also be included, and the terminal and the server 01 can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application. Specifically, the terminal can include a physical device such as a smart phone, a desktop computer, a tablet computer, a notebook computer, a digital assistant, an augmented reality (AR) / virtual reality (VR) device, a smart voice interaction device, a smart home appliance, a smart wearable device, and a vehicle terminal device, or can include software running in the physical device, such as an application program and the like.

[0038] It can be understood that in the detailed description of the present application, the data related to the sample image and the target image are involved, and when the embodiments of the present application are applied to specific products or technologies, the permission or consent of the user side and the related party is required, and the collection, use, and processing of the related data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0039] The technical solutions of the present application are introduced below based on the above application environment. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, intelligent transportation, and assisted driving. Please refer to Figure 2 , Figure 2is a flowchart of an object detection method provided by an embodiment of the present application. The present specification provides method operation steps as embodiments or flowcharts, but more or fewer operation steps can be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is only one of the many execution orders of the steps, and does not represent the only execution order. In actual system or server product execution, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, as shown in Figure 2 The method can include the following steps S201-S205:

[0040] S201: Obtain a target image and a target detection model.

[0041] Specifically, the target image is an image with object detection requirements, which can be a picture or a video. If it is a video, key frame extraction is performed on the video, and then the target detection model is input.

[0042] Specifically, the target detection model includes a detection module and a prediction module. The input of the detection module is the target image, and the output is the features obtained by processing the target image, and the multiple candidate boxes and class logic information of the candidate boxes obtained by preliminary prediction. The input of the prediction module is the output of the detection module, and the output of the prediction module is the object detection result.

[0043] S203: Input the target image into the detection module for feature extraction, perform feature processing based on the attention mechanism on the obtained image features to obtain attention features, and perform classification prediction and boundary box regression based on the attention features to obtain class logic information and frame position information of each candidate box.

[0044] Specifically, the attention features include the position and category information of the potential candidate boxes in the target image. The candidate box is used to indicate the candidate region corresponding to the object query in the target image predicted by the detection module. The class logic information is used to indicate the class probability distribution of the candidate box belonging to each category. The frame position information is used to indicate the coordinate position of the candidate box in the target image. Exemplarily, it can include vertex coordinates and reference point coordinates (such as center point coordinates).

[0045] In some embodiments, the reference Figure 3 and Figure 5The detection module includes a backbone network and a compiler network (Transformer). The backbone network is configured to extract features of the input target image to obtain image features, and the image features include deep features of the target image. For example, the backbone network can be a convolutional neural network (CNN), such as ResNet.

[0046] S205: input the attention feature, the class information and the bounding box position information into the prediction module, and perform deduplication prediction on the multiple candidate bounding boxes for the target object in the target image to obtain an object detection result.

[0047] Specifically, the object detection result is used to indicate the position of the prediction bounding box corresponding to the target object in the multiple candidate bounding boxes and the object category of the prediction bounding box predicted by the prediction module. The target object refers to an object, such as an article object or a living object, carried in the target image and needed to be detected. The prediction bounding box is used to indicate the region of the target object in the target image, the position of the prediction bounding box is used to indicate the position of the image region of the target object in the target image, and the object category is used to indicate the category of the target object in the prediction bounding box, such as “dog” and the like.

[0048] Specifically, the prediction module is constructed based on a neural network, takes the attention feature output by the decoder as input, and takes the class logic information and the box position information of the candidate box of the object query output by the detection module as auxiliary information, combines the initial class prediction likelihood score information and the detection score generated in the prediction process of the prediction module to obtain the final object detection result.

[0049] In an embodiment, the attention feature can include self-attention features between object queries output by a self-attention layer and cross-attention features between image embedding features and object queries output by a cross-attention layer. The semantic part input to the prediction module includes the self-attention features, the cross-attention features, and the class logic, and the position information part input to the prediction module includes the box position information of the candidate box (such as the reference point and the vertex coordinates of the candidate box). Each component (the self-attention features, the cross-attention features, the class logic, and the box position information) is flattened into a one-dimensional vector, and each one-dimensional vector is connected to obtain an output X, which is taken as the input of the prediction module to obtain the object detection result. Since there is a dimension imbalance problem in each component, in the process of generating the input X, each component is multiplied by a coefficient inversely proportional to its dimension d, that is, The reweighting is performed to balance the dimensions of each component.

[0050] It should be noted that, unlike the conventional compiler network, the output of the detection module in the above technical solution is not the predicted box and the class information and position information of the predicted box obtained by the final prediction, but the class probability distribution and position of the candidate box corresponding to each object query, so that the subsequent detection module can further perform a learnable de-duplication prediction, and the attention feature is predicted and updated again in combination with the initial prediction result output by the detection module, and the one-to-one binary matching and the predicted box output are not performed in the detection module stage, so as to improve the stability of iterative adjustment in the training stage, improve the importance of the overlapping area in the training and learning, and further improve the accuracy of the object detection result through the learnable prediction module, and improve the detection capability of the model in the overlapping area and the target dense area.

[0051] In summary, the above technical solution has strong relationship modeling capability based on the attention mechanism, considers the object detection as a set prediction problem, avoids the operation mode of manual customization rules such as NMS post-processing, and performs de-duplication prediction based on the learnable prediction module, without performing one-to-one binary matching, so as to improve the object detection capability in the overlapping area and the target dense image.

[0052] Based on the above part or all of the above, the embodiment of the present application further provides a training method of an object detection model, which is described with reference to Figure 4 , including S301-S309:

[0053] S301: Obtain a plurality of sample images and label information of the sample images.

[0054] Specifically, the sample image includes at least one sample target object, and the sample target object is similar to the target object and can be an article or a living body, etc. The label information includes a category label and a position ground truth of the sample target object. The category label is used to indicate a real category to which the sample target object belongs, such as "person", etc. The position ground truth is used to indicate an image position of a real bounding box that frames the sample target object in the sample image, and the expression form can be similar to the position of the foregoing predicted bounding box.

[0055] S303: In the iterative training process, the object detection is performed on the plurality of sample images as input of the initial detection model to obtain class logic information and bounding box position information of each of a plurality of candidate bounding boxes corresponding to the sample images and sample detection results of the sample images.

[0056] Specifically, the candidate bounding box is similar to the foregoing candidate bounding box, and the class logic information and the bounding box position information of the candidate bounding box are also similar to the class logic information and the bounding box position information of the foregoing candidate bounding box, which will not be described herein. Similar to the object detection result, the sample detection result is used to indicate a sample predicted bounding box corresponding to the sample target object in the plurality of candidate bounding boxes predicted by the initial detection model. The target detection model is obtained by constraint training of the initial detection model in combination with the plurality of sample images and the label information and the label assigner.

[0057] In some embodiments, the initial detection model includes a detection module and a prediction module, and S303 includes S3031-S3032:

[0058] S3031: The sample image is input into the detection module for feature extraction, the obtained sample image feature is processed based on an attention mechanism to obtain attention features of the sample image, and classification prediction and bounding box regression are performed based on the attention features to obtain class logic information and bounding box position information of the plurality of candidate bounding boxes;

[0059] S3032: The attention features, the class logic information and the bounding box position information of the plurality of candidate bounding boxes are input into the prediction module for object prediction to obtain the sample detection result.

[0060] It can be understood that the generation manner of the attention features of the sample image is similar to the generation manner of the attention features of the target image, the generation manner of the class logic information and the bounding box position information of the candidate bounding box is similar to the generation manner of the class logic information and the bounding box position information of the candidate bounding box, and the generation manner of the sample detection result is similar to the generation manner of the object detection result, which will not be described herein.

[0061] In one embodiment, the input of the prediction module is defined as X, and the sample detection result is y∈R qThe one-dimensional vector can be obtained by relationship deduplication through the prediction module to estimate the possibility of retaining a single prediction result for each sample target object. By setting an additional learnable prediction module, combining the output features of the detection module and the preliminary prediction result for further deduplication detection, the prediction module can inherit the prediction ability of the detection module and iteratively update the detection score to obtain more accurate prediction ability and improve object detection accuracy.

[0062] S305: If the current iteration is the first iteration, input the class logical information and the box position information of the plurality of candidate boxes, and the annotation information into the label assigner to perform label assignment between the plurality of candidate boxes and each sample target object, and obtain a target assignment result.

[0063] Specifically, the label assigner is constructed based on a bipartite matching algorithm, which is used to perform bipartite matching between the plurality of candidate boxes and each sample target object based on the class logical information and the box position information in each iteration to obtain an initial candidate box corresponding to each sample target object in the current iteration. In addition, the label assigner is also used to adaptively assign a reference target box to each sample target object based on a non-replacement sampling bipartite matching method in the multiple iterations of iterative training to generate a candidate prediction set of each sample target object, i.e., to generate a candidate prediction set of each real box.

[0064] Specifically, the target assignment result is used to indicate the target candidate box matched by the label assigner to the sample target object in the plurality of candidate boxes in the current iteration, and a prediction class index of the target candidate box. Each sample target object corresponds to a target candidate box, and the prediction class index is used to indicate the probability that the target candidate box belongs to the class label of the sample target object.

[0065] In the first iteration, for each sample target object, the matching result based on the non-replacement sampling bipartite matching method is consistent with the matching result of the bipartite matching method, i.e., it is directly taken as the target matching result to obtain the target candidate box corresponding to each sample target object. In addition, the target candidate box obtained in the first iteration is added to the candidate prediction set of the sample target object, so that each sample target object has at least one reference target box.

[0066] S307: If the current iteration is not the first iteration, use the class logical information and the box position information of the plurality of candidate boxes, and the annotation information as the input of the label assigner, combine the historical target boxes assigned by the label assigner to the sample target objects in the historical iterations, and perform label assignment between the plurality of candidate boxes and each sample target object to obtain a target assignment result.

[0067] Specifically, the historical target frame is a target candidate frame corresponding to a sample target object in a target assignment result of a historical iteration. In combination with the output of the detection module of the current iteration and the historical target frame in the historical iteration, label matching of the current iteration is performed by taking into account the label assignment results of at least two iterations, so that the loss calculation in the iteration can be optimized in combination with historical information, and the training effect is improved.

[0068] In some embodiments, the label assigner constructs a candidate prediction set for each sample target object, and accordingly, S307 includes S3071-S3072:

[0069] S3071: If the current iteration is not the first iteration, the class logic information and the frame position information of the plurality of candidate frames, and the annotation information are input into the label assigner, and label assignment between the plurality of candidate frames and each sample target object is performed to obtain an initial assignment result.

[0070] Specifically, the initial assignment result is used to indicate the initial candidate frame matched by the label assigner for the sample target object in the plurality of candidate frames in the current iteration. The label assignment here is achieved by binary matching, and the binary matching in each iteration is performed by using the full set of the plurality of candidate frames for matching. Understandably, the initial assignment result of the first iteration is the target assignment result. Starting from the second iteration, the target assignment result is obtained by modifying the initial assignment result.

[0071] S3072: The initial assignment result is modified to obtain a target assignment result based on the inclusion relationship between the candidate prediction set corresponding to the sample target object and the initial candidate frame of the sample target object, and the historical target frame of the sample target object in the previous iteration.

[0072] Specifically, the candidate prediction set is obtained by non-replacement sampling label assignment in at least one iteration including the first iteration. The candidate prediction set includes at least one reference target frame obtained by non-replacement sampling label assignment of the label assigner for the sample target object from the plurality of candidate frames in the historical iteration and the current iteration. The non-replacement sampling here is for each iteration in the construction of the candidate prediction set. The candidate frame added to the candidate prediction set in the previous iteration is not put into the candidate frame set for label assignment in the next iteration, so that the reference target frames obtained by each sampling are different from each other.

[0073] Specifically, the candidate prediction set is constructed in combination with the non-replacement label assignment results of the current iteration and the historical iteration, and the assignment results of the previous iteration and the current iteration are considered when judging the inclusion relationship, so that the finally matched target candidate frame can take the historical target frame as a redundant result, breaking the unstable matching result caused by generating top-1 candidate for each real frame in the traditional binary matching, and further avoiding the problem of iteration optimization target shock caused by the unstable result.

[0074] In some embodiments, the initial assignment result, the target assignment result and the candidate prediction set construction all involve a label assignment process, and the label assignment of the present application is performed based on the matching cost between the candidate box and the real box. The greater the matching cost is, the smaller the matching degree between the candidate box and the real box is, and the lower the matching possibility is. Conversely, the greater the matching possibility is. Figure 5 In one embodiment, the matching cost indicates the feature distance between the candidate box and the real box, and the feature distance includes a first matching cost indicating the position distance and a second matching cost indicating the category distance.

[0075] Correspondingly, S3071 can include S30711-S30713:

[0076] S30711: determining a first matching cost between the candidate box and the sample target object based on the box position information of the candidate box and the position ground truth of the sample target object;

[0077] S30712: determining a second matching cost between the candidate box and the sample target object based on the category logic information of the candidate box and the category label of the sample target object;

[0078] S30713: fusing the first matching cost and the second matching cost, and performing bipartite matching between each candidate box and each sample target object based on the obtained matching cost to obtain the initial candidate box corresponding to each sample target object, and generating the initial assignment result.

[0079] Specifically, the first matching cost is used to indicate the position matching degree between the candidate box and the sample target object, and specifically can indicate the positioning loss between the candidate box i and the real box j. The second matching cost is used to indicate the category matching degree between the candidate box and the sample target object, and specifically can indicate the category loss between the candidate box P i and the real box G i . Exemplarily, the positioning loss can be L1 (Mean Absolute Error, MAE) loss or GIoU (Generalized Intersection over Union) loss, etc.; and the category loss can be L1 loss or cross-entropy loss, etc.

[0080] Specifically, the fusion here refers to simply adding or weighted adding the first matching cost and the second matching cost, and the weight can be selected and set based on the training effect or business demand. Through bipartite matching, the sum of the total distances between each candidate box and real box is minimized to obtain the initial candidate box corresponding to each real box. In one embodiment, the matching cost D(P i , G i ) = L cls (P i , Gi )+L box (P i G i ), where L box (P i G i L represents the first matching cost. cls (P i G i The second matching cost is denoted as . By combining the location matching cost and the category matching cost, binary matching is performed, ensuring that the initial candidate boxes encompass the similarity of location information and category prediction information, thus improving the accuracy of one-to-one matching.

[0081] It should be noted that, in the preferred embodiment, the label allocation of the initial allocation result, the label allocation of the target allocation result, and the label allocation of the candidate prediction set all use the same matching cost calculation and binary matching method, except for the difference between sampling with and without replacement.

[0082] In some embodiments, reference is made to Figure 5 In the label assigner section, the construction of the candidate prediction set for the sample target object includes S401-S413:

[0083] S401: Use the candidate bounding boxes of the sample target objects in the first iteration as reference bounding boxes and add them to the candidate prediction set of the sample target objects;

[0084] S403: Calculate the average matching cost between the target object and the target candidate box for each sample in the first iteration to obtain the cost threshold;

[0085] S405: If the matching cost of the reference target box in the first iteration is lower than the cost threshold, the candidate prediction set is determined to be a fixed set;

[0086] S407: If the matching cost of the reference target box in the first iteration is higher than the cost threshold, from the second iteration onwards, based on the class logic information, box position information, and annotation information of multiple candidate boxes, label assignment is performed on each sample target object in multiple candidate boxes without replacement to obtain the initial reference box corresponding to the sample target object.

[0087] S409: Add the initial reference box to the candidate prediction set and determine the mean cost corresponding to the matching cost of each element in the candidate prediction set;

[0088] S411: If the average cost is lower than the cost threshold, the candidate prediction set will be determined as a fixed set;

[0089] S413: If the average cost is higher than the cost threshold, continue to add the initial reference frame corresponding to the sample target object to the candidate prediction set in the subsequent iteration, until the average cost corresponding to the matching cost of each element in the candidate prediction set is lower than the cost threshold, and perform the step of determining the candidate prediction set as the fixed set.

[0090] As described above, in the first round of iteration, the matching result obtained by sampling label matching without replacement is consistent with the initial allocation result obtained by label matching with replacement, i.e., the initial allocation result does not need to be corrected and is determined as the target matching result, and the initial candidate frame is directly used as the target candidate frame. The target candidate frame is also added to the candidate prediction set as an element in the candidate prediction set. After the first round of iteration, the candidate prediction set of each sample target object includes a reference candidate frame to ensure that the candidate prediction set contains at least one element for the redundant prediction result in the subsequent iteration, thereby serving as a backup for the subsequent target candidate frame.

[0091] Specifically, the matching cost in S403 is consistent with the matching cost calculation method of the label matching in the aforementioned S3071. The target candidate frame of each sample target object is obtained in the first round of iteration, the average of the matching cost between each target candidate frame and the real frame is calculated, and the cost threshold τ is obtained, which is used to indicate the upper limit of the average cost of each element in each candidate prediction set, i.e., the average cost of each candidate prediction set formed finally is lower than the cost threshold. Here, lower can mean less than or less than or equal to.

[0092] At this time, each candidate prediction set only includes the reference target frame of the first round of iteration, if the matching cost of the reference target frame is already lower than the cost threshold, the candidate prediction set is determined as the fixed set, and in the subsequent iteration process, the candidate prediction set does not add new elements; if the matching cost of the reference target frame is higher than the cost threshold, step S407 is performed in the second round of iteration. It can be understood that the matching cost of the first round of iteration is lower than the cost threshold, which indicates that the corresponding sample target object obtains better results in the prediction of the first round of iteration, which is a simple object with low prediction difficulty, such as an image area or a common category object. In the iteration training process, the number of redundant predictions in the candidate prediction set is reduced through the cost threshold, avoiding the iteration optimization shock of simple objects, and ensuring the stability of parameter adjustment in the optimization process.

[0093] Specifically, in the second round of iteration, for the sample target object corresponding to the candidate prediction set of the non-fixed set, the matched reference target box in the first round of iteration is removed from the plurality of candidate boxes, the label matching between the real box and the candidate box is performed from the remaining candidate boxes, the initial reference box of this round is obtained, the initial reference box is directly added to the candidate prediction set as the target reference box, and then the cost mean of the matching cost of each target reference box in the candidate prediction set after this round is calculated; the cost mean is compared with the cost threshold value, if lower, it is determined that the optimization of the sample target object in the second round of iteration has developed in a better direction, the candidate prediction set is set as the fixed set, and no new element is added subsequently. If the cost mean is still higher than the cost threshold value, in the next round of iteration, the matched reference target box in each historical iteration (the first round and the second round of iteration) is removed from the plurality of candidate boxes, the label matching between the real box and the candidate box is performed from the remaining candidate boxes, the initial reference box of this round is obtained, and then the cost mean is calculated and compared with the cost threshold value to determine whether the candidate prediction set is the fixed set, and the cycle is repeated to obtain the candidate prediction set of each sample target object whose type is the fixed set. Understandably, the more the rounds of iteration in which the cost mean is higher than the cost threshold value, the greater the average distance between the candidate box obtained by each round of prediction and the real box, indicating that the sample target object is a relatively difficult object with a higher prediction difficulty, such as a small area or rare class object. The cost mean of the candidate prediction set is updated by the initial reference box generated in each iteration, and is compared with the cost mean of the overall sample target object in the first round of iteration, to realize the adaptive capacity limitation of the candidate prediction set based on the object prediction difficulty. More reference target boxes are matched for the more difficult objects to be used as the target candidate box for subsequent iteration update. Understandably, due to the discreteness of the bipartite matching, a slight change in the matching cost matrix formed by each candidate box and each real box may lead to a significant change in the matching result. In the early iteration training process, due to the random optimization of the parameters, the model parameters are updated frequently and drastically, so some object queries may have unstable optimization targets and receive inconsistent gradients, thereby slowing down the convergence speed in the training process. The technical scheme of the present application realizes the adaptive capacity label assignment of the candidate prediction set by combining the cost threshold value calculated based on the matching cost, matches the element capacity suitable for each real box, provides more abundant positive sample assignment labels, so that the historical target box obtained in the historical iteration affects the optimization direction of the current iteration, improves the stability of the optimization target, and improves the convergence speed and training effect.

[0094] In some other embodiments, the construction method of the candidate prediction set of the sample target object includes S401-S411 and S415:

[0095] S401: The target candidate box of the sample target object in the first round of iteration is added to the candidate prediction set of the sample target object as the reference target box;

[0096] S403: Calculate the mean value of the matching cost between each sample target object and the target candidate box in the first round of iteration to obtain a cost threshold;

[0097] S405: If the matching cost of the reference target box in the first round of iteration is lower than the cost threshold, determine the candidate prediction set as the fixed set;

[0098] S407: If the matching cost of the reference target box in the first round of iteration is higher than the cost threshold, from the second round of iteration, based on the class logic information and the box position information of the plurality of candidate boxes, and the label information, perform non-replacement sampling label assignment for each sample target object in the plurality of candidate boxes to obtain the initial reference box corresponding to the sample target object;

[0099] S409: Add the initial reference box to the candidate prediction set, and determine the cost mean value corresponding to the matching cost of each element in the candidate prediction set;

[0100] S411: If the cost mean value is lower than the cost threshold, determine the candidate prediction set as the fixed set;

[0101] S415: If the cost mean value is higher than the cost threshold, continue to add the initial reference box corresponding to the sample target object to the candidate prediction set in the subsequent iteration, until the cost mean value corresponding to the matching cost of each element in the candidate prediction set is lower than the cost threshold, and execute the step of determining the candidate prediction set as the fixed set.

[0102] Specifically, in another specific embodiment, a preset round is set to limit the number of iterations in the construction of the candidate prediction set, and the preset round K is the upper limit of the elements of the candidate prediction set. For a sample target object, if the cost mean value in the second round of iteration is higher than the cost threshold, the subsequent iteration continues to add the reference target box, and in the nth round (n>2), if the reference target box obtained in the nth round is added to the candidate prediction set, the cost mean value of the candidate prediction set has been lower than the cost threshold, or n=K, then the candidate prediction set is determined as the fixed set, and no reference target box is added in the subsequent iteration; if the cost mean value of the candidate prediction set is higher than the cost threshold, and / or n Figure 5 It can be understood that, based on the cost threshold regulation and limitation, the reference target box is added to the candidate prediction set in the iteration, and the candidate prediction set is determined as the fixed set when the cost mean value is lower than the cost threshold or n=K. It can be understood that, based on the cost threshold regulation and limitation, the reference target box is added to the candidate prediction set in the iteration, and the candidate prediction set is determined as the fixed set when the cost mean value is lower than the cost threshold or n=K.

[0103] Thus, by setting the preset rounds to limit the upper limit of the candidate prediction set of the more difficult to predict object, that is, to limit the upper limit of the redundant prediction of each real frame, the label matching for the candidate prediction set is not needed after the preset rounds are reached, the calculation resource consumption of the label matching is reduced, and the change amount of the target distribution result used in the subsequent prediction module training adjustment is reduced, thereby further reducing the calculation resource consumption in the model training process and improving the convergence speed.

[0104] In an example, referring to Figure 6 , the sample image of Fig. a has three sample target objects (a person, a baseball bat, and a baseball), Figs. b-d show the label matching results of non-replacement sampling in three rounds of iterations, the detection module outputs 8 valid candidate frames, which are candidate frames 1-8 from top to bottom in the figure, and there are 3 real frames, which are real frame 1 of the baseball, real frame 2 of the person, and real frame 3 of the baseball bat from top to bottom in the figure. The construction process of the candidate prediction set is as follows: in the first round of iteration, the label matching result is real frame 1-candidate frame 1, real frame 2-candidate frame 4, and real frame 3-candidate frame 5, and the candidate prediction sets 1-3 of real frames 1-3 are {candidate frame 1}, {candidate frame 4}, and {candidate frame 5} in turn, at the same time, the average cost of candidate prediction set 2 (i.e., the matching cost of candidate frame 4) is lower than the cost threshold, candidate prediction set 2 is a fixed set, and the average costs of candidate prediction sets 1 and 3 (i.e., the matching costs of candidate frame 1 and candidate frame 5) are both higher than the cost threshold, and the next round continues to add elements; in the second round of iteration, the non-replacement label matching result is real frame 1-candidate frame 3, real frame 3-candidate frame 7, and the candidate prediction sets 1 and 3 of real frames 1 and 3 are {candidate frame 1, candidate frame 3} and {candidate frame 5, candidate frame 7} in turn, at the same time, the average cost of candidate prediction set 1 is lower than the cost threshold, and candidate prediction set 1 is a fixed set, and the average cost of candidate prediction set 3 is higher than the cost threshold, and the next round continues to add elements; in the third round of iteration, the non-replacement label matching result is real frame 3-candidate frame 6, and the candidate prediction set 3 of real frame 3 is {candidate frame 5, candidate frame 7, candidate frame 6}, at the same time, the average cost of candidate prediction set 3 is lower than the cost threshold, and candidate prediction set 3 is determined as a fixed set, and the construction of the candidate prediction set of each sample target object is completed, and the obtained result is real frame 1-{candidate frame 1, candidate frame 3}, real frame 2-{candidate frame 4}, and real frame 3-{candidate frame 5, candidate frame 7, candidate frame 6}.

[0105] In an example, referring to the predicted instances in the figures, the left figure is the prediction of the conventional DETR, and the right figure is the prediction of the present application. The dashed box is the assigned redundant prediction, and the solid box is the real box. There are two real boxes in the image: a person and a baseball bat. Since the baseball bat is more rare than the human, the model regards it as a difficult sample, i.e., a more difficult object, and adaptively assigns two predictions (two queries) to it because the value of the cost matrix (a matrix formed by matching costs) in the label assignment process is larger. The algorithm adaptively assigns two predictions (two queries) to it, while the DETR only assigns one. Compared with the prior art, the present application adaptively increases the positive query matching of difficult samples to improve the training speed. As for the human, the algorithm regards it as a simple sample, and therefore assigns one query to it.

[0106] In some embodiments, S3072 can include S30721-30722:

[0107] S30721: For each sample target object, if the initial candidate box and the historical target box of the previous iteration both belong to the candidate prediction set of the current iteration, the historical target box of the previous iteration is determined as the target candidate box of the sample target object in the current iteration.

[0108] S30722: For each sample target object, if at least one of the initial candidate box and the historical target box of the previous iteration does not belong to the candidate prediction set of the current iteration, the initial candidate box is determined as the target candidate box of the sample target object in the current iteration.

[0109] Specifically, the historical target box of the previous iteration refers to the target candidate box obtained in the previous iteration, i.e., the target candidate box used in the loss calculation of the previous iteration, and the candidate prediction set of the current iteration refers to a set that already includes the reference target box determined in the current iteration. From the second iteration, in the determination process of the target candidate box, whether the label matching of the two consecutive training rounds exceeds the candidate prediction set is considered, i.e., whether the initial candidate box and the historical target box of the previous iteration both match the consistent reference target box in the candidate prediction set. If so, it indicates that the optimization target corresponding to the sample target object has progressed towards the positive optimization direction, i.e., the parameter adjustment and gradient of the previous iteration have a positive effect on the detection of the sample target object. Therefore, the historical target box used in the loss calculation of the previous iteration is continued to be used as the target candidate box of the sample target object in the current iteration for the loss calculation of the current iteration, which significantly reduces the optimization shock of adjacent iterations and improves the training stability. If the initial candidate box and / or the historical target box of the previous iteration does not belong to the candidate prediction set of the current iteration, it indicates that the tuning effect of the previous iteration for the sample target object is not good, and therefore the initial candidate box of the current round is used as the target candidate box to continue to find a better prediction result, ensuring the training stability while improving the model capability and further improving the object detection effect.

[0110] In an example, for each real box Gi , the label assigner predicts a pair of few-matching output candidate sets S based on adaptive capacity e,i , where e represents a training round, and o e,i satisfies the following formula, where o e-1,i is a historical target box of the previous iteration, and arg min D(P j , G i ) refers to an initial candidate box obtained by label matching with replacement in the e-th round. Understandably, the target candidate box in the first iteration is also obtained by label matching, that is, arg min D(P j , G i ).

[0111]

[0112] The above technical solution performs two-way matching in a single iteration to obtain the reference target box and the initial candidate box of the current round, and realizes the redundancy of the candidate prediction set for each sample target object, that is, the relationship between the redundant candidate of a single object and the label assignment of a single candidate (initial candidate box) is constructed. In the training process, a target candidate box o e,i is selected from S e,i as the learning target of the prediction module, instead of generating a top-1 candidate box for each real box only by two-way matching in the current round, which reduces the change frequency of the optimization target of the object query, stabilizes the optimization target of the detection module and the prediction module, effectively avoids the problem of unstable two-way matching results, ensures the stable and positive tuning of the optimization target region of each object query in different iterations, and improves the training effect and convergence speed.

[0113] S309: Based on the sample detection result, the target assignment result, and the annotation information, the parameters of the detection module and the prediction module are adjusted to obtain a target detection model.

[0114] The existing method adopts NMS post-processing to remove unnecessary prediction, but NMS is semantically irrelevant, usually only processes each class separately, and only considers the confidence score and IoU relationship of the prediction, which is poor in information dimension and accuracy, or adopts strict one-to-one matching such as Hungarian matching, and the training is unstable. The above technical solution additionally sets a prediction module, uses attention features and class logic as semantic input, combines the box position to obtain sample detection results, effectively utilizes image semantic information, and in the training process, uses the sample detection results output by the prediction module, the target assignment results output by the label assigner and the annotation information to determine the loss at the same time, to adjust the parameters of the initial detection model. The setting of the label assigner can introduce non-strict one-to-one label assignment, and each sample target object can be predicted through multiple object queries, thereby bringing redundant results, improving the stability, convergence speed of parameter adjustment, and learning ability for overlapping regions and target dense regions, and improving the training effect and object detection effect.

[0115] In some embodiments, S309 includes S3091-S3093:

[0116] S3091: determining a first loss based on the difference between the sample prediction box corresponding to each sample target object in the sample detection result and the target candidate box corresponding to each sample target object in the target assignment result;

[0117] S3092: determining a second loss based on the difference between the box position information of the target candidate box and the position ground truth, and the difference between the predicted class index of the target candidate box and the class label;

[0118] S3093: adjusting the parameters of the initial detection model according to the first loss and the second loss in the iterative training process to obtain a target detection model.

[0119] Specifically, the sample detection result includes a first correspondence relationship between the sample target object and the sample prediction box, the target assignment result includes a second correspondence relationship between the sample target object and the target candidate box, and the annotation information includes the real box corresponding to the sample target object. It can be understood that the first loss is determined by combining the difference between the first correspondence relationship and the second correspondence relationship, i.e. the distance between the sample prediction box and the target candidate box of each sample target object. The distance between the sample prediction box and the target candidate box used here can be a position distance, a predicted class distance, or a fusion of the two, such as the aforementioned positioning loss L box , class loss L clsor a total loss D of fusion of both, or a parameter such as an intersection over union between the two can be used to represent the distance. Alternatively, the target matching result can be regarded as a first vector formed by the target candidate box corresponding to a real box vector formed by the real box of the plurality of sample target objects, the detection result can be regarded as a first vector formed by the target candidate box corresponding to a vector formed by the real box of the plurality of sample target objects, and the sample detection result can be regarded as a second vector formed by the sample prediction box corresponding to the real box vector, and the first loss is determined in combination with the feature distance between the first vector and the second vector.

[0120] In one embodiment, a weighted mean square error loss is used for the calculation of the first loss to avoid the influence of label imbalance. Exemplarily, the first loss is calculated by the following formula, wherein Z e ∈R q represents the learning target of y output by the prediction module, and is an approximation target of the output of the prediction module, q is the number of object queries, and Z e,i =[i∈{o e,j |j=1,2,…,g}] is the target candidate box o e,i characterizing the target candidate box o e,j of the sample target object i in the target matching result, wherein g is the number of real boxes (i.e., sample target objects); I + ={o e,j |j=1,2,…,g}, and |I + | is the number of elements of I + , I - =U-I + , and U={1,2,…,g} is a universal set of query indexes, |I - | is the number of elements of this I - , Z e,j -Y j is the distance between the target candidate box and the sample prediction box of the real box j.

[0121]

[0122] Specifically, in combination with the distance between the target candidate box corresponding to each sample target object and the corresponding real box, the second loss is determined, and the distance between the target candidate box and the corresponding real box used here can be a position distance between the box position information and the position true value, a prediction category distance between the prediction category index and the category true value, or a fusion of both, such as the aforementioned positioning loss L box , the category loss L cls , or a total loss D of fusion of both, or a parameter such as an intersection over union between the two can be used to represent the distance.

[0123] Thus, the target candidate frame output by the label assigner and the position true value determine the loss, adjust the parameters of the initial prediction model, and make the model predict the position and category of the candidate frame based on the accurate matching result of the label assignment, improve the accuracy of the detection module output to express the object in the image, and the target candidate frame is determined based on the current iteration of the label assignment and the reference candidate frame of the redundant backup in the candidate prediction set, which improves the stability and positive influence of the optimization target. Moreover, the target candidate frame is used as the learning target of the sample prediction frame, so that the output of the prediction module tends to approach the output of the label assigner, which can not only learn the matching ability of the label assigner and keep the consistency with the optimization of the detection module, but also can improve the prediction accuracy without using binary matching for label matching in the actual inference process.

[0124] In some embodiments, the first loss is used to adjust the parameters of the prediction module, and the second loss is used to adjust the parameters of the detection module. The parameters of the detection module are adjusted based on the second loss, which can optimize the attention features, class logic information and frame position information of the detection module output based on the difference between the output of the label assigner and the true value, improve the feature processing accuracy of the detection module, and adjust the parameters of the prediction module based on the first loss, which can optimize the prediction module based on the difference between the output of the label assigner and the output of the prediction module, so that the output of the prediction module tends to approach the output of the label assigner, and the accurate frame matching ability is learned.

[0125] Specifically, the detection module takes a plurality of sample images as input, and takes the class logic information and frame position information of a plurality of candidate frames output by the detection module as input of the label assigner for label assignment, takes the annotation information as the expected output of the label assigner, and adjusts the parameters of the detection module based on the difference between the target assignment result output by the label assigner and the annotation information. Wherein, the label assignment is to construct a candidate prediction set of the target object of the sample based on the non-replacement sampling binary matching, and to perform the target candidate frame screening based on the redundant prediction of the initial candidate frame obtained by the replacement sampling binary matching. The specific screening process is based on the inclusion relationship among the initial candidate frame, the historical target frame and the candidate prediction set.

[0126] Specifically, the prediction module takes a plurality of sample images as input of the detection module, and takes the attention features, class logic information and frame position information of a plurality of candidate frames output by the detection module as input of the prediction module for relationship deduplication prediction, takes the target matching result output by the label assigner as the expected output, and adjusts the parameters of the prediction module based on the difference between the sample detection result output by the prediction module and the target matching result.

[0127] In summary, the preferred embodiment proposes an object detection method that alleviates the limitations of one-to-one matching prediction based on attention mechanism (such as DETR (DEtection Transformer)) and existing redundant prediction, NMS scheme (such as RCNN (Region-CNN)). In the model training process, the adaptive capacity label assignment strategy and the relationship duplicate data deduplication prediction module work together to produce superior performance and end-to-end inference. During the training process, the network sets different capacities for each real box, providing more abundant positive sample assignment labels. In testing, with the help of the prediction module, the capacity of each real box is changed to 1. The present application has carried out experimental tests on the benchmark test of MS COCO dataset and LVIS dataset, which shows that the method of the present application is significantly better than the traditional DETR and RCNN class model, and can be applied as a plug-and-play component to many benchmark methods with significant performance improvement. The object detection method of the present application supports general computer vision capability incubation. It can serve AI solution products such as video analysis platforms, etc., and can realize the rapid incubation of industry tracks, such as algorithm capability output in a monthly level time, industry solution output in a quarterly level time, etc.

[0128] Referring to Table 1 below, in a group of tests on the same dataset, the object detection method of the present application and NMS are integrated into Deformable-DETR and DETR, and the data in the table show that the AP (Average Precision, the average value of the precision of each target at different recalls) obtained by using the prediction module of the present application is significantly higher than that of NMS. The prediction module can be used as a relational deduplication module.

[0129]

[0130] In addition, referring to Figure 8 the effect diagram shown in the figure, the object detection results of the same detection module structure combined with NMS (left) and the prediction module of the present application (right) are shown, and it can be seen that the detection accuracy of the present application for overlapping regions and target dense regions is higher, and it can distinguish the detection of occluded human bodies (box B1 in the right figure).

[0131] Semantic-aware redundant prediction filter. We explain in Figure 6 the predictions handled by NMS and our relational deduplication module. Although NMS relies on some previous assumptions that close predictions can estimate the same object, sometimes there are different objects with similar positions in an image. Compared with NMS, our method does not simply consider the IoU metric, but makes predictions for two people.

[0132] The embodiment of the present application also provides an object detection device 500, as shown in Figure 9 The embodiment of the present application shows a structural schematic diagram of an object detection device, and the device can include the following modules: Figure 9 The embodiment of the present application shows a structural schematic diagram of an object detection device, and the device can include the following modules:

[0133] The acquisition module 10 is configured to acquire a target image and a target detection model, and the target detection model includes a detection module and a prediction module.

[0134] The detection module 20 is configured to input the target image into the detection module for feature extraction, perform feature processing on the obtained image features based on an attention mechanism to obtain attention features, and perform classification prediction and boundary box regression based on the attention features to obtain class logic information and frame position information of each of the plurality of candidate boxes.

[0135] The prediction module 30 is configured to input the attention features, the class logic information and the frame position information into the prediction module, perform deduplication prediction on the plurality of candidate boxes for target objects in the target image, and obtain an object detection result, which is used to indicate the position of a prediction frame corresponding to a target object predicted by the prediction module in the plurality of candidate boxes and the object category of the prediction frame.

[0136] In some embodiments, the acquisition module 10 includes:

[0137] The sample acquisition sub-module is configured to acquire a plurality of sample images and label information of the sample images, the sample images including at least one sample target object, and the label information including a category label and a position true value of the sample target object.

[0138] The object detection sub-module is configured to, in an iterative training process, perform object detection on the plurality of sample images as input of an initial detection model to obtain class logic information and frame position information of each of a plurality of candidate boxes corresponding to the sample images and a sample detection result of the sample images, and the sample detection result is used to indicate sample prediction frames corresponding to sample target objects predicted by the initial detection model in the plurality of candidate boxes.

[0139] The label assignment sub-module is configured to, if the current iteration is not the first iteration, input the class logic information and the frame position information of the plurality of candidate boxes and the label information into a label assigner as input, combine a historical target frame assigned to the sample target object by the label assigner in a historical iteration, perform label assignment between the plurality of candidate boxes and each sample target object, and obtain a target assignment result, and the target assignment result is used to indicate a target candidate frame matched by the label assigner to the sample target object in the plurality of candidate boxes in the current iteration and a predicted category index of the target candidate frame, and the historical target frame is a target candidate frame corresponding to the sample target object in the target assignment result of the historical iteration.

[0140] The parameter adjustment sub-module is configured to adjust parameters of the detection module and the prediction module based on the sample detection result, the target assignment result, and the annotation information, to obtain a target detection model.

[0141] In some embodiments, the label assignment sub-module is further configured to: if the current iteration is the first round of iteration, input the class logic information and the box position information of the plurality of candidate boxes, and the annotation information into a label assigner, to perform label assignment between the plurality of candidate boxes and the target objects of each sample, to obtain the target assignment result.

[0142] In some embodiments, the label assignment sub-module can further include:

[0143] The assigner unit is configured to: if the current iteration is not the first round of iteration, input the class logic information and the box position information of the plurality of candidate boxes, and the annotation information into the label assigner, to perform label assignment between the plurality of candidate boxes and the target objects of each sample, to obtain an initial assignment result; the initial assignment result is used to indicate an initial candidate box of the target objects of the sample in the current iteration, which is matched by the label assigner.

[0144] The correction unit is configured to: based on a containing relationship between a candidate prediction set corresponding to the target objects of the sample and the initial candidate box of the target objects of the sample, and a historical target box of the target objects of the sample in a previous round of iteration, correct the initial assignment result, to obtain the target assignment result; the candidate prediction set includes at least one reference target box obtained by the label assigner through non-replacement sampling label assignment between the plurality of candidate boxes and the target objects of the sample in the historical iteration and the current iteration.

[0145] In some embodiments, the assigner unit is specifically configured to:

[0146] based on the box position information of the candidate box and the position ground truth of the target objects of the sample, determine a first matching cost between the candidate box and the target objects of the sample, the first matching cost being used to indicate a position matching degree between the candidate box and the target objects of the sample;

[0147] based on the class logic information of the candidate box and the class label of the target objects of the sample, determine a second matching cost between the candidate box and the target objects of the sample, the second matching cost being used to indicate a class matching degree between the candidate box and the target objects of the sample;

[0148] fuse the first matching cost and the second matching cost, and based on the obtained matching cost, perform two-way matching between each candidate box and each target object of the sample, to obtain the initial candidate box corresponding to each target object of the sample, to generate the initial assignment result.

[0149] In some embodiments, the modifying unit is specifically configured to: for each sample target object, if both the initial candidate box and the historical target box of the previous iteration belong to the candidate prediction set of the current iteration, determining the historical target box of the previous iteration as the target candidate box of the sample target object in the current iteration.

[0150] In some embodiments, the modifying unit is further specifically configured to: for each sample target object, if at least one of the initial candidate box and the historical target box of the previous iteration does not belong to the candidate prediction set of the current iteration, determining the initial candidate box as the target candidate box of the sample target object in the current iteration.

[0151] In some embodiments, the label matching sub-module further comprises a candidate prediction set constructing unit configured to:

[0152] adding the target candidate box of the sample target object in the first iteration as a reference target box to the candidate prediction set of the sample target object;

[0153] performing mean calculation on the matching cost between each sample target object and the target candidate box in the first iteration to obtain a cost threshold value;

[0154] if the matching cost of the reference target box in the first iteration is lower than the cost threshold value, determining the candidate prediction set as a fixed set;

[0155] if the matching cost of the reference target box in the first iteration is higher than the cost threshold value, from the second iteration, performing non-replacement sampling label assignment on each sample target object based on the class logical information and the box position information of the plurality of candidate boxes and the annotation information to obtain an initial reference box corresponding to the sample target object;

[0156] adding the initial reference box to the candidate prediction set and determining a cost mean value corresponding to the matching cost of each element in the candidate prediction set;

[0157] if the cost mean value is lower than the cost threshold value, determining the candidate prediction set as the fixed set.

[0158] In some embodiments, the candidate prediction set constructing unit is further configured to: if the cost mean value is higher than the cost threshold value, continuously adding the initial reference box corresponding to the sample target object to the candidate prediction set in subsequent iterations until the cost mean value corresponding to the matching cost of each element in the candidate prediction set is lower than the cost threshold value, and then performing the step of determining the candidate prediction set as the fixed set.

[0159] In other embodiments, the candidate prediction set constructing unit is further configured to: if the cost mean value is higher than the cost threshold value, continuously adding the initial reference box corresponding to the sample target object to the candidate prediction set in subsequent iterations until the cost mean value corresponding to the matching cost of each element in the candidate prediction set is lower than the cost threshold value or the current iteration round reaches a preset round, and then performing the step of determining the candidate prediction set as the fixed set.

[0160] In some embodiments, the object detection submodule is specifically configured to:

[0161] input the sample image into the detection module for feature extraction, perform feature processing on the obtained sample image features based on an attention mechanism to obtain attention features of the sample image, and perform classification prediction and bounding box regression based on the attention features to obtain class logic information and box position information of the multiple candidate boxes;

[0162] input the attention features, the class logic information and the box position information of the multiple candidate boxes into the prediction module for object prediction to obtain a sample detection result.

[0163] In some embodiments, the parameter adjustment submodule is specifically configured to:

[0164] determine a first loss based on a difference between a sample prediction box corresponding to each sample target object in the sample detection result and a target candidate box corresponding to the sample target object in the target assignment result;

[0165] determine a second loss based on a difference between the box position information of the target candidate box and a position ground truth, and a difference between a predicted class index of the target candidate box and a class label;

[0166] In the iterative training process, the parameters of the initial detection model are adjusted according to the first loss and the second loss to obtain a target detection model.

[0167] In some embodiments, the first loss is used to adjust the parameters of the prediction module, and the second loss is used to adjust the parameters of the detection module.

[0168] It should be noted that the above device embodiments and method embodiments are based on the same implementation.

[0169] Embodiments of the present application provide a device, which can be a terminal or a server, comprising a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the object detection method provided by the above method embodiments.

[0170] Memory is used to store software programs and modules. The processor executes these stored software programs and modules to perform various functional applications and detect anomalies. Memory can primarily consist of a program storage area and a data storage area. The program storage area stores the operating system, application programs required for functionality, etc.; the data storage area stores data created based on device usage, etc. Furthermore, memory can include high-speed random access memory (RAM) and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0171] The methods and embodiments provided in this application can be executed in electronic devices such as mobile terminals, computer terminals, servers, or similar computing devices. Figure 10 This is a hardware structure block diagram of an electronic device for an object detection method provided in an embodiment of this application. For example... Figure 10 As shown, the electronic device 900 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 910 (CPUs 910 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 930 for storing data, and one or more storage media 920 (e.g., one or more mass storage devices) for storing application programs 923 or data 922. The memory 930 and storage media 920 may be temporary or persistent storage. The program stored in the storage media 920 may include one or more modules, each module including a series of instruction operations on the electronic device. Furthermore, the CPU 910 may be configured to communicate with the storage media 920 and execute a series of instruction operations in the storage media 920 on the electronic device 900. The electronic device 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input / output interfaces 940, and / or one or more operating systems 921, such as Windows Server. TM Mac OS X TM Unix TM Linux™, FreeBSD™, etc.

[0172] The input / output interface 940 can be configured to receive or transmit data via a network. The network can include a wireless network provided by a communication provider of the electronic device 900. In an example, the input / output interface 940 includes a network interface controller (NIC) that can be connected to other network devices through a base station to communicate with the Internet. In an example, the input / output interface 940 can be a radio frequency (RF) module that is configured to communicate with the Internet through a wireless manner.

[0173] Those skilled in the art can understand that, Figure 10 The structure shown is only schematic, and does not limit the structure of the electronic device. For example, the electronic device 900 can further include more or less components than those shown, or have a different configuration of components than those shown. Figure 10 Figure 10 The structure shown is only schematic, and does not limit the structure of the electronic device. For example, the electronic device 900 can further include more or less components than those shown, or have a different configuration of components than those shown.

[0174] The embodiments of the present application further provide a computer readable storage medium, which can be arranged in an electronic device to store at least one instruction or at least one program for implementing an anomaly detection method in the method embodiments. The at least one instruction or the at least one program is loaded and executed by the processor to implement the anomaly detection method provided by the above method embodiments.

[0175] Optionally, in the present embodiment, the storage medium can be located in at least one of a plurality of network servers in a computer network. Optionally, in the present embodiment, the storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0176] According to an aspect of the present application, a computer program product or a computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the method provided in the various optional implementation manners.

[0177] ​According to the object detection method, device, equipment, storage medium, server, terminal and program product provided by the application, the training of the object detection model is performed in combination with the initial video feature of the to-be-processed video, the graph data and the respective associated node features of the associated nodes thereof, the graph data is obtained by sampling at least one layer of associated nodes from the total graph data with the to-be-processed video as a starting node, the topological structure information between the video and other nodes in the graph data and the attribute information of the associated nodes are embedded in the target video feature, the information dimension and extension range of the object detection are improved, and the accuracy of the video feature expression is improved.

[0178] It should be noted that the above-mentioned embodiments of the application are only for description, and do not represent the advantages and disadvantages of the embodiments. The above-mentioned embodiments of the application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

[0179] Each of the embodiments in the application is described in a progressive manner, and the same or similar parts of each embodiment can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, the device, equipment and storage medium embodiments are described simply because they are basically similar to the method embodiments. The relevant parts can be referred to the part of the method embodiment.

[0180] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing relevant hardware to complete. The program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0181] The above is only a preferred embodiment of the application, and does not limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. An object detection method, characterized in that, The method includes: Acquire a target image and a target detection model, wherein the target detection model includes a detection module and a prediction module; The target image is input into the detection module for feature extraction. The obtained image features are processed based on an attention mechanism to obtain attention features. Based on the attention features, classification prediction and bounding box regression are performed to obtain the class logic information and box position information of each of the multiple candidate boxes. The attention features, class logic information, and box position information are input into the prediction module to perform deduplication prediction of the target object in the target image on the multiple candidate boxes, and the object detection result is obtained. The object detection result is used to indicate the position of the prediction box corresponding to the target object in the multiple candidate boxes and the object category of the prediction box.

2. The method according to claim 1, characterized in that, The methods for obtaining the target detection model include: Acquire multiple sample images and their annotation information, wherein each sample image includes at least one sample target object, and the annotation information includes the category label and location ground truth value of the sample target object; During iterative training, object detection is performed using the multiple sample images as input to the initial detection model, resulting in class logic information and box position information for each of the multiple candidate boxes corresponding to the sample images, as well as the sample detection results for the sample images. The sample detection results are used to indicate the sample prediction boxes corresponding to the target object predicted by the initial detection model in the multiple candidate boxes. If the current iteration is not the first iteration, the class logic information and box position information of the multiple candidate boxes, as well as the annotation information, are used as inputs to the label allocator. Combined with the historical target boxes assigned to the sample target objects by the label allocator in previous iterations, label allocation is performed between the multiple candidate boxes and each sample target object to obtain the target allocation result. The target allocation result is used to indicate the target candidate boxes matched by the label allocator for the sample target object in the multiple candidate boxes in the current iteration, as well as the predicted category index of the target candidate boxes. The historical target boxes are the target candidate boxes corresponding to the sample target objects in the target allocation results of previous iterations. Based on the sample detection results, the target allocation results, and the annotation information, the parameters of the detection module and the prediction module are adjusted to obtain the target detection model.

3. The method according to claim 2, characterized in that, The method further includes: If the current iteration is the first iteration, the class logic information and box position information of the multiple candidate boxes, as well as the annotation information, are input into the label allocator to perform label allocation between the multiple candidate boxes and each of the sample target objects, and the target allocation result is obtained.

4. The method according to claim 2, characterized in that, If the current iteration is not the first iteration, the class logic information and box position information of the multiple candidate boxes, as well as the annotation information, are used as inputs to the label allocator. Combined with the historical target boxes assigned to the sample target objects by the label allocator in previous iterations, label allocation is performed between the multiple candidate boxes and each sample target object to obtain the target allocation result, including: If the current iteration is not the first iteration, the class logic information and box position information of the multiple candidate boxes, as well as the annotation information, are input into the label allocator to perform label allocation between the multiple candidate boxes and each of the sample target objects, and an initial allocation result is obtained; the initial allocation result is used to indicate the initial candidate box that the label allocator matches in the multiple candidate boxes for the sample target object in the current iteration; Based on the inclusion relationships between the candidate prediction set corresponding to the sample target object and the initial candidate box of the sample target object and the historical target box of the sample target object in the previous iteration, the initial allocation result is corrected to obtain the target allocation result; the candidate prediction set includes at least one reference target box obtained by the label allocator from the multiple candidate boxes in the historical iteration and the current iteration through label allocation of the sample target object from the multiple candidate boxes without replacement.

5. The method according to claim 4, characterized in that, The step of inputting the class logic information and box position information of the multiple candidate boxes, as well as the annotation information, into the label allocator to perform label allocation between the multiple candidate boxes and each of the sample target objects, and obtaining the initial allocation result includes: Based on the box position information of the candidate box and the ground truth position of the sample target object, a first matching cost is determined between the candidate box and the sample target object. The first matching cost is used to indicate the degree of position matching between the candidate box and the sample target object. Based on the class logic information of the candidate box and the category label of the sample target object, a second matching cost is determined between the candidate box and the sample target object. The second matching cost is used to indicate the degree of category matching between the candidate box and the sample target object. The first matching cost and the second matching cost are combined, and a binary matching is performed between each candidate box and each sample target object based on the obtained matching cost to obtain an initial candidate box corresponding to each sample target object, thereby generating the initial allocation result.

6. The method according to claim 4, characterized in that, The initial allocation result is corrected based on the inclusion relationships between the candidate prediction set corresponding to the sample target object and the initial candidate bounding box of the sample target object and the historical bounding box of the sample target object in the previous iteration, to obtain the target allocation result, which includes: For each sample target object, if both the initial candidate box and the historical target box from the previous iteration belong to the candidate prediction set of the current iteration, the historical target box from the previous iteration is determined as the target candidate box of the sample target object in the current iteration.

7. The method according to claim 4, characterized in that, Based on the inclusion relationships between the candidate prediction set corresponding to the sample target object and the initial candidate bounding box of the sample target object and the historical target bounding box of the sample target object in the previous iteration, the initial allocation result is corrected to obtain the target allocation result including: For each of the sample target objects, if at least one of the initial candidate box and the historical target box of the previous iteration does not belong to the candidate prediction set of the current iteration, the initial candidate box is determined as the target candidate box of the sample target object in the current iteration.

8. The method according to claim 4, characterized in that, The construction methods for the candidate prediction set of the sample target object include: The candidate bounding boxes of the sample target objects in the first iteration are used as reference bounding boxes and added to the candidate prediction set of the sample target objects; The average matching cost between the target object and the target candidate box in each sample during the first iteration is calculated to obtain the cost threshold. If the matching cost of the reference target box in the first iteration is lower than the cost threshold, the candidate prediction set is determined to be a fixed set; If the matching cost of the reference target box in the first iteration is higher than the cost threshold, from the second iteration onwards, based on the class logic information and box position information of multiple candidate boxes, as well as the annotation information, label assignment is performed on each of the sample target objects in the multiple candidate boxes without replacement to obtain the initial reference box corresponding to the sample target object. The initial reference box is added to the candidate prediction set, and the average cost corresponding to the matching cost of each element in the candidate prediction set is determined. If the average cost is lower than the cost threshold, the candidate prediction set is determined as a fixed set.

9. The method according to claim 8, characterized in that, The method further includes: If the average cost is higher than the cost threshold, the initial reference box corresponding to the sample target object is added to the candidate prediction set in subsequent iterations until the average cost corresponding to the matching cost of each element in the candidate prediction set is lower than the cost threshold, and the step of determining the candidate prediction set as a fixed set is executed.

10. The method according to claim 8, characterized in that, The method further includes: If the average cost is higher than the cost threshold, the initial reference box corresponding to the sample target object will continue to be added to the candidate prediction set in subsequent iterations. When the average cost corresponding to the matching cost of each element in the candidate prediction set is lower than the cost threshold or the current iteration round reaches the preset round, the step of determining the candidate prediction set as a fixed set will be executed.

11. The method according to any one of claims 2-10, characterized in that, The process of using the multiple sample images as input to an initial detection model for object detection, obtaining the class logic information and box position information of multiple candidate boxes corresponding to the sample images, and the sample detection results of the sample images, includes: The sample image is input into the detection module for feature extraction. The obtained sample image features are processed based on an attention mechanism to obtain the attention features of the sample image. Based on the attention features, classification prediction and bounding box regression are performed to obtain the class logic information and box position information of the multiple candidate boxes. The attention features of the multiple candidate boxes, the class logic information, and the box position information are input into the prediction module to perform object prediction, thereby obtaining the sample detection result.

12. The method according to any one of claims 2-10, characterized in that, The step of adjusting the parameters of the detection module and the prediction module based on the sample detection results, the target allocation results, and the annotation information to obtain the target detection model includes: Based on the difference between the sample prediction bounding box corresponding to each sample target object in the sample detection results and the target candidate bounding box corresponding to each sample target object in the target allocation results, a first loss is determined. The second loss is determined based on the difference between the box position information of the target candidate box and the ground truth position, as well as the difference between the predicted category index of the target candidate box and the category label. During iterative training, the parameters of the initial detection model are adjusted based on the first loss and the second loss to obtain the target detection model.

13. The method according to claim 12, characterized in that, The first loss is used to adjust the parameters of the prediction module, and the second loss is used to adjust the parameters of the detection module.

14. An object detection device, characterized in that, The device includes: Acquisition module: used to acquire the target image and the target detection model, wherein the target detection model includes a detection module and a prediction module; Detection module: used to input the target image into the detection module for feature extraction, perform feature processing based on the attention mechanism on the obtained image features to obtain attention features, and perform classification prediction and bounding box regression based on the attention features to obtain the class logical information and box position information of each of the multiple candidate boxes; Prediction module: Used to input the attention features, the class logic information and the box position information into the prediction module, perform deduplication prediction on the multiple candidate boxes for the target object in the target image, and obtain object detection results. The object detection results are used to indicate the position of the prediction box corresponding to the target object predicted by the prediction module in the multiple candidate boxes and the object category of the prediction box.

15. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the object detection method as described in any one of claims 1-13.

Images