Multi-category target detection model training method, multi-category target detection model using method, medium and device

Abstract

The invention provides a multi-category target detection model training method, a multi-category target detection model using method, a medium and equipment, and is applied to the technical field of computer vision. According to the method, target object information is labeled through multi-scene image data, a multi-category target detection model is constructed, and the multi-category target detection model comprises a feature extraction layer, a detection branch and an attribute recognition branch which are used for recognizing non-vehicle and vehicle feature and state information respectively. And after iterative training, the model realizes accurate identification and output of multiple types of targets in an actual scene video. According to the method, through multi-category target detection model training based on multi-scene annotation data, and in combination with modular design and iterative optimization of a feature extraction layer and classification and attribute identification branches, higher precision and stability of multi-category targets in real-time detection are realized, so that reliable support is provided for an automatic parking technology.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a training method, a use method, a medium and a device for a multi-category target detection model. Background Art

[0002] With the development of autonomous driving technology, automatic parking technology, as an important component, has received extensive attention and research. During the automatic parking process, the vehicle needs to monitor the surrounding environment in real time during the parking search, parking in and parking out phases, and make corresponding decisions. This process not only requires the perception algorithm to provide a 360° all-round view, but also needs to support the recognition of multiple types of target objects, such as parking lot pillars, different types of cars, pedestrians, and wheel blocks. This information is the basis for subsequent path planning and driving control.

[0003] The safety of automatic parking places extremely high demands on the accuracy and efficiency of perception algorithms. However, in practical applications, due to the complexity of the environment, the diversity of target types, the quality of camera equipment, and the diversity of platform deployment, the optimization direction of existing surround perception algorithms is often unclear, making it difficult to effectively improve the overall performance of the system.

[0004] Therefore, how to improve the accuracy and stability of real-time detection of multiple categories of targets has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of the above problems, the present invention provides a training method, a use method, a medium and a device for a multi-category target detection model that overcomes the above problems or at least partially solves the above problems. The technical solution is as follows:

[0006] A training method for a multi-category target detection model, comprising:

[0007] Obtaining an image data training set, wherein the image data training set includes image data in multiple scenes, and the image data is annotated with at least one target object, non-vehicle grounding point label information when the target object is a non-vehicle object, and vehicle state label information and vehicle grounding point label information when the target object is a vehicle object;

[0008] Build a multi-category target detection model, wherein the multi-category target detection model is based on a target detection network, the target detection network includes a feature extraction layer, a detection branch, a non-vehicle attribute recognition branch and a vehicle attribute recognition branch, the feature extraction layer is used to extract basic image features, the detection branch is used to output object feature information of the target object based on the basic image features, the non-vehicle attribute recognition branch is used to output the first grounding point position of the non-vehicle object according to the identified object feature information, and the vehicle attribute recognition branch is used to output the vehicle state information and the second grounding point position of the vehicle object according to the identified object feature information;

[0009] Each of the image data in the image data training set is input into the multi-category target detection model for model training, and the loss is calculated based on the difference between the output results of the detection branch, the non-vehicle attribute recognition branch and the vehicle attribute recognition branch and the label information of the corresponding target object. The multi-category target detection model is iterated through the back propagation algorithm to obtain the trained multi-category target detection model.

[0010] Optionally, the target detection network is based on the YoloV7 algorithm, generates a detection priori box based on a clustering algorithm and target size, and adopts a PAFPN multi-scale structure that integrates high- and low-level features to solve the problem of small target detection.

[0011] Optionally, the loss function of the classifier and confidence of the detection head corresponding to the detection branch adopts cross entropy, and a specific IoU threshold is set for each type of target in the confidence loss. When the sample confidence IoU is greater than the corresponding IoU threshold, it is set to 1, otherwise it remains unchanged; in the bounding box loss calculation, Inner-CIoU is used as the loss function, a scale factor is introduced and an auxiliary bounding box is generated.

[0012] Optionally, the key point head classifiers of the non-vehicle attribute recognition branch and the vehicle attribute recognition branch use cross entropy as the loss function, the ground point regression uses Smooth L1 loss function, and / or the vehicle state head classifier of the vehicle attribute recognition branch uses cross entropy as the loss function.

[0013] Optionally, in the process of inputting each of the image data in the image data training set into the multi-category target detection model for model training, the method further includes:

[0014] The original label generated when the non-vehicle object is a pillar object is expanded leftward and rightward by 3 pixels and downward by 5 pixels to form a new label for the pillar object, and the grounding point of the pillar remains unchanged.

[0015] Optionally, before inputting each of the image data in the image data training set into the multi-category object detection model for model training, the method further includes:

[0016] The MOSiAC algorithm and the MIXUP algorithm are used to perform data enhancement processing on the image data training set to obtain the image data training set after data enhancement.

[0017] Optionally, before inputting each of the image data in the image data training set into the multi-category object detection model for model training, the method further includes:

[0018] The target objects in the image data whose areas or lengths do not meet corresponding target thresholds are filtered out.

[0019] A method for using a multi-category target detection model, comprising:

[0020] Obtain video frames collected by the vehicle in actual scenes;

[0021] The video frame is input into the multi-category target detection model trained according to any of the above-mentioned training methods to obtain a multi-category target recognition result output by the multi-category target detection model, wherein the multi-category target recognition result includes object feature information of each target object in the video frame, the first grounding point position of the non-vehicle object, and the vehicle status information and the second grounding point position of the vehicle object.

[0022] A computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the program implements the training method of the multi-category target detection model described in any one of the above items, and / or implements the method of using the multi-category target detection model described in any one of the above items.

[0023] An electronic device, comprising at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute any of the above-mentioned methods for training a multi-category target detection model and / or a method for using a multi-category target detection model.

[0024] By means of the above technical scheme, the present invention provides a training method, a use method, a medium and a device for a multi-category target detection model, obtains an image data training set, wherein the image data training set includes image data in multiple scenes, and the image data is annotated with at least one target object, non-vehicle grounding point label information when the target object is a non-vehicle object, and vehicle status label information and vehicle grounding point label information when the target object is a vehicle object; builds a multi-category target detection model, wherein the multi-category target detection model is based on a target detection network, and the target detection network includes a feature extraction layer, a detection branch, a non-vehicle attribute recognition branch and a vehicle attribute recognition branch, the feature extraction layer is used to extract basic image features, and the detection branch The branch is used to output the object feature information of the target object based on the basic features of the image, the non-vehicle attribute recognition branch is used to output the first grounding point position of the non-vehicle object according to the recognized object feature information, and the vehicle attribute recognition branch is used to output the vehicle state information and the second grounding point position of the vehicle object according to the recognized object feature information; each image data in the image data training set is input into the multi-category target detection model for model training, and the loss is calculated according to the difference between the output results of the detection branch, the non-vehicle attribute recognition branch and the vehicle attribute recognition branch and the label information of the corresponding target object, and the multi-category target detection model is iterated through the back propagation algorithm to obtain a trained multi-category target detection model. The video frame collected by the vehicle in the actual scene is obtained; the video frame is input into the trained multi-category target detection model to obtain the multi-category target recognition result output by the multi-category target detection model, wherein the multi-category target recognition result includes the object feature information of each target object in the video frame, the first grounding point position of the non-vehicle object, and the vehicle state information and the second grounding point position of the vehicle object. The present invention achieves higher accuracy and stability in real-time detection of multi-category targets by training a multi-category target detection model based on multi-scene annotated data, combined with modular design and iterative optimization of the feature extraction layer and classification and attribute recognition branches, thereby providing reliable support for automatic parking technology.

[0025] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0027] Figure 1 A flow chart showing an implementation of a method for training a multi-category target detection model provided by an embodiment of the present invention;

[0028] Figure 2 A schematic diagram illustrating the Inner-CIoU principle provided by an embodiment of the present invention is shown;

[0029] Figure 3 A schematic diagram illustrating label extension provided by an embodiment of the present invention is shown;

[0030] Figure 4 A schematic diagram of the network structure of a multi-category target detection model provided by an embodiment of the present invention is shown;

[0031] Figure 5 The overall process diagram of the training method for multi-category target detection provided by an embodiment of the present invention is shown;

[0032] Figure 6 A flowchart showing an implementation method of a method for using a multi-category target detection model provided by an embodiment of the present invention is shown;

[0033] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0034] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0035] With the rapid development of autonomous driving technology, automatic parking, as one of its key technologies, has become a research hotspot. During the automatic parking process, the vehicle needs to perceive the environmental information in real time during the parking search, parking in and parking out phases, and make decisions based on the perception results. This process requires the perception algorithm to have a 360° all-round view and be able to accurately identify a variety of target objects, including parking lot pillars, different types of vehicles, pedestrians, wheel blocks, etc. The accurate detection and identification of these targets is the core foundation of path planning and safe driving control.

[0036] However, automatic parking scenarios usually face complex real-world environments, such as light changes, occlusion, multi-target interference, etc. In addition, the diversity of target types (such as vehicles and non-vehicle targets) and the specific attribute information of different targets (such as vehicle status, location and grounding point, etc.) further increase the difficulty of perception algorithms. At the same time, limited by the resolution of camera equipment, sensor performance and hardware conditions of the computing platform, existing technologies still face many challenges in the real-time, accuracy and stability of target detection.

[0037] Although the current multi-target detection algorithm performs well in specific scenarios, it lacks robustness and accuracy in the face of multi-category targets, especially in dynamically changing complex environments. At the same time, the output consistency and accuracy of attribute information of different target categories (such as the location or status of the vehicle's grounding point, specific location information of non-vehicles, etc.) have not reached the ideal level. These problems have led to bottlenecks in the system's detection accuracy and stability, affecting the overall safety and reliability of the automatic parking system. Therefore, there is an urgent need for a technology that can take into account the accuracy, efficiency and robustness of multi-category target detection to optimize the perception and decision-making capabilities during the automatic parking process.

[0038] Based on this, in an embodiment of the present invention, a training method and a method for using a multi-category target detection model are provided. First, by obtaining an image data training set, the training set contains image data in multiple scenes, and annotates the target objects and their corresponding grounding points or status information, such as non-vehicle grounding point information of non-vehicle objects and vehicle status and grounding point information of vehicle objects. Secondly, a multi-category target detection model is built. The model consists of a feature extraction layer, a detection branch, and non-vehicle and vehicle attribute recognition branches. It can effectively extract basic image features and output detailed feature information of the target object. It can be seen that by training the model on the image data training set and inputting actual scene video frames, the model can output multi-category target recognition results, including object features, grounding point locations of non-vehicle objects, and status and grounding point locations of vehicle objects, thereby improving the accuracy and stability of multi-category target detection and providing reliable support for automatic parking technology.

[0039] like Figure 1 As shown, a flowchart of an implementation of a method for training a multi-category target detection model provided by an embodiment of the present invention is provided. The method may include:

[0040] S100. Obtain an image data training set, wherein the image data training set includes image data in multiple scenes, and the image data is annotated with at least one target object, non-vehicle grounding point label information when the target object is a non-vehicle object, and vehicle state label information and vehicle grounding point label information when the target object is a vehicle object.

[0041] The image data training set is a set of image data used to train a multi-category target detection model. Each image contains annotation information to help the model learn to identify specific target objects and their attributes. The image data in the image data training set is usually collected from real scenes, covering a variety of environments and conditions to improve the generalization ability of the model.

[0042] The target object refers to the specific object that is annotated in the image data and needs to be recognized by the model. The target object represents the thing that the model needs to detect and recognize in practical applications.

[0043] Optionally, an embodiment of the present invention may annotate target objects in image data in the form of closed rectangles, and annotate non-vehicle grounding point label information and vehicle grounding point label information in the form of solid dots, wherein the vehicle grounding point label information includes tire grounding point position information and grounding point position status information of vehicle objects, wherein the grounding point position status information includes a visible state and a predicted state, and the vehicle status label information includes a normal state, a truncated state, and an occluded state.

[0044] Optionally, the target objects annotated in the image data training set provided by the embodiment of the present invention include 28 types of targets including cars, buses, trucks, construction vehicles, pedestrians, bicycles, cyclists, tricycles, standing ground locks, cones, wheel chocks, water barriers, fire hydrants, shopping carts, lying ground locks, parking lot pillars, warning signs, trash cans, balance bikes, baby strollers, shopping carts, warning posts, spherical obstacles, scooters, flatbed trucks, parking poles, speed bumps, fire extinguisher boxes, etc., and each target object is annotated with its location, category and grounding point label information.

[0045] Among them, non-vehicle objects refer to all other objects in the image data that are not vehicles, such as pedestrians, pillars, and buildings. Labeling non-vehicle objects in the image data training set helps the model distinguish between vehicles and non-vehicle targets.

[0046] Vehicle objects refer to objects in the image data that belong to the vehicle category, including various types of vehicles, such as cars, trucks, and motorcycles. Labeling vehicle objects in the image data training set helps the model learn to specifically recognize vehicles and their attributes.

[0047] Among them, the grounding point label information refers to the specific location information of the target object in contact with the ground. For non-vehicle and vehicle objects, marking the grounding point information can help the model clearly identify the location of the object in space. For non-vehicle target objects (such as pedestrians), mark their specific location information (grounding point label) in contact with the ground. For vehicle target objects, the grounding point status of the vehicle is detailed and annotated, including: visible points (the grounding point of the vehicle tire is clearly visible) and predicted points (the grounding point is not visible due to the tire being blocked or other factors, and the model needs to predict the grounding point of the tire).

[0048] Among them, the vehicle status label information is used to describe the specific status of vehicle objects in the image data, which helps the model learn how to correctly identify and judge vehicles under different occlusion and viewing angle conditions during training. For vehicle targets such as cars, trucks, buses and engineering vehicles, their status information is further annotated, including: Normal (the vehicle is not occluded or only partially occluded), Truncated (the vehicle is at the edge of the image and partially truncated) and Crowded (the vehicle is occluded by other objects).

[0049] The image data training set provided by the embodiment of the present invention provides the model with rich target object information, while covering a variety of scenarios and complex conditions, which helps to improve the accuracy and robustness of the model in target recognition and attribute analysis.

[0050] S110. Build a multi-category target detection model, wherein the multi-category target detection model is based on a target detection network, and the target detection network includes a feature extraction layer, a detection branch, a non-vehicle attribute recognition branch, and a vehicle attribute recognition branch. The feature extraction layer is used to extract basic image features, the detection branch is used to output object feature information of the target object based on the basic image features, the non-vehicle attribute recognition branch is used to output the first grounding point position of the non-vehicle object according to the recognized object feature information, and the vehicle attribute recognition branch is used to output the vehicle status information and the second grounding point position of the vehicle object according to the recognized object feature information.

[0051] Among them, the multi-category target detection model is a computer vision model based on deep learning, which is used to detect and identify target objects of multiple categories in an image. The multi-category target detection model can not only accurately identify the category of the target, but also locate the specific position of the target in the image and extract relevant attribute information of the target (such as vehicle status and grounding point location). Optionally, the multi-category target detection model can be built based on the YOLOv7 framework.

[0052] Among them, the target detection network is the core structure for building a multi-category target detection model. It is a deep learning framework that is responsible for extracting features from input images, detecting target objects, and extracting relevant attribute information of the target. The target detection network consists of multiple functional modules, each of which completes different tasks.

[0053] Among them, the feature extraction layer is the basic part of the target detection network. Its main task is to extract the basic features of the image from the input image. These features include information such as edges, textures, shapes, etc., which can provide high-quality image feature representation for subsequent detection and attribute recognition tasks. Common feature extraction layers are based on convolutional neural networks (CNN), such as ResNet (Residual Network), VGG (Visual Geometry Group), etc.

[0054] Optionally, in order to adapt to the actual deployment environment, the model uses a lightweight Backbone network for efficient feature extraction. On this basis, combined with the PAFPN (Path Aggregation Feature Pyramid Network) module, multi-scale feature information is integrated to ensure good detection effects for both large and small targets. The feature extraction layer is responsible for extracting the basic features of the image to provide support for subsequent tasks.

[0055] The detection branch is the module in the object detection network responsible for detecting the target object. The detection branch generates object feature information of the target object, such as the category (such as vehicle or pedestrian) and location (such as the coordinates of the bounding box) of the target based on the basic image features output by the feature extraction layer.

[0056] Among them, the non-vehicle attribute recognition branch is specifically used to process detected non-vehicle target objects. Based on the object feature information provided by the detection branch, the non-vehicle attribute recognition branch further analyzes and outputs the attribute information of non-vehicle objects, especially the first contact point position of non-vehicle objects (that is, the specific position where the non-vehicle target contacts the ground). In order to process non-vehicle targets, the non-vehicle attribute recognition branch shares the feature extraction network with the detection branch, and introduces the KeypointHead module by intercepting the regional features corresponding to the positive samples, focusing on predicting the first contact point position of the target, providing a basis for subsequent ranging tasks.

[0057] Among them, the vehicle attribute recognition branch is used to process vehicle-type target objects. Based on the object feature information provided by the detection branch, the vehicle attribute recognition branch further analyzes and outputs two core attribute information of vehicle-type objects: vehicle status information: for example, whether the vehicle is obscured (Normal, Truncated, or Crowded). Second grounding point location: the specific location where the vehicle tire contacts the ground, which may include visible grounding points or grounding points that need to be predicted. For vehicle-type targets, the vehicle attribute recognition branch also shares the feature extraction network, and uses the CarStateHead module to determine the vehicle status (such as the direction of the vehicle head, dynamic information, etc.), and outputs the location of the vehicle's second grounding point. The design of the vehicle attribute recognition branch aims to solve the problem of vehicle status recognition under different perspectives and ensure the accuracy and rationality of perspective selection.

[0058] The embodiment of the present invention is based on the multi-category target detection model of YOLOv7, focusing on the optimization and improvement of surround perception tasks. The model aims to improve the detection accuracy and stability of targets of different distances and types within the 360° surround range of the vehicle by introducing multiple optimization strategies. Based on modular design and strategy optimization, the model can efficiently handle multi-category target detection tasks, while ensuring real-time and high precision, and realize comprehensive judgment of target status and position. It is not only easy to deploy, but also widely applicable to scenarios such as automatic parking, providing strong technical support for intelligent driving systems.

[0059] S120. Input each image data in the image data training set into the multi-category target detection model for model training. Calculate the loss based on the difference between the output results of the detection branch, the non-vehicle attribute recognition branch, and the vehicle attribute recognition branch and the label information of the corresponding target object. Iterate the multi-category target detection model through the back propagation algorithm to obtain a trained multi-category target detection model.

[0060] In the model training stage, the embodiment of the present invention can input each image data in the image data training set into the multi-category target detection model. The model extracts the basic features of the image through the feature extraction layer, and outputs the corresponding prediction results through the detection branch, non-vehicle attribute recognition branch and vehicle attribute recognition branch. The goal of the training is to optimize the parameters of the model based on the difference between these output results and the true label, so that the output results are as close to the true label as possible, so as to detect and classify the target object more accurately.

[0061] In the model training stage, the embodiment of the present invention quantifies the difference between the output result of the model and the real label through the loss function. The detection branch loss includes classification loss and bounding box regression loss, wherein the classification loss is used to measure the difference between the predicted target category and the real category, and methods such as cross entropy loss or focal loss can be used. The bounding box regression loss is used to compare the difference between the predicted bounding box and the real bounding box, and cross entropy loss, L1 loss, L2 loss or IoU (Intersection over Union) or Inner-CIoU (Inner Complete Intersection over Union) can be used. The non-vehicle attribute recognition branch loss is mainly the ground point position loss. The ground point position loss is used to measure the difference between the first ground point coordinates predicted by the model and the real ground point coordinates. The difference can be measured based on the Smooth L1 loss function. The vehicle attribute recognition branch loss includes vehicle state loss and second ground point position loss, wherein the vehicle state loss is used to measure the difference between the predicted vehicle state and the real state, and cross entropy loss can be used. The second grounding point position loss can also use the Smooth L1 loss function loss to compare the coordinate difference between the predicted grounding point and the actual grounding point. The embodiment of the present invention can use the above losses to perform weighted summation to obtain the total loss. It can be understood that the weight coefficient of each loss can be set according to actual needs.

[0062] After calculating the total loss, the embodiment of the present invention can use the chain rule of the backpropagation algorithm to calculate the partial derivatives (gradients) of the loss function relative to the model parameters. The gradient describes the sensitivity of the loss function to the change of each parameter. Based on the calculated gradient, the model parameters are updated using an optimization algorithm (such as SGD (StochasticGradient Descent), Adam (Adaptive Moment Estimation), etc.) to reduce the loss function.

[0063] In the embodiment of the present invention, after updating the model parameters, the above steps may be repeated until a preset training condition is met, which may be that a maximum number of iterations is reached or the loss value no longer decreases significantly.

[0064] The embodiment of the present invention provides a training method for a multi-category target detection model, which constructs a high-quality image data training set containing multi-scene and multi-category annotation information, and designs a multi-category target detection model, which integrates multi-branch structures such as feature extraction, target detection, non-vehicle attribute recognition and vehicle attribute recognition to achieve comprehensive detection and classification of target objects. By optimizing the model with the help of the back propagation algorithm, it can accurately identify the grounding point of non-vehicle objects, the status of vehicle objects and their grounding point information. This technology can effectively improve the accuracy and stability of multi-category targets in real-time detection, provide accurate environmental perception capabilities for automatic parking scenarios, and ensure that vehicles can complete parking tasks safely and efficiently in complex parking environments, thereby significantly improving the reliability and adaptability of automatic parking systems.

[0065] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the embodiment of the present invention, the target detection network is based on the YoloV7 algorithm, generates a detection prior frame based on the clustering algorithm and the target size, and adopts the PAFPN multi-scale structure that integrates high and low layer features to solve the problem of small target detection.

[0066] The embodiment of the present invention optimizes the network architecture of the target detection network based on the high efficiency and real-time performance of the YoloV7 algorithm, which can significantly improve the detection accuracy while ensuring the detection speed.

[0067] Among them, the detection prior box is a set of bounding boxes initially set by the target detection network, which is an important reference for matching the real target position when training the model. The embodiment of the present invention can use a clustering algorithm (such as K-Means or K-Means++) to generate a prior box based on the target object size distribution, and can generate a more realistic frame configuration based on the target object size distribution in the image data training set. In this way, the model can effectively adapt to targets of different scales, thereby improving the accuracy of detection, especially in small target detection.

[0068] Among them, PAFPN (Path Aggregation Feature Pyramid Network) is a multi-scale feature fusion network structure for target detection. By introducing the PAFPN multi-scale structure, the embodiment of the present invention can better retain the detailed features of small targets and enhance the global context information, thereby effectively solving the problem of easy feature loss in small target detection.

[0069] The embodiment of the present invention generates a priori frames through clustering to improve the initial frame matching capability for small targets, and combines the multi-scale feature fusion of the PAFPN network structure to significantly improve the detection accuracy and stability of the model for small targets.

[0070] Optionally, the loss function of the classifier and confidence of the detection head corresponding to the detection branch provided in an embodiment of the present invention adopts cross entropy, and sets a specific IoU threshold for each type of target in the confidence loss. When the sample confidence IoU is greater than the corresponding IoU threshold, it is set to 1, otherwise it remains unchanged; in the bounding box loss calculation, Inner-CIoU is used as the loss function, a scale factor is introduced and an auxiliary bounding box is generated.

[0071] The embodiment of the present invention can set specific IoU thresholds for different target categories, so that the model can better adapt to the characteristics of different targets, improve the generalization ability of multi-category targets, and through Inner-CIoU and auxiliary bounding boxes, the model can optimize small and complex targets more accurately, reduce attention to unimportant details, and thus improve the adaptability of the model to various scenarios. At the same time, due to the efficient alignment mechanism of Inner-CIoU, the matching error between the predicted box and the real box can be measured more accurately, so that the model can converge to a reasonable matching box more quickly, and through the guidance of the auxiliary bounding box, the model is provided with additional optimization targets to help the bounding box regression task approach the real box faster.

[0072] Figure 2 FIG. 2 is a schematic diagram illustrating the Inner-CIoU principle provided by an embodiment of the present invention. The calculation formula of the internal complete intersection-over-union loss is:

[0073]

[0074]

[0075]

[0076] in, is the internal complete intersection loss; For complete exchange and comparison loss; It is the intersection-over-union ratio, which is used to measure the overlap between the predicted box and the true box; It is the internal intersection-union ratio, which is used to measure the ratio of the red area (the inner part of the intersection between the predicted box and the true box) to the yellow area (the union of the entire predicted box and the true box); is the center point of the prediction box The center point of the real frame Euclid example square between ; The square of the diagonal length of the minimum enclosing box containing the predicted box and the true box; is a weight factor used to balance the shape consistency loss; is the difference in aspect ratio between the predicted box and the true box.

[0077] The embodiments of the present invention provide these optimization strategies in the detection branch, so that the multi-category target detection model can improve the generalization ability while accelerating the regression speed of the bounding box, thereby demonstrating more efficient and more accurate target detection capabilities in actual application scenarios.

[0078] Optional, in the above Figure 1 Based on the corresponding one or more embodiments, in another optional embodiment provided by the embodiment of the present invention, the key point head classifier of the non-vehicle attribute recognition branch and the vehicle attribute recognition branch adopts cross entropy as the loss function, the ground point regression adopts the Smooth L1 loss function, and / or the vehicle state head classifier of the vehicle attribute recognition branch adopts cross entropy as the loss function.

[0079] Among them, the calculation formula of Smooth L1 loss function is:

[0080]

[0081] in, Represents the value of SmoothL1 loss, which is used to measure the difference between the predicted value and the true value; is the error between the predicted value and the true value.

[0082] In the embodiment of the present invention, the detection task of the key point head classifier and the recognition task of the vehicle state head classifier are optimized simultaneously in the same training cycle. The design of the loss function allows the two tasks to be collaboratively optimized by sharing weights and gradients. By jointly optimizing the loss functions of each task, the model can improve the detection accuracy and attribute recognition performance at the same time in one training, avoiding the repeated steps of separate training and accelerating the gradient calculation and weight optimization process.

[0083] The embodiments of the present invention significantly improve the training speed of the model by reasonably selecting the loss function and jointly optimizing the detection and recognition tasks, while ensuring the efficiency and accuracy of the detection and recognition tasks.

[0084] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the embodiment of the present invention, in the process of inputting each image data in the image data training set into the multi-category target detection model for model training, the method may further include:

[0085] The original label generated for the non-vehicle object as a pillar object is expanded by 3 pixels to the left and right and 5 pixels downward to form a new label for the pillar object, and the ground point of the pillar remains unchanged.

[0086] In order to improve the problem of the detection frame being biased upward when detecting a column-type target at a close distance, the embodiment of the present invention is as follows. Figure 3As shown, when generating the label of the column object, the original label ( Figure 3 The left and right borders of the red box in the figure are extended by 3 pixels each, and the bottom border is extended downward by 5 pixels to obtain the new label of the column object ( Figure 3 yellow box in the middle), while keeping the column ground point ( Figure 3 The position of the red dot in the image remains unchanged. This design is mainly based on empirical analysis and test results of the characteristics of pillar-type targets. Considering that pillars are long vertical targets, their detection frames often shift upward at close distances. Therefore, the labels are appropriately adjusted downward during the training phase to improve the model's detection accuracy of the bottom of the pillars. In order to balance the integrity of the labels, the left and right boundaries are also expanded evenly.

[0087] It should be noted that the adjustment of the label (such as the specific number of pixels to expand downward) needs to be adaptively adjusted according to the characteristics of the column target to ensure that the detection effect of the model is maximized in different application scenarios.

[0088] The embodiment of the present invention expands the label of the column object by 3 pixels to the left and right and 5 pixels downward, while keeping the ground point unchanged, which helps to optimize the model's positioning accuracy for the column target and the rationality of the detection frame. The expanded label is more in line with the characteristics of the long column target, thereby guiding the model to focus on the bottom area of ​​the target and reducing the problem of the detection frame being biased upward in close-range scenes. At the same time, the left and right expansion enhances the model's ability to learn edge features and improves the integrity and discrimination of the target feature expression. This label optimization for specific target characteristics not only significantly improves the model's accuracy in detecting columns, but also helps the model to more stably handle targets of different shapes and characteristics in multi-category target detection tasks, reducing false detections and missed detections, thereby comprehensively improving the overall accuracy and stability of real-time detection.

[0089] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the embodiment of the present invention, before inputting each of the image data in the image data training set into the multi-category target detection model for model training, the method may further include:

[0090] The MOSiAC algorithm and the MIXUP algorithm are used to perform data enhancement processing on the image data training set to obtain the image data training set after data enhancement.

[0091] Among them, the MOSiAC algorithm (Mosaic Augmentation) is used to randomly crop and scale four images and stitch them into a new image, combined with the adjustment of the target bounding box, to generate diverse image training samples. The MOSiAC algorithm generates image data of highly complex scenes in the image data training set for model training to improve the model's adaptability to complex environments with multiple targets.

[0092] Among them, the MIXUP algorithm (MixUp Data Augmentation) is used to generate image training samples by linearly weighted mixing of two images and their corresponding labels. The MIXUP algorithm generates transition samples by linearly mixing images and labels, further enriching the distribution of image data used for model training in the image data training set.

[0093] The embodiments of the present invention improve the robustness and generalization of the model through image space splicing and feature linear mixing. Using these two algorithms in the training set can effectively generate more diverse samples, help the model better adapt to complex environments, and thus improve the accuracy and stability of multi-category target detection.

[0094] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the embodiment of the present invention, before inputting each image data in the image data training set into the multi-category target detection model for model training, the method may further include:

[0095] Filter out target objects in image data whose area or length does not meet the corresponding target threshold.

[0096] When filtering target objects in image data, the embodiment of the present invention sets different area or length thresholds for each type of target, and processes target objects that do not meet the threshold requirements (such as too small area or too short length). The specific operation includes filling these target areas (such as filling with black or white) to cover these target areas that do not meet the requirements, thereby reducing their interference with the image feature extraction process. At the same time, by deleting the corresponding target labels, the filtering of extremely small targets can be effectively achieved.

[0097] The embodiments of the present invention can help the model focus on more meaningful target objects by filtering extremely small target objects, thereby improving the training efficiency and detection performance of the model.

[0098] Figure 4The diagram shows a schematic diagram of the network structure of a multi-category target detection model provided by an embodiment of the present invention. The input of the multi-category target detection model is a single-frame image. Before the input image is subjected to the feature extraction layer, a preprocessing operation is performed, including adjusting the size of the column, filtering small targets, and data enhancement. The general feature extraction layer uses Inner-CIoU to further improve the accuracy of bounding box regression, so that the model can more accurately predict the location of the target, extract the features of the input image, and generate a visual feature representation of multi-category targets. Then, the PAFPN feature pyramid network is used to fuse feature layer information of different scales to detect targets of different sizes, thereby improving the model's ability to detect multi-category targets. After feature extraction and fusion, the network completes the target category recognition, ground point coordinate prediction, and vehicle state classification tasks through a series of fully connected layers. The Softmax classifier is used to predict the category of the features of each region and output the category of the detected target. The SmoothL1 loss function is used to regress and predict the docking point coordinates. The vehicle state is classified by the Softmax classifier and the vehicle's motion state is output. Through the above process, the model finally generates three outputs: target category recognition, ground point coordinates, and vehicle state. The network structure of the multi-category target detection model provided by the embodiment of the present invention can achieve higher accuracy and stability when detecting multi-category targets based on powerful feature extraction capabilities (such as PAFPN) and optimized loss functions (such as Inner-CIoU, SmoothL1, etc.). Through comprehensive recognition of target categories, grounding point locations, and vehicle status, the model provides comprehensive and reliable technical support for application scenarios such as intelligent driving, automatic parking, and traffic monitoring.

[0099] Figure 5The figure shows a schematic diagram of the overall process of the training method for multi-category target detection provided by an embodiment of the present invention. The embodiment of the present invention first obtains raw data through data acquisition. The data includes images and their corresponding various annotation information, such as target box annotations, ground point annotations, and vehicle status annotations. In the data enhancement stage, various data enhancement operations are performed on the collected images and annotation information to improve the robustness and generalization ability of the model. The enhancement operations include: resize: resize the image to adapt to the image size of the model input. MOSAiC: use splicing enhancement technology to splice multiple images into one to increase the diversity of training samples. Mixup: linearly mix two images and their labels to generate new training samples and smooth the decision boundary of the model. Random flip: randomly flip the image (horizontally or vertically) to expand the data sample. Random Affine transformation: geometrically transform the image through affine transformation (such as rotation, translation, scaling, etc.). HSVrandomAug: randomly adjust the hue, saturation, and brightness of the image to enhance the model's adaptability to lighting changes. The data enhancement operation makes the training data more diverse and improves the adaptability of the model to different scenes and complex environments. The images and annotation information processed by data enhancement are input into the target detection and classification module to complete the target detection task, the touchdown point detection task and the vehicle status classification task. After target detection and classification, the system outputs structured information including target information, vehicle status, touchdown point and vehicle touchdown point category. The embodiment of the present invention effectively improves the accuracy and stability of multi-category targets in real-time detection through a variety of data enhancement methods and target detection technologies, and provides accurate environmental perception capabilities for automatic parking scenarios.

[0100] like Figure 6 As shown, a flowchart of an implementation method of a method for using a multi-category target detection model provided by an embodiment of the present invention is provided. The method may include:

[0101] S600: Obtain video frames captured by the vehicle in an actual scene.

[0102] The actual scene may refer to a naturally occurring environment or application scene, such as urban roads, highways, industrial sites, ground parking lots, underground parking lots, and parking buildings.

[0103] Specifically, the embodiment of the present invention can collect a video of the vehicle's surrounding environment in an actual scene through the vehicle's onboard camera, and then extract a real-time video frame.

[0104] S610. Input the video frame into a trained multi-category target detection model obtained by the training method provided by an embodiment of the present invention, and obtain a multi-category target recognition result output by the multi-category target detection model, wherein the multi-category target recognition result includes object feature information of each target object in the video frame, the first grounding point position of the non-vehicle object, and the vehicle status information and the second grounding point position of the vehicle object.

[0105] The embodiment of the present invention can obtain accurate multi-category target recognition results by collecting video frames in actual scenarios and inputting them into an optimized and trained multi-category target detection model. The output results include detailed information such as the category, location, and grounding point of the target object, as well as vehicle status information. This achieves high accuracy and stability in target detection, effectively identifies vehicles, obstacles, and surrounding environments in parking lots, provides accurate environmental perception and reliable decision support for automatic parking technology, and helps improve the safety and efficiency of automatic parking.

[0106] Although operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order.Multitasking and parallel processing may be advantageous under certain circumstances.

[0107] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0108] An embodiment of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the training method of the multi-category target detection model and / or the method of using the multi-category target detection model.

[0109] An embodiment of the present invention provides a processor, which is used to run a program, wherein when the program is running, the training method of the multi-category target detection model and / or the method of using the multi-category target detection model are executed.

[0110] like Figure 7As shown, an embodiment of the present invention provides an electronic device 1000, which includes at least one processor 1001, and at least one memory 1002 and a bus 1003 connected to the processor 1001; wherein the processor 1001 and the memory 1002 communicate with each other through the bus 1003; the processor 1001 is used to call the program instructions in the memory 1002 to execute the training method of the multi-category target detection model and / or the method of using the multi-category target detection model. The electronic device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0111] The present invention also provides a computer program product, which, when executed on an electronic device, is suitable for executing a program that initializes the training method of the multi-category target detection model and / or the steps of the method for using the multi-category target detection model.

[0112] The present invention is described with reference to flowcharts and / or block diagrams of methods, electronic devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] In a typical configuration, an electronic device includes one or more processors (CPU), a memory, and a bus. The electronic device may also include an input / output interface, a network interface, and the like.

[0114] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.

[0115] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0116] In the description of the present invention, it should be understood that the terms "up", "down", "front", "back", "left" and "right" etc. indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the positions or elements referred to must have specific directions, be constructed and operate in specific directions. Therefore, they should not be understood as limitations of the present invention.

[0117] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0118] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A training method for a multi-category target detection model, characterized in that: include: Obtaining an image data training set, wherein the image data training set includes image data in multiple scenes, and the image data is annotated with at least one target object, non-vehicle grounding point label information when the target object is a non-vehicle object, and vehicle state label information and vehicle grounding point label information when the target object is a vehicle object; Build a multi-category target detection model, wherein the multi-category target detection model is based on a target detection network, the target detection network includes a feature extraction layer, a detection branch, a non-vehicle attribute recognition branch and a vehicle attribute recognition branch, the feature extraction layer is used to extract basic image features, the detection branch is used to output object feature information of the target object based on the basic image features, the non-vehicle attribute recognition branch is used to output the first grounding point position of the non-vehicle object according to the identified object feature information, and the vehicle attribute recognition branch is used to output the vehicle state information and the second grounding point position of the vehicle object according to the identified object feature information; Each of the image data in the image data training set is input into the multi-category target detection model for model training, and the loss is calculated based on the difference between the output results of the detection branch, the non-vehicle attribute recognition branch and the vehicle attribute recognition branch and the label information of the corresponding target object. The multi-category target detection model is iterated through the back propagation algorithm to obtain the trained multi-category target detection model.

2. The training method according to claim 1, characterized in that: The target detection network is based on the YoloV7 algorithm, generates a detection priori box based on the clustering algorithm and the target size, and adopts the PAFPN multi-scale structure that integrates high- and low-level features to solve the problem of small target detection.

3. The training method according to claim 1, characterized in that: The loss function of the classifier and confidence of the detection head corresponding to the detection branch adopts cross entropy, and a specific IoU threshold is set for each type of target in the confidence loss. When the sample confidence IoU is greater than the corresponding IoU threshold, it is set to 1, otherwise it remains unchanged; in the bounding box loss calculation, Inner-CIoU is used as the loss function, the scale factor is introduced and the auxiliary bounding box is generated.

4. The training method according to claim 1, characterized in that: The key point head classifiers of the non-vehicle attribute recognition branch and the vehicle attribute recognition branch use cross entropy as the loss function, the ground point regression uses the Smooth L1 loss function, and / or the vehicle state head classifier of the vehicle attribute recognition branch uses cross entropy as the loss function.

5. The training method according to claim 1, characterized in that: In the process of inputting each of the image data in the image data training set into the multi-category target detection model for model training, the method further includes: The original label generated when the non-vehicle object is a pillar object is expanded left and right by 3 pixels and downward by 5 pixels to form a new label for the pillar object, and the grounding point of the pillar remains unchanged.

6. The training method according to claim 1, characterized in that: Before inputting each of the image data in the image data training set into the multi-category target detection model for model training, the method further includes: The MOSiAC algorithm and the MIXUP algorithm are used to perform data enhancement processing on the image data training set to obtain the image data training set after data enhancement.

7. The training method according to any one of claims 1 to 6, characterized in that: Before inputting each of the image data in the image data training set into the multi-category target detection model for model training, the method further includes: The target objects in the image data whose areas or lengths do not meet corresponding target thresholds are filtered out.

8. A method for using a multi-category target detection model, characterized in that: include: Obtain video frames collected by the vehicle in actual scenes; The video frame is input into the multi-category target detection model trained according to any one of the training methods in claims 1 to 7 above, and a multi-category target recognition result output by the multi-category target detection model is obtained, wherein the multi-category target recognition result includes object feature information of each target object in the video frame, the first grounding point position of the non-vehicle object, and the vehicle status information and the second grounding point position of the vehicle object.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, it implements the training method of the multi-category target detection model as described in any one of claims 1 to 7, and / or implements the method for using the multi-category target detection model as described in claim 8.

10. An electronic device, characterized in that: The electronic device includes at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call the program instructions in the memory to execute the training method of the multi-category target detection model as described in any one of claims 1 to 7, and / or the method for using the multi-category target detection model as described in claim 8.

Images