Target detection model improvement method and device, equipment and storage medium
By constructing first and second loss functions to eliminate mismatched ground truth data, the problem of low training efficiency caused by the difference in field of view between the camera and LiDAR is solved, and efficient training and accuracy improvement of the target detection model are achieved.
Patent Information
- Application Number
- CN202511098963.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-06
- Publication Date
- 2025-11-21
AI Technical Summary
Existing target detection models suffer from low training efficiency and excessively long manual data processing time in vehicle driving scenarios due to the difference in field of view between cameras and LiDAR.
A first loss function is constructed to remove data from the ground truth that is outside the camera's field of view. A second loss function is constructed to remove data whose intersection-over-union ratio (IoU) does not exceed a preset IoU. The loss function of the initial object detection model is then reconstructed.
By automatically removing mismatched ground truth data, model training efficiency is improved, manual intervention is reduced, and the training speed and accuracy of the model are increased.
Smart Images
Figure CN120997798A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of target detection, and particularly relates to a target detection model improvement method and device, equipment and a storage medium. BACKGROUND
[0002] In a vehicle driving scene, a camera is usually combined with a laser radar to provide training data for a target detection model to realize 3D target detection. However, the camera carried on the vehicle is usually a monocular camera, and there is a large difference between the field of view (for example, 30 degrees) of the camera and the field of view (for example, 360 degrees) of the laser radar. At this time, if the original data collected by the laser radar is used for model training, the model may not converge.
[0003] The solution to the above problem provided by the prior art is to manually screen the original data collected by the laser radar, so that appropriate data is reserved before model training. However, this method needs to additionally increase a lot of data processing time, thereby causing low model training efficiency. SUMMARY
[0004] The main purpose of the present application is to provide a target detection model improvement method, device, equipment and storage medium, which aims to solve the technical problem of low training efficiency of the existing target detection model.
[0005] To achieve the above purpose, the present application provides a target detection model improvement method, which comprises the following steps:
[0006] Obtaining ground truth data of a sample image, the sample image being collected by a camera, and the ground truth data being generated by projecting three-dimensional data corresponding to the sample image;
[0007] Constructing a first loss function, the first loss function being used to eliminate data in the ground truth data which is located outside a visible range of the camera;
[0008] Constructing a second loss function, the second loss function being used to eliminate data in the ground truth data which does not exceed a preset intersection over union with the sample image;
[0009] Reconstructing a loss function in an initial target detection model based on the first loss function and the second loss function to obtain an improved target detection model.
[0010] In an embodiment, the step of constructing the first loss function comprises:
[0011] Obtaining a first prediction data set corresponding to the ground truth data under the condition that the ground truth data is located within the visible range of the camera;
[0012] obtaining a second prediction data set corresponding to the ground truth data in a case that the camera is located out of the visual range;
[0013] determining a conditional dependency between the visual range and the prediction data based on the first prediction data set and the second prediction data set, and constructing a first loss function according to the conditional dependency.
[0014] In an embodiment, the step of constructing the second loss function comprises:
[0015] randomly extracting a plurality of test images from the image database, wherein the test images contain a plurality of test bounding boxes truncated by image boundaries;
[0016] determining a detection index corresponding to the test bounding box in target detection, and an intersection over union between the test bounding box and the image boundary;
[0017] constructing a second loss function based on the detection index and the intersection over union.
[0018] In an embodiment, the step of constructing the second loss function based on the detection index and the intersection over union comprises:
[0019] selecting a preset index from the detection index, and obtaining a preset intersection over union corresponding to the preset index, wherein the preset index represents a critical index meeting detection requirements;
[0020] constructing a second loss function according to the preset intersection over union.
[0021] In an embodiment, the step of reconstructing the loss function in the initial target detection model based on the first loss function and the second loss function to obtain an improved target detection model comprises:
[0022] embedding the first loss function and the second loss function into the initial target detection model, and training the initial target detection model;
[0023] in a case that the total loss value of the trained model is less than a preset threshold value after the training is completed, determining the trained model as the improved target detection model.
[0024] In an embodiment, the initial target detection model comprises a detection head and an auxiliary training head, and the step of training the initial target detection model after embedding the first loss function and the second loss function into the initial target detection model comprises:
[0025] embedding the first loss function and the second loss function into the auxiliary training head, and predicting the attributes of the sample images in the three-dimensional space through the detection head to obtain prediction data.
[0026] training the initial target detection model based on the predicted data and the ground truth data.
[0027] In addition, the present application also provides a target detection method, which comprises the following steps:
[0028] inputting a target image collected by a vehicle in a driving scene into the improved target detection model to obtain a model output result;
[0029] generating a three-dimensional bounding box corresponding to the target image according to the model output result;
[0030] The improved target detection model is obtained by reconstructing a loss function in an initial target detection model based on a first loss function and a second loss function, the first loss function is used to eliminate data in the ground truth data that is located outside a visible range of the camera, the second loss function is used to eliminate data in the ground truth data that has an intersection over union with the sample image that does not exceed a preset intersection over union, the ground truth data is generated by projecting three-dimensional data corresponding to the sample image, and the sample image is collected by a camera.
[0031] In addition, to achieve the above object, the present application also provides a target detection model improvement device, which comprises:
[0032] a data acquisition module configured to acquire ground truth data of a sample image, the sample image is collected by a camera, and the ground truth data is generated by projecting three-dimensional data corresponding to the sample image;
[0033] a first function construction module configured to construct a first loss function, the first loss function is used to eliminate data in the ground truth data that is located outside a visible range of the camera;
[0034] a second function construction module configured to construct a second loss function, the second loss function is used to eliminate data in the ground truth data that has an intersection over union with the sample image that does not exceed a preset intersection over union;
[0035] a model improvement module configured to reconstruct a loss function in an initial target detection model based on the first loss function and the second loss function to obtain an improved target detection model.
[0036] In addition, to achieve the above object, the present application also provides a target detection model improvement device, which comprises: a memory, a processor, and a target detection model improvement program stored in the memory and executable on the processor, the target detection model improvement program is configured to implement the steps of the target detection model improvement method as described above.
[0037] In addition, to achieve the above object, the present application also provides a storage medium, which is a computer readable storage medium, and a target detection model improvement program is stored on the storage medium. The target detection model improvement program, when executed by a processor, implements the steps of the target detection model improvement method described above.
[0038] In addition, to achieve the above object, the present application also provides a computer program product, which comprises a target detection model improvement program. The target detection model improvement program, when executed by a processor, implements the steps of the target detection model improvement method described above.
[0039] The present application obtains ground truth data of a sample image, the sample image is collected by a camera, and the ground truth data is generated by projecting three-dimensional data corresponding to the sample image; a first loss function is constructed, the first loss function is used to eliminate data in the ground truth data that is located outside the visible range of the camera; a second loss function is constructed, the second loss function is used to eliminate data in the ground truth data that has an intersection over union with the sample image less than a preset intersection over union; and a loss function in an initial target detection model is reconstructed based on the first loss function and the second loss function to obtain an improved target detection model. The above method of the present application reconstructs the loss function in the initial target detection model by the first loss function and the second loss function constructed, so that the data in the ground truth data that is located outside the visible range of the camera and the data in the ground truth data that has an intersection over union with the sample image less than the preset intersection over union can be eliminated without relying on manual work, so that the ground truth data used for training the model can be automatically matched with the camera field of view, and the training efficiency of the model is improved. BRIEF DESCRIPTION OF DRAWINGS
[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.
[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.
[0042] Figure 1 Flowchart of the first embodiment of the target detection model improvement method of the present application;
[0043] Figure 2 Flowchart of the second embodiment of the target detection model improvement method of the present application;
[0044] Figure 3A flowchart of a third embodiment of the target detection model improvement method of the present application;
[0045] Figure 4 A structural block diagram of a first embodiment of the target detection model improvement device of the present application;
[0046] Figure 5 A structural diagram of the target detection model improvement device of the hardware running environment involved in the embodiment scheme of the present application.
[0047] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION
[0048] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0049] It should be noted that the execution subject of the embodiments of the present application can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, such as the above target detection model improvement device. The following embodiments will be described taking the target detection model improvement device as an example.
[0050] The embodiments of the present application provide a target detection model improvement method, which refers to Figure 1 , Figure 1 A flowchart of a first embodiment of the target detection model improvement method of the present application.
[0051] In the present embodiment, the target detection model improvement method comprises the following steps:
[0052] Step S1: Obtain the ground truth data of the sample image, the sample image is collected by a camera, and the ground truth data is generated by projecting the three-dimensional data corresponding to the sample image.
[0053] It should be noted that the above sample image can be an image suitable for training a target detection model. Specifically, the sample image needs to contain objects related to the detection scene, so that the model finally trained can be applicable to the detection scene. For example, for a vehicle driving scene, the sample image at least needs to contain motor vehicles, non-motor vehicles, pedestrians, traffic signs, obstacles, etc., which are not limited by the present embodiment. In addition, the present embodiment does not limit the number of sample images (such as 100, 1000, etc.).
[0054] It should be understood that the above camera can be a monocular camera, or other cameras that can be configured on a vehicle, such as a binocular camera, a trinocular camera, etc. The above three-dimensional data can be point cloud data, mesh data, volume data, etc.
[0055] In a specific implementation, first, a sample image needs to be collected by a camera, and at the same time, three-dimensional data corresponding to the sample image needs to be collected in the same scene as the sample image by a laser radar. After that, the three-dimensional data can be projected into the sample image to bind the three-dimensional attributes contained in the three-dimensional data with the two-dimensional attributes in the sample image to realize the projection conversion of the three-dimensional data and obtain the ground truth data corresponding to the sample image.
[0056] Step S2: constructing a first loss function, the first loss function being used to eliminate data in the ground truth data that is located outside the visible range of the camera.
[0057] It should be understood that, taking the collection of three-dimensional data by a surround-view laser radar as an example, since the ground truth data is generated by projecting and converting the three-dimensional data corresponding to the sample image, it inherits the field of view range (i.e. 360 degrees) corresponding to the surround-view laser radar. The field of view range corresponding to a traditional monocular camera is usually 30 degrees, which results in that part of the data in the ground truth data is not matched with the field of view range of the monocular camera. If the target detection model is still trained by using the ground truth data containing this part of data, the model may not converge. If this part of data is screened by manual labeling, the training efficiency of the model will be low due to the extra time spent on screening.
[0058] Therefore, the embodiment proposes to construct a first loss function to eliminate data in the ground truth data that is located outside the visible range of the camera. Specifically, the first loss function can be constructed by observing the influence of the ground truth data located outside the visible range of the camera and the ground truth data located within the visible range of the camera on the output of the model.
[0059] Step S3: constructing a second loss function, the second loss function being used to eliminate data in the ground truth data that does not exceed a preset intersection over union with the sample image.
[0060] It should be noted that the above ground truth data can contain a target detection box, and therefore the intersection over union of the ground truth data and the sample image can be understood as: the proportion of the part of the target detection box truncated by the image boundary of the sample image to the complete part of the target detection box. For example, the ground truth data contains a target detection box corresponding to a complete vehicle, and the target detection box is truncated by half by the image boundary of the sample image, so the intersection over union of the target detection box and the sample image is 50%.
[0061] It should be understood that when the intersection-over-union of the ground truth data with the sample image exceeds a certain value (i.e., the preset intersection-over-union described above), the ground truth data can cause a misjudgment in the field of view of the camera, such as misjudging a large truck with only the front part exposed as a small car. Based on this, for the ground truth data within the visual range of the camera, there is still a case of negative impact on the model training process (such as causing the model to be forced to fit noise and reducing the sensitivity to complete targets). Therefore, the embodiment proposes to construct a second loss function to exclude data in the ground truth data whose intersection-over-union with the sample image does not exceed the preset intersection-over-union. The preset intersection-over-union can be calibrated by experiment (for example, it can be set to 75%), that is, different target detection effects are verified based on different intersection-over-unions of different target detection boxes in the ground truth data and the sample image.
[0062] Step S4: reconstructing the loss function in the initial target detection model based on the first loss function and the second loss function to obtain an improved target detection model.
[0063] It should be noted that the initial target detection model described above can be a MonoCon model. For the MonoCon model, given a sample image (for example, a 3xHxW RGB image, 3 representing the number of image channels, H representing the height of the image, and W representing the width of the image), a feature backbone can be obtained through a convolutional neural network, and a feature map of DxhXw (D represents the dimension of the output feature map, h represents the height of the output feature map, and w represents the width of the output feature map) is output, h=H / s, w=W / s, and s is the overall stride / subsampling rate of the feature backbone (for example, s=4). Then, a lightweight network head is used as a regression branch in the regression branch, and the DxhXw feature map is input into the regression branch. Each regression branch (usually composed of a small number of convolutional layers) processes the feature map described above, and finally infers a 3D bounding box according to the output of each regression branch, thereby completing 3D target detection on the sample image.
[0064] In a specific implementation, the first loss function described above can be replaced into an existing regression branch of the initial target detection model, and a new regression branch is added to the initial target detection model, and the second loss function described above is added to the new regression branch, thereby reconstructing the loss function in the initial target detection model to obtain an improved target detection model.
[0065] The embodiment obtains sample image true value data, the sample image is collected by a camera, and the true value data is generated by projecting three-dimensional data corresponding to the sample image; a first loss function is constructed, the first loss function is used to eliminate data in the true value data located outside the visual range of the camera; a second loss function is constructed, the second loss function is used to eliminate data in the true value data whose intersection over union with the sample image does not exceed a preset intersection over union; and the loss function in the initial target detection model is reconstructed based on the first loss function and the second loss function to obtain an improved target detection model. The above method of the embodiment reconstructs the loss function in the initial target detection model by constructing the first loss function and the second loss function, so that the data in the true value data located outside the visual range of the camera and the data in the true value data whose intersection over union with the sample image does not exceed the preset intersection over union can be eliminated without relying on manual work, the true value data used for training the model can be automatically matched with the camera field of view, and the training efficiency of the model is improved.
[0066] Reference Figure 2 , Figure 2 The flowchart of the second embodiment of the target detection model improvement method of the present application is shown.
[0067] In a possible implementation, the step S2 can include:
[0068] Step S21: obtaining a first prediction data set corresponding to the true value data in the case of being located within the visual range of the camera.
[0069] Step S22: obtaining a second prediction data set corresponding to the true value data in the case of being located outside the visual range of the camera.
[0070] It should be noted that the first prediction data set can be a set of prediction data corresponding to the true value data located within the visual range of the camera in the initial target detection model, and the second prediction data set can be a set of prediction data corresponding to the true value data located outside the visual range of the camera in the initial target detection model.
[0071] In a specific implementation, the visual range of the camera can be labeled in the true value data, the true value data located in the visual range region corresponds to the case of being located within the visual range of the camera, and the true value data located outside the visual range region corresponds to the case of being located outside the visual range of the camera.
[0072] Step S23: determining a conditional dependence relationship between the visual range and the prediction data based on the first prediction data set and the second prediction data set, and constructing a first loss function according to the conditional dependence relationship.
[0073] It should be understood that the first common attribute of the prediction data when located within the visual range of the camera can be determined based on the first prediction data set, the second common attribute of the prediction data when located outside the visual range of the camera can be determined based on the second prediction data set, then the first common attribute and the second common attribute are taken as the conditional dependence between the visual range and the prediction data, and finally the first loss function is constructed according to the conditional dependence. For example, it is assumed that the first common attribute is f>0 which is determined based on the first prediction data set, and the second common attribute is f<0 which is determined based on the second prediction data set, where f represents the prediction data. Therefore, the conditional dependence between the visual range and the prediction data can be determined based on the first common attribute and the second common attribute as follows: f>0 indicates that the prediction data is located within the visual range of the camera, and f<0 indicates that the prediction data is located outside the visual range of the camera. Therefore, the first loss function constructed according to the above conditional dependence can be represented as:
[0074]
[0075] where loss1 represents the first loss function, f represents the prediction data, and y represents the true value data. In particular, when f<0, loss1 is equal to 0.1, which means that the prediction data located outside the visual range of the camera is weakly punished. Here, 0 is not directly given to prevent the gradient from being 0 during network back propagation, so that the network can be normally trained and the situation that loss1 cannot be converged can be avoided. In summary, the purpose of loss1 is to eliminate the projection points that are not within the visual range of the camera.
[0076] In a possible implementation, the step S3 can include:
[0077] Step S31: randomly extracting a plurality of test images from the image database, wherein the test images contain a plurality of test bounding boxes truncated by image boundaries.
[0078] It should be noted that the test images also need to contain objects related to the detection scene, so that the finally trained model can be applied to the detection scene. The image boundary represents the image boundary of the test image.
[0079] Step S32: determining a detection index corresponding to the test bounding box when the target is detected, and an intersection over union between the test bounding box and the image boundary.
[0080] It should be noted that the detection index can include the detection rate of the object in the test image, the false detection rate, etc., which is not limited in the embodiment.
[0081] Step S33: constructing a second loss function based on the detection index and the intersection over union.
[0082] In a specific implementation, the expected indicator can be determined according to the accuracy requirement of target detection, then the intersection over union that can meet the expected indicator is determined through the detection indicator, and then the second loss function is constructed based on the intersection over union.
[0083] In a possible implementation, the step S33 can include:
[0084] Step S331: selecting a preset indicator from the detection indicators, and obtaining a preset intersection over union corresponding to the preset indicator, the preset indicator representing a critical indicator meeting the detection requirement.
[0085] It should be noted that the preset indicator can be the closest to the expected indicator among all the detection indicators, and the error between the preset indicator and the expected indicator is not more than a preset error (which can be customized according to the accuracy requirement of target detection, for example, not more than ±1% of the expected indicator).
[0086] Step S332: constructing a second loss function according to the preset intersection over union.
[0087] It should be understood that, taking the preset intersection over union of 75% as an example, the second loss function can be constructed as follows:
[0088]
[0089] wherein loss2 represents the second loss function, iou represents the intersection over union of the target detection frame in the ground truth data and the sample image, 0.75 represents the preset intersection over union. When iou < 0, loss2 is equal to 0.1, which means that the intersection over union exceeding the preset intersection over union is weakly punished. Here, 0 is not directly given to prevent the gradient from being 0 when the network is back propagated, so that the network can be normally trained and the situation that loss2 cannot be converged does not occur.
[0090] The embodiment obtains a first prediction data set corresponding to the ground truth data in a case of being located within a visual range of the camera, obtains a second prediction data set corresponding to the ground truth data in a case of being located outside the visual range of the camera, determines a conditional dependency relationship between the visual range and the prediction data based on the first prediction data set and the second prediction data set, constructs a first loss function according to the conditional dependency relationship, randomly extracts a plurality of test images from an image database, the test images contain a plurality of test bounding boxes truncated by image boundaries, determines a detection index corresponding to the test bounding box in target detection and an intersection over union between the test bounding box and the image boundary, selects a preset index from the detection index, and obtains a preset intersection over union corresponding to the preset index, the preset index represents a critical index meeting the detection requirement, and constructs a second loss function according to the preset intersection over union. The above-mentioned method of the embodiment can obtain the conditional dependency relationship between the visual range and the prediction data by observing the relationship between the ground truth data and the prediction data and the relationship between the visual range of the camera and the ground truth data, so that the first loss function constructed can accurately exclude the data in the ground truth data located outside the visual range of the camera; and the second loss function can be constructed according to the relationship between the intersection over union and the detection index in the test bounding box truncated by the image boundary, so that the second loss function can accurately exclude the data in the ground truth data with the intersection over union not exceeding the preset intersection over union with the sample image.
[0091] Reference Figure 3 , Figure 3 FIG. 1 is a flowchart of a target detection model improvement method according to a third embodiment of the present application.
[0092] In a feasible implementation manner, the step S4 can include:
[0093] Step S41: embedding the first loss function and the second loss function into an initial target detection model, and training the initial target detection model.
[0094] Step S42: determining the trained model as an improved target detection model in a case that the training is ended and a total loss value of the trained model is less than a preset threshold.
[0095] It should be noted that the above-mentioned preset threshold can be set by the user, for example, can be set as 0.1.
[0096] It should be understood that the first loss function and the second loss function can be respectively assigned with a first weight and a second weight, and the weighted first loss function and the second loss function are embedded into the initial target detection model, and the initial target detection model is trained.
[0097] In addition, the embodiment also compares the training effects of different target detection models, and the number of sample images used for training is 10,000: for the initial target detection model, the corresponding mAP (mean Average Precision) is 0.8015, and the total loss value cannot converge within 0.1 and remains at about 0.3; for the target detection model in which only loss1 is modified, the corresponding mAP is 0.8321, and the total loss value converges within 0.1 at a speed of about 15h30min; for the target detection model in which only loss2 is modified, the corresponding mAP is 0.8325, and the total loss value converges within 0.1 at a speed of about 15h20min; for the improved target detection model, the corresponding mAP is 0.8523, and the total loss value converges within 0.1 at a speed of about 15h20min. It can be seen that, compared with the initial target detection model, the total loss value of the improved target detection model converges at a higher speed and can converge within 0.1.
[0098] In a feasible implementation, the initial target detection model comprises a detection head and an auxiliary training head, and the step S41 can comprise:
[0099] Step S411: After embedding the first loss function and the second loss function into the auxiliary training head, the attributes of the sample image in the three-dimensional space are predicted by the detection head to obtain prediction data.
[0100] Step S412: The initial target detection model is trained based on the prediction data and the true value data.
[0101] It can be understood that, in the initial target detection model, the detection head can comprise five branches, wherein branch one predicts rough coordinates of a 2D frame center point and a class score through an output heat map, branch two predicts an offset between the rough coordinates of the 2D frame center point and a real 3D projection coordinate, branch three predicts a depth value of the rough coordinates of the 2D frame center point and an uncertainty thereof, branch four predicts a size of a 3D frame, and branch five predicts an observation angle. The auxiliary training head can comprise five branches, wherein branch one is used to determine eight projection angle points and a projection center of a 3D frame, branch two is used to determine an offset of the eight projection angle points to a 2D frame center, branch three is used to determine a size of a 2D frame, branch four is used to model a 2D frame center quantization error, and branch five is used to model an 8-projection angle point quantization error. In the embodiment, the first loss function can be used to replace the original loss functions in branch one and branch three of the auxiliary training head, and then a sixth branch is added to the auxiliary training head, and the second loss function is added to the sixth branch, so as to embed the first loss function and the second loss function into the auxiliary training head.
[0102] The first loss function and the second loss function are embedded into the initial target detection model, and the initial target detection model is trained; in the case that the training is completed and the total loss value of the trained model is less than a preset threshold, the trained model is determined as the improved target detection model; the initial target detection model comprises a detection head and an auxiliary training head, and the step of embedding the first loss function and the second loss function into the initial target detection model and training the initial target detection model comprises: embedding the first loss function and the second loss function into the auxiliary training head, predicting the attributes of the sample image in the three-dimensional space through the detection head to obtain prediction data, and training the initial target detection model based on the prediction data and the true value data. The above method of the embodiment can accurately determine whether the model converges according to the total loss value of the trained model, so as to obtain the improved target detection model. Meanwhile, the initial target detection model is divided into the detection head and the auxiliary training head in the embodiment, the auxiliary training head is only effective in the training stage, the supervision signal is increased without increasing the inference calculation amount, and thus the training speed of the model is improved.
[0103] In addition, the embodiment further provides a target detection method, which comprises the following steps:
[0104] Step S10: inputting a target image collected by a vehicle in a driving scene into the improved target detection model to obtain a model output result.
[0105] Step S20: generating a three-dimensional bounding box corresponding to the target image according to the model output result.
[0106] The improved target detection model is obtained by reconstructing a loss function in an initial target detection model based on a first loss function and a second loss function, the first loss function is used to eliminate data in true value data that is located outside a visible range of a camera, the second loss function is used to eliminate data in the true value data that does not exceed a preset intersection over union with a sample image, the true value data is generated by projecting three-dimensional data corresponding to the sample image, and the sample image is collected by a camera.
[0107] It should be understood that the specific acquisition method of the improved target detection model can refer to the first embodiment, the second embodiment and the third embodiment of the target detection model improvement method, which will not be repeated here. It should be emphasized that the improved target detection model mentioned in the target detection model improvement method is the same model as the improved target detection model mentioned in the target detection method, and the corresponding implementation manners and technical effects are also the same.
[0108] In a specific implementation, after inputting a target image collected by a vehicle in a driving scene into the improved target detection model, the model outputs prediction data of each branch for a target detection object in the target image, and labels the prediction data to the target image respectively, so as to generate a three-dimensional bounding box corresponding to the target image.
[0109] The target detection method provided in the embodiment is based on the first loss function and the second loss function to reconstruct the loss function in the initial target detection model, and obtain the improved target detection model, and based on the improved target detection model, the detection accuracy of target detection of the vehicle in the driving scene can be improved.
[0110] Reference Figure 4 , Figure 4 FIG. 1 is a structural block diagram of a first embodiment of a target detection model improvement device according to the present application.
[0111] As shown in FIG. 1, the target detection model improvement device provided in the embodiment of the present application comprises: Figure 4
[0112] The data acquisition module 401 is configured to acquire ground truth data of a sample image, the sample image is collected by a camera, and the ground truth data is generated by projecting three-dimensional data corresponding to the sample image;
[0113] The first function construction module 402 is configured to construct a first loss function, and the first loss function is used to eliminate data in the ground truth data which is located outside a visible range of the camera;
[0114] The second function construction module 403 is configured to construct a second loss function, and the second loss function is used to eliminate data in the ground truth data which does not exceed a preset intersection over union with the sample image;
[0115] The model improvement module 404 is configured to reconstruct a loss function in an initial target detection model based on the first loss function and the second loss function, and obtain an improved target detection model.
[0116] The embodiment obtains sample images collected by a camera, and true value data of the sample images is generated by projecting three-dimensional data corresponding to the sample images; a first loss function is constructed, the first loss function is used to eliminate data in the true value data located outside a visible range of the camera; a second loss function is constructed, the second loss function is used to eliminate data in the true value data with an intersection over union less than a preset intersection over union; and a loss function in an initial target detection model is reconstructed based on the first loss function and the second loss function, to obtain an improved target detection model. The above method reconstructs the loss function in the initial target detection model by the first loss function and the second loss function constructed, so that the data in the true value data located outside the visible range of the camera and the data in the true value data with the intersection over union less than the preset intersection over union can be eliminated without relying on manual work, the true value data used for training the model can be automatically matched with the field of view of the camera, and the training efficiency of the model is improved.
[0117] Based on the first embodiment of the target detection model improvement device, the second embodiment of the target detection model improvement device is provided.
[0118] In the embodiment, the first function construction module 402 is further configured to obtain a first prediction data set corresponding to the true value data in a case of being located within the visible range of the camera; obtain a second prediction data set corresponding to the true value data in a case of being located outside the visible range of the camera; determine a conditional dependence relationship between the visible range and the prediction data based on the first prediction data set and the second prediction data set, and construct a first loss function according to the conditional dependence relationship.
[0119] Further, the second function construction module 403 is further configured to randomly extract a plurality of test images from an image database, the test images containing a plurality of test detection boxes truncated by image boundaries; determine a detection index corresponding to the test detection box in target detection, and an intersection over union between the test detection box and the image boundary; and construct a second loss function based on the detection index and the intersection over union.
[0120] Further, the second function construction module 403 is further configured to select a preset index from the detection index, and obtain a preset intersection over union corresponding to the preset index, the preset index representing a critical index meeting the detection requirement; and construct a second loss function according to the preset intersection over union.
[0121] Further, the model improvement module 404 is further configured to embed the first loss function and the second loss function into an initial target detection model, and train the initial target detection model; and in a case where training is completed and a total loss value of the trained model is less than a preset threshold, determine the trained model as an improved target detection model.
[0122] Further, the model improvement module 404 is further configured to embed the first loss function and the second loss function into the auxiliary training head, predict an attribute of the sample image in a three-dimensional space through the detection head to obtain prediction data, and train the initial target detection model based on the prediction data and the true value data.
[0123] Other embodiments or specific implementations of the target detection model improvement apparatus provided in the present application can refer to the above-mentioned method embodiments, and will not be described here again.
[0124] The present application provides a target detection model improvement device, which comprises at least one processor and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the target detection model improvement method in Embodiment I.
[0125] Reference will be made to the following description of the drawings Figure 5 , which shows a structural schematic diagram of a target detection model improvement device suitable for implementing embodiments of the present application. The target detection model improvement device in the embodiments of the present application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 5 The target detection model improvement device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.
[0126] As Figure 5As shown, the target detection model improvement device can include a processing apparatus 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 or loaded from a storage apparatus 1003 into a random access memory 1004. Various programs and data required for the operation of the target detection model improvement device are also stored in the random access memory 1004. The processing apparatus 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other by a bus 1005. An input / output interface 1006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 1006: input apparatuses 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output apparatuses 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 1009. The communication apparatus 1009 can allow the target detection model improvement device to communicate with other devices wirelessly or by wire to exchange data. Although the target detection model improvement device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.
[0127] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by a communication apparatus, or installed from the storage apparatus 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing apparatus 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.
[0128] The target detection model improvement device provided in the present application adopts the target detection model improvement method in the above-mentioned embodiments, and can solve the technical problem of low training efficiency of the existing target detection model. Compared with the prior art, the target detection model improvement device provided in the present application has the same beneficial effects as the target detection model improvement method provided in the above-mentioned embodiments, and other technical features in the target detection model improvement device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.
[0129] It should be understood that various parts of the present disclosure can be realized in hardware, software, firmware, or a combination thereof. In the above description of the embodiments, specific features, structures, materials or characteristics can be combined in any appropriate manner in any one or more embodiments or examples.
[0130] The above description is merely illustrative of the application and the application should not be limited thereto. Any variations and modifications of the application, which fall within the technical scope of the application, should be covered by the scope of the application. Therefore, the scope of the application should be determined by the scope of the claims.
[0131] The application provides a computer readable storage medium having computer readable program instructions (i.e., a computer program) stored thereon, the computer readable program instructions being used to perform the object detection model improvement method in the above-described embodiments.
[0132] The computer readable storage medium provided by the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system or device, or any combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection having one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer readable storage medium can be transmitted by any appropriate medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), etc., or any appropriate combination thereof.
[0133] The above-described computer readable storage medium can be included in the object detection model improvement device; or can exist separately and not be assembled into the object detection model improvement device.
[0134] The computer-readable storage medium described above can bear one or more programs, when the one or more programs are executed by the target detection model improvement device, the target detection model improvement device can be written in one or more programming languages or combinations thereof for computer program codes for executing the operations of the present application, the programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. The program code can be executed completely on a user computer, partially on a user computer, as an independent software package, partially on a user computer and partially on a remote computer, or completely on a remote computer or server. In the case involving a remote computer, the remote computer can be connected to the user computer through any kind of network, for example, through a local area network (LAN) or a wide area network (WAN), or connected to an external computer (for example, connected through the Internet by using an Internet service provider).
[0135] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a part of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than those noted in the accompanying drawings. For example, two blocks represented in succession can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.
[0136] The modules involved in the embodiments of the present application can be implemented in software or hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.
[0137] The readable storage medium provided by the application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the target detection model improvement method described above, and can solve the technical problem of low training efficiency of the existing target detection model. Compared with the prior art, the computer readable storage medium provided by the application has the same beneficial effects as the target detection model improvement method provided by the above-mentioned embodiments, and will not be repeated here.
[0138] The application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the target detection model improvement method as described above.
[0139] The computer program product provided by the application can solve the technical problem of target detection model improvement. Compared with the prior art, the computer program product provided by the application has the same beneficial effects as the target detection model improvement method provided by the above-mentioned embodiments, and will not be repeated here.
[0140] The above-mentioned is only part of the embodiments of the application, and does not limit the protection scope of the application, and any equivalent structural transformation, direct / indirect application in other related technical fields within the technical concept of the application, and the contents of the specification and drawings of the application are included in the protection scope of the application.
Claims
1. A method for improving a target detection model, characterized in that, The method comprises the following steps: Obtaining ground truth data of a sample image, the sample image being collected by a camera, the ground truth data being generated by projecting three-dimensional data corresponding to the sample image; Constructing a first loss function for eliminating data in the ground truth data that is outside a visible range of the camera; Constructing a second loss function for eliminating data in the ground truth data that has an intersection-over-union (IoU) with the sample image that does not exceed a preset IoU; Reconstructing a loss function in an initial target detection model based on the first loss function and the second loss function to obtain an improved target detection model. 2.The target detection model improvement method of claim 1, wherein, The step of constructing the first loss function comprises: Obtaining a first prediction data set corresponding to the ground truth data under the condition that the ground truth data is within the visible range of the camera; Obtaining a second prediction data set corresponding to the ground truth data under the condition that the ground truth data is outside the visible range of the camera; Determining a conditional dependence relationship between the visible range and the prediction data based on the first prediction data set and the second prediction data set, and constructing the first loss function according to the conditional dependence relationship. 3.The target detection model improvement method of claim 1, wherein, The step of constructing the second loss function comprises: Randomly extracting a plurality of test images from an image database, the test images containing a plurality of test bounding boxes that are truncated by image boundaries; Determining a detection index corresponding to the test bounding boxes in target detection, and an intersection-over-union (IoU) between the test bounding boxes and the image boundaries; Constructing the second loss function based on the detection index and the IoU. 4.The target detection model improvement method of claim 3, wherein, The step of constructing the second loss function based on the detection index and the IoU comprises: Selecting a preset index from the detection index, the preset index representing a critical index that meets detection requirements, and obtaining a preset IoU corresponding to the preset index; Constructing the second loss function according to the preset IoU. 5.The target detection model improvement method of claim 1, wherein, The step of reconstructing a loss function in an initial target detection model based on the first loss function and the second loss function to obtain an improved target detection model comprises: Embedding the first loss function and the second loss function into the initial target detection model, and training the initial target detection model; In a case where training is completed and a total loss value of a trained model is less than a preset threshold, determining the trained model as the improved target detection model. 6.The target detection model improvement method of claim 5, wherein, The initial target detection model comprises a detection head and an auxiliary training head, and the step of embedding the first loss function and the second loss function into the initial target detection model and training the initial target detection model comprises: Embedding the first loss function and the second loss function into the auxiliary training head, predicting properties of the sample image in a three-dimensional space through the detection head to obtain prediction data; Training the initial target detection model based on the prediction data and the ground truth data.
7. A target detection method characterized by, The method comprises the following steps: Inputting a target image collected by a vehicle in a driving scene into the improved target detection model to obtain a model output result; According to the model output result, a three-dimensional bounding box corresponding to the target image is generated; The improved target detection model is obtained by reconstructing a loss function in an initial target detection model based on a first loss function and a second loss function, the first loss function is used to eliminate data in ground truth data that is located outside a visible range of a camera, the second loss function is used to eliminate data in the ground truth data that has an intersection over union with a sample image that does not exceed a preset intersection over union, the ground truth data is generated by projecting three-dimensional data corresponding to the sample image, and the sample image is collected by the camera.
8. An object detection model improvement apparatus, characterized by comprising: The target detection model improvement device comprises: A data acquisition module is configured to acquire ground truth data of a sample image, the sample image is collected by a camera, and the ground truth data is generated by projecting three-dimensional data corresponding to the sample image; A first function construction module is configured to construct a first loss function, the first loss function is used to eliminate data in the ground truth data that is located outside a visible range of the camera; A second function construction module is configured to construct a second loss function, the second loss function is used to eliminate data in the ground truth data that has an intersection over union with the sample image that does not exceed a preset intersection over union; A model improvement module is configured to reconstruct a loss function in an initial target detection model based on the first loss function and the second loss function to obtain an improved target detection model. 9.A target detection model improvement device characterized by comprising: The device comprises a memory, a processor, and a target detection model improvement program stored on the memory and executable on the processor, the target detection model improvement program is configured to implement the steps of the target detection model improvement method according to any one of claims 1 to 6.
10. A storage medium, characterized by The storage medium is a computer readable storage medium, the storage medium stores a target detection model improvement program, and the target detection model improvement program implements the steps of the target detection model improvement method according to any one of claims 1 to 6 when executed by a processor.