Method and device for generating difficult example samples, computer device and storage medium thereof

Abstract

The application relates to a difficult example sample generation method and device, computer equipment and a storage medium thereof, and relates to the technical field of data analysis. The method comprises the following steps: acquiring at least one object detection result of each identification object in to-be-mined data; determining a target detection bounding box of each identification object according to the overlapping degree of each candidate detection bounding box of each identification object and the true value bounding box of each identification object; determining a difficult example bounding box from the target detection bounding boxes according to at least one of the corresponding overlapping degree, confidence and object category of each target detection bounding box; and extracting a difficult example sample from the to-be-mined data according to the true value bounding box corresponding to the difficult example detection bounding box. The application realizes the operation of screening difficult example samples from to-be-mined data, and guarantees the efficiency and accuracy of difficult example sample acquisition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, and in particular to a difficult example sample generation method and device, computer equipment and a storage medium thereof. BACKGROUND

[0002] With the continuous development of the field of autonomous driving, people have higher requirements for the intelligence of autonomous driving; in order to improve the intelligence of autonomous driving, the perception model of autonomous driving can be trained through deep learning technology, so as to improve the perception accuracy of the perception model and achieve the purpose of improving the intelligence of autonomous driving.

[0003] In the prior art, as the perception model of autonomous driving is continuously deepened, simply increasing the model training samples cannot meet the training needs of the perception model, therefore, in order to effectively improve the perception accuracy of the perception model, difficult example samples need to be selected from the training data, so as to train the perception model according to the difficult example samples.

[0004] However, the efficiency of the staff in selecting difficult example samples from the training data is low, and it is difficult to ensure the accuracy of the selection of difficult example samples. SUMMARY

[0005] Therefore, it is necessary to provide a difficult example sample generation method and device, computer equipment and a storage medium thereof, which can accurately obtain difficult example samples, in order to solve the above technical problems.

[0006] In a first aspect, the present application provides a difficult example sample generation method. The method comprises:

[0007] Obtaining at least one object detection result of each recognition object in the to-be-mined data; wherein the object detection result comprises: each candidate detection bounding box of the recognition object, the confidence of the candidate detection bounding box, and the object category;

[0008] Determining a target detection bounding box of each recognition object according to the overlap degree of each candidate detection bounding box of each recognition object and the ground truth bounding box of each recognition object;

[0009] Determining a difficult example bounding box from each target detection bounding box according to at least one of the overlap degree, the confidence, and the object category corresponding to each target detection bounding box;

[0010] Extracting a difficult example sample from the to-be-mined data according to the ground truth bounding box corresponding to the difficult example detection bounding box.

[0011] In one embodiment, the target detection bounding box of each recognition object is determined according to the overlap degree of each candidate detection bounding box of each recognition object and the ground truth bounding box of each recognition object, comprising

[0012] For each identified object, determine an overlap degree between each candidate detection bounding box of the identified object and the ground truth bounding box;

[0013] The candidate detection bounding box with the largest overlap degree is taken as a target detection bounding box of the identified object.

[0014] In one of the embodiments, the difficult example bounding box is determined from the target detection bounding boxes according to at least one of the overlap degrees, the confidence degrees and the object categories corresponding to the target detection bounding boxes, including

[0015] Determine a first size relationship between the overlap degree of each target detection bounding box and an overlap degree threshold value;

[0016] Determine a second size relationship between the detection confidence degree of each target detection bounding box and a confidence degree threshold value;

[0017] Determine a belonging relationship between the predicted category of each target detection bounding box and the ground truth category;

[0018] The difficult example bounding box is determined from the target detection bounding boxes according to at least one of the first size relationship, the second size relationship and the belonging relationship.

[0019] In one of the embodiments, the difficult example bounding box is determined from the target detection bounding boxes according to the first size relationship, the second size relationship and the belonging relationship;

[0020] For each target detection bounding box, if the first size relationship is that the overlap degree of the target detection bounding box is less than the overlap degree threshold value, it is determined that the target detection bounding box satisfies a difficult example condition;

[0021] If the second size relationship is that the detection confidence degree of the target detection bounding box is less than the confidence degree threshold value, it is determined that the target detection bounding box satisfies the difficult example condition;

[0022] If the belonging relationship is that the predicted category of the target detection bounding box is different from the ground truth category, it is determined that the target detection bounding box satisfies the difficult example condition;

[0023] If the number of times that the target detection bounding box satisfies the difficult example condition is greater than a number threshold value, it is determined that the target detection bounding box is a difficult example bounding box.

[0024] In one of the embodiments, at least one candidate detection bounding box of each identified object in the data to be mined is obtained, including:

[0025] The object recognition is performed on the data to be mined through a plurality of neural networks to obtain at least one candidate detection bounding box of each candidate object in the data to be mined;

[0026] At least one identified object is determined from the candidate objects according to a difficult example category requirement;

[0027] The at least one candidate detection bounding box of each recognition object is extracted from the at least one candidate detection bounding box of each candidate object.

[0028] In one embodiment, the hard example sample is generated according to hard example data in the data to be mined, which belongs to the hard example detection bounding box, and includes the following steps:

[0029] The data corresponding to the ground truth bounding box corresponding to the hard example detection bounding box in the data to be mined is extracted as the hard example data.

[0030] The hard example sample is generated according to the hard example data, hard example position information of the hard example data in the data to be mined, the ground truth bounding box corresponding to the hard example detection bounding box, the object category of the recognition object, and the number of the hard example data.

[0031] In one embodiment, the data corresponding to the ground truth bounding box corresponding to the hard example detection bounding box in the data to be mined is extracted as the hard example data, and includes the following steps:

[0032] The bounding box position information of the ground truth bounding box corresponding to the hard example detection bounding box in the data to be mined is determined, and candidate position information of each candidate data in the data to be mined is determined.

[0033] The hard example data of the ground truth bounding box corresponding to the hard example detection bounding box in the data to be mined is determined according to the bounding box position information and the candidate position information.

[0034] In a second aspect, the present application further provides a hard example sample generation device. The device includes:

[0035] An acquisition module is configured to acquire at least one object detection result of each recognition object in data to be mined, wherein the object detection result includes a candidate detection bounding box of the recognition object, a confidence of the candidate detection bounding box, and an object category.

[0036] A first determination module is configured to determine a target detection bounding box of each recognition object according to an overlap degree of each candidate detection bounding box of each recognition object and a ground truth bounding box of each recognition object.

[0037] A second determination module is configured to determine a hard example bounding box from each target detection bounding box according to at least one of an overlap degree, a confidence, and an object category corresponding to each target detection bounding box.

[0038] An extraction module is configured to extract a hard example sample from the data to be mined according to a ground truth bounding box corresponding to the hard example detection bounding box.

[0039] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor. The memory stores a computer program. The processor implements the following steps when executing the computer program:

[0040] At least one object detection result of each identified object in the to-be-mined data is obtained. The object detection result comprises a candidate detection bounding box of the identified object, a confidence of the candidate detection bounding box, and an object category.

[0041] A target detection bounding box of each identified object is determined according to the overlap degree of each candidate detection bounding box of each identified object and the ground truth bounding box of each identified object.

[0042] A difficult example bounding box is determined from the target detection bounding boxes according to at least one of the overlap degree, the confidence, and the object category corresponding to each target detection bounding box.

[0043] A difficult example sample is extracted from the to-be-mined data according to the ground truth bounding box corresponding to the difficult example detection bounding box.

[0044] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the following steps:

[0045] At least one object detection result of each identified object in the to-be-mined data is obtained. The object detection result comprises a candidate detection bounding box of the identified object, a confidence of the candidate detection bounding box, and an object category.

[0046] A target detection bounding box of each identified object is determined according to the overlap degree of each candidate detection bounding box of each identified object and the ground truth bounding box of each identified object.

[0047] A difficult example bounding box is determined from the target detection bounding boxes according to at least one of the overlap degree, the confidence, and the object category corresponding to each target detection bounding box.

[0048] A difficult example sample is extracted from the to-be-mined data according to the ground truth bounding box corresponding to the difficult example detection bounding box.

[0049] At least one object detection result of each identified object in the to-be-mined data is obtained. The object detection result comprises a candidate detection bounding box of the identified object, a confidence of the candidate detection bounding box, and an object category.

[0050] A target detection bounding box of each identified object is determined according to the overlap degree of each candidate detection bounding box of each identified object and the ground truth bounding box of each identified object.

[0051] determine a difficult example bounding box from the target detection bounding boxes according to at least one of the overlap degree, the confidence and the object category corresponding to each target detection bounding box;

[0052] extract the difficult example sample from the data to be mined according to the ground truth bounding box corresponding to the difficult example detection bounding box.

[0053] The method, device, computer device and storage medium for generating a difficult example sample provided in the embodiments of the present application determine a target detection bounding box for each recognition object by obtaining at least one object detection result of each recognition object in the data to be mined, determine a difficult example bounding box from the target detection bounding boxes, and then extract a difficult example sample from the data to be mined according to the ground truth bounding box corresponding to the difficult example detection bounding box. In the process of obtaining the difficult example sample, first, the target detection bounding box with the highest overlap degree with the ground truth bounding box of each recognition object is selected from each candidate detection bounding box of the recognition object, and then the target detection bounding box with the highest overlap degree is subjected to difficult example detection according to at least one of the overlap degree, the confidence and the object category corresponding to each target detection bounding box. If there is still a difficult example bounding box that meets the difficult example condition, it means that there is still a bounding box that does not meet the detection accuracy requirement of the overlap degree, the confidence and the object category in the target detection bounding box with the highest overlap degree with the ground truth bounding box. Therefore, the bounding box that does not meet the detection accuracy requirement is taken as the difficult example detection bounding box, and at this time, the difficult example sample can be extracted from the ground truth bounding box corresponding to the difficult example detection bounding box. In this way, the operation of effectively screening the difficult example sample from the data to be mined is realized, and the efficiency and accuracy of obtaining the difficult example sample are ensured. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 An application environment diagram of the method for generating a difficult example sample provided in the embodiments of the present application;

[0055] Figure 2 A flowchart of the method for generating a difficult example sample provided in the embodiments of the present application;

[0056] Figure 3 A step flowchart of determining a target detection bounding box provided in the embodiments of the present application;

[0057] Figure 4 A step flowchart of determining a difficult example bounding box provided in the embodiments of the present application;

[0058] Figure 5 A step flowchart of determining a candidate detection bounding box provided in the embodiments of the present application;

[0059] Figure 6 A step flowchart of generating a difficult example sample provided in the embodiments of the present application;

[0060] Figure 7 A flow chart of another example sample generation method provided by an embodiment of the present application is shown in FIG. 6.

[0061] Figure 8 A structural block diagram of a first example sample generation device provided by an embodiment of the present application is shown in FIG. 7.

[0062] Figure 9 A structural block diagram of a second example sample generation device provided by an embodiment of the present application is shown in FIG. 8.

[0063] Figure 10 A structural block diagram of a third example sample generation device provided by an embodiment of the present application is shown in FIG. 9.

[0064] Figure 11 A structural block diagram of a fourth example sample generation device provided by an embodiment of the present application is shown in FIG. 10.

[0065] Figure 12 A structural block diagram of a fifth example sample generation device provided by an embodiment of the present application is shown in FIG. 11.

[0066] Figure 13 A structural block diagram of a sixth example sample generation device provided by an embodiment of the present application is shown in FIG. 12.

[0067] Figure 14 A structural block diagram of a seventh example sample generation device provided by an embodiment of the present application is shown in FIG. 13.

[0068] Figure 15 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0069] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0070] It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In the description of the present application, the description of the terms “one embodiment”, “some embodiments”, “an example”, “a specific example”, or “some examples” means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0071] Based on the above, the difficult example sample generation method provided by the embodiments of the present application can be applied to the application environment as shown in Figure 1 In one embodiment, a computer device is provided, which can be a server, and its internal structure diagram can be as shown in Figure 1 The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the acquisition data of the difficult example sample generation method. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a difficult example sample generation method.

[0072] The present application discloses a difficult example sample generation method, device, computer device and storage medium thereof. The computer device acquires at least one object detection result of each recognition object in the to-be-mined data, determines a target detection bounding box of each recognition object, determines a difficult example bounding box from each target detection bounding box, and then extracts a difficult example sample from the to-be-mined data according to the ground truth bounding box corresponding to the difficult example detection bounding box.

[0073] In one embodiment, as shown in Figure 2 , Figure 2 A flowchart of the difficult example sample generation method provided by the embodiments of the present application provides a difficult example sample generation method, Figure 1 The difficult example sample generation method executed by the computer device in the present application can include the following steps:

[0074] Step 201, acquiring at least one object detection result of each recognition object in the to-be-mined data; wherein the object detection result includes: each candidate detection bounding box of the recognition object, the detection confidence of the candidate detection bounding box and the object class.

[0075] The candidate detection bounding box of the recognition object refers to the predicted position of the recognition object.

[0076] The detection confidence of the candidate detection bounding box refers to the prediction accuracy of the object class corresponding to the candidate detection bounding box.

[0077] The object class of the candidate detection bounding box refers to the predicted class of the recognition object in the candidate detection bounding box.

[0078] It should be noted that when it is necessary to obtain at least one object detection result of each identified object in the to-be-mined data, the identified object in the to-be-mined data can be identified according to a pre-trained neural network, and an output result of the neural network is obtained, which is the at least one object detection result of each identified object.

[0079] The neural network can be trained according to the pre-set sample data, the sample bounding box of the sample object, the detection confidence of the sample bounding box, and the sample category, so as to obtain the trained neural network, so as to ensure that the neural network can predict the input data and output the candidate detection bounding box of the identified object of the data, the detection confidence of the candidate detection bounding box, and the object category.

[0080] It should be noted that when it is necessary to obtain at least one object detection result of each identified object in the to-be-mined data, the identified object in the to-be-mined data can be identified according to a pre-trained neural network, and an output result of the neural network is obtained, which is the at least one object detection result of each identified object.

[0081] In an embodiment of the present application, if two candidate objects and at least one candidate detection bounding box of the two candidate objects are identified in the to-be-mined data, the two candidate objects are a human object and a vehicle object, and it is determined that the pre-set difficult example category requirement is a vehicle, the vehicle object in the candidate object is taken as an identified object, and at least one candidate detection bounding box corresponding to the vehicle object is selected from the at least one candidate detection bounding box of the two candidate objects.

[0082] Step 202, determining the target detection bounding box of each identified object according to the overlap degree of each candidate detection bounding box of each identified object and the ground truth bounding box of each identified object.

[0083] The ground truth bounding box of the identified object refers to a bounding box that can correctly represent the position of the identified object. Further, compared with the candidate detection bounding box of the identified object, the ground truth bounding box can truly reflect the position of the identified object, while the candidate detection bounding box is only the predicted position of the identified object.

[0084] It should be noted that when it is necessary to determine the target detection bounding box of each identified object, the candidate detection bounding box with the maximum overlap degree with the ground truth bounding box of each identified object can be selected from each candidate detection bounding box of each identified object according to the overlap degree of each candidate detection bounding box of each identified object and the ground truth bounding box of each identified object. The candidate detection bounding box is the target detection bounding box of each identified object.

[0085] For example, if it is necessary to determine the target detection bounding box of the identified object A, the overlap degrees of each candidate detection bounding box of the identified object A and the ground truth bounding box of the identified object A are calculated. If the identified object A includes three candidate detection bounding boxes, the three candidate detection bounding boxes are candidate detection bounding box a, candidate detection bounding box b and candidate detection bounding box c, and the overlap degrees of the three candidate detection bounding boxes and the ground truth bounding box are 30%, 35% and 40% respectively, it is determined that the overlap degree of candidate detection bounding box c and the ground truth bounding box is the largest, and candidate detection bounding box c is determined as the target detection bounding box of the identified object A.

[0086] In an embodiment of the present application, after the target detection bounding box of the identified object is determined, the target detection bounding box, the detection confidence of the target detection bounding box and the object category are stored in gt_to_argmax (a function storage position) as the index of the target detection bounding box, so as to facilitate subsequent processing of the target detection bounding box in gt_to_argmax.

[0087] In step 203, at least one of the corresponding overlap degree, detection confidence and object category of each target detection bounding box is determined from each target detection bounding box to determine a hard example bounding box.

[0088] It should be noted that when it is necessary to determine a hard example bounding box from each target detection bounding box, a hard example judgment can be performed on each target detection bounding box according to at least one of the corresponding overlap degree, detection confidence and object category of each target detection bounding box, and then a hard example bounding box is determined from each target detection bounding box according to the hard example judgment result.

[0089] The hard example judgment for the overlap degree corresponding to the target detection bounding box can be that whether the overlap degree of the target detection bounding box is less than an overlap degree threshold value. Further, if the overlap degree of the target detection bounding box is less than the overlap degree threshold value, it is determined that the hard example judgment result for the overlap degree corresponding to the target detection bounding box is passed, and if the overlap degree of the target detection bounding box is not less than the overlap degree threshold value, it is determined that the hard example judgment result for the overlap degree corresponding to the target detection bounding box is not passed.

[0090] The hard example judgment for the detection confidence corresponding to the target detection bounding box can be that whether the detection confidence of the target detection bounding box is less than a confidence threshold value. Further, if the detection confidence of the target detection bounding box is less than the confidence threshold value, it is determined that the hard example judgment for the detection confidence corresponding to the target detection bounding box is passed, and if the detection confidence of the target detection bounding box is not less than the confidence threshold value, it is determined that the hard example judgment for the detection confidence corresponding to the target detection bounding box is not passed.

[0091] The difficulty example judgment for the predicted category of the target detection bounding box can be whether the predicted category of the target detection bounding box is the same as the true value category. Further, if the predicted category of the target detection bounding box is not the same as the true value category, it is determined that the difficulty example judgment for the predicted category of the target detection bounding box is passed; if the predicted category of the target detection bounding box is the same as the true value category, it is determined that the difficulty example judgment for the predicted category of the target detection bounding box is not passed.

[0092] In an embodiment of the present application, if the difficulty example judgment result of any one of the overlap degree, the detection confidence and the object category corresponding to the target detection bounding box is passed, it is determined that the target detection bounding box is a difficult example bounding box. Specifically, at least one of the overlap degree, the detection confidence and the object category corresponding to the target detection bounding box m is used to judge the target detection bounding box m, if the difficulty example judgment result of the overlap degree corresponding to the target detection bounding box m is passed, and / or the difficulty example judgment result of the detection confidence corresponding to the target detection bounding box m is passed, and / or the difficulty example judgment result of the predicted category of the target detection bounding box m is passed, it is determined that the target detection bounding box m is a difficult example bounding box.

[0093] In step 204, the difficult example sample is extracted from the to-be-mined data according to the true value bounding box corresponding to the difficult example detection bounding box.

[0094] It should be noted that when the difficult example sample is extracted from the to-be-mined data, the difficult example data belonging to the true value bounding box corresponding to the difficult example detection bounding box can be determined from the to-be-mined data, and then the difficult example sample is generated according to the difficult example data.

[0095] The difficult example sample can include but is not limited to: difficult example data, difficult example position information of the difficult example data in the to-be-mined data, the true value bounding box corresponding to the difficult example detection bounding box, the object category of the identified object, and the number of difficult example data.

[0096] Further, when it is necessary to determine the difficult example data belonging to the true value bounding box corresponding to the difficult example detection bounding box from the to-be-mined data, the bounding box position information of the true value bounding box corresponding to the difficult example detection bounding box in the to-be-mined data can be determined, and the candidate position information of each candidate data in the to-be-mined data can be determined, and then the difficult example data of the true value bounding box corresponding to the difficult example detection bounding box in the to-be-mined data is determined according to the bounding box position information and the candidate position information.

[0097] The method for generating the above difficult example sample comprises the following steps: obtaining at least one object detection result of each recognition object in the to-be-mined data, determining a target detection bounding box of each recognition object, and determining a difficult example bounding box from each target detection bounding box, and then extracting a difficult example sample from the to-be-mined data according to a ground truth bounding box corresponding to the difficult example detection bounding box. In the process of obtaining the difficult example sample, first, the target detection bounding box with the highest overlap degree with the ground truth bounding box of each recognition object is selected from each candidate detection bounding box of the recognition object, and then the target detection bounding box with the highest overlap degree is difficult example detected according to at least one of the overlap degree, the detection confidence and the object category corresponding to each target detection bounding box. If there is still a difficult example bounding box that meets the difficult example condition, it means that even the target detection bounding box with the highest overlap degree with the ground truth bounding box still has a bounding box that does not meet the detection accuracy requirement corresponding to the overlap degree, the detection confidence and the object category. Therefore, the bounding box that does not meet the detection accuracy requirement is taken as a difficult example detection bounding box. At this time, the difficult example sample can be extracted from the ground truth bounding box corresponding to the difficult example detection bounding box. In this way, the operation of effectively screening the difficult example sample from the to-be-mined data is realized, and the efficiency and accuracy of obtaining the difficult example sample are ensured.

[0098] With the increasing depth of model training of the perception model of the autonomous driving, simply increasing the model training sample cannot meet the training requirement of the perception model. Therefore, in order to effectively improve the perception accuracy of the perception model, the staff needs to screen difficult example samples from the training data. However, the screening efficiency of the difficult example sample is low, and it is difficult to ensure the accuracy of the screening of the difficult example sample. To solve the above technical problem, the computer device of the embodiment can determine the target detection bounding box of each recognition object according to the overlap degree between each candidate detection bounding box of each recognition object and the ground truth bounding box of each recognition object in the manner shown in Figure 3 The computer device of the embodiment can determine the target detection bounding box of each recognition object according to the overlap degree between each candidate detection bounding box of each recognition object and the ground truth bounding box of each recognition object in the manner shown in

[0099] In step 301, for each recognition object, the overlap degree between each candidate detection bounding box of the recognition object and the ground truth bounding box is determined.

[0100] In an embodiment of the present application, when it is necessary to determine the overlap degree between each candidate detection bounding box of the recognition object and the ground truth bounding box, the overlap degree between each candidate detection bounding box of the recognition object and the ground truth bounding box can be calculated by using the bbox_overlaps python (a function for calculating the overlap degree of the bounding box).

[0101] It should be noted that the IOU (Intersection over Union) between each candidate detection bounding box of the identified object and the ground truth bounding box can also be calculated, and the IOU between each candidate detection bounding box of the identified object and the ground truth bounding box is used as the overlap degree between each candidate detection bounding box of the identified object and the ground truth bounding box.

[0102] Specifically, the IOU between the candidate detection bounding box and the ground truth bounding box refers to the ratio of the intersection area of the candidate detection bounding box and the ground truth bounding box to the union area of the candidate detection bounding box and the ground truth bounding box. Therefore, when the IOU between the candidate detection bounding box and the ground truth bounding box needs to be determined, the intersection area of the candidate detection bounding box and the ground truth bounding box and the union area of the candidate detection bounding box and the ground truth bounding box are calculated respectively. Then, the ratio of the intersection area to the union area is calculated, and the result is the IOU between the candidate detection bounding box and the ground truth bounding box. The IOU is the overlap degree between the candidate detection bounding box and the ground truth bounding box.

[0103] Step 302: The candidate detection bounding box with the largest overlap degree is taken as the target detection bounding box of the identified object.

[0104] In an embodiment of the present application, when the target detection bounding box of the identified object needs to be determined, the candidate detection bounding boxes corresponding to the identified object can be arranged in order of the overlap degree from large to small, and the candidate detection bounding box in the first position in the arrangement result is taken as the target detection bounding box.

[0105] It should be noted that if the overlap degrees of at least two candidate detection bounding boxes of the identified object are the same, and the overlap degrees of the at least two candidate detection bounding boxes with the same overlap degree are the largest, the at least two candidate detection bounding boxes with the largest overlap degree are taken as the target detection bounding box.

[0106] The generation method of the above difficult example sample obtains the target detection bounding box of the identified object by screening each candidate detection bounding box of the identified object. The screening condition is provided for subsequent determination of the difficult example bounding box from the target detection bounding box of the identified object, and the accuracy of the subsequent determination of the difficult example bounding box is ensured.

[0107] In an embodiment, the difficult example bounding box can be determined from the target detection bounding boxes according to at least one of the overlap degree, the detection confidence, and the object category corresponding to each target detection bounding box, as shown in Figure 4 The method comprises:

[0108] Step 401: Determine the first size relationship between the overlap degree of each target detection bounding box and the overlap degree threshold.

[0109] The overlap threshold is used to represent the minimum overlap between the target detection bounding box and the ground truth bounding box, and the overlap threshold can be set and adjusted according to the historical experience and actual situation of the staff, and the value range of the overlap threshold is not limited herein.

[0110] For example, when the overlap threshold needs to be determined, the mean value of the overlap of each target detection bounding box can be calculated, and the obtained mean value is used as the overlap threshold.

[0111] It should be noted that, since the overlap threshold is used to represent the minimum overlap between the target detection bounding box and the ground truth bounding box, the first size relationship between the overlap of the target detection bounding box and the overlap threshold can be used to determine whether the overlap of the target detection bounding box is greater than the minimum overlap between the target detection bounding box and the ground truth bounding box.

[0112] In an embodiment of the present application, if the first size relationship is that the overlap of the target detection bounding box is greater than or equal to the overlap threshold, it means that the overlap of the target detection bounding box is greater than or equal to the minimum overlap between the target detection bounding box and the ground truth bounding box, and therefore, the target detection bounding box is not a difficult example bounding box; if the first size relationship is that the overlap of the target detection bounding box is less than the overlap threshold, it means that the overlap of the target detection bounding box is less than the minimum overlap between the target detection bounding box and the ground truth bounding box, and therefore, the target detection bounding box is a difficult example bounding box.

[0113] Step 402, determining a second size relationship between the detection confidence of each target detection bounding box and the confidence threshold.

[0114] The confidence threshold is used to represent the minimum prediction accuracy of the object category corresponding to the target detection bounding box, and the confidence threshold can be set and adjusted according to the historical experience and actual situation of the staff, and the value range of the confidence threshold is not limited herein.

[0115] It should be noted that, since the confidence threshold is used to represent the minimum prediction accuracy of the object category corresponding to the target detection bounding box, the second size relationship between the detection confidence of the target detection bounding box and the confidence threshold can be used to determine whether the detection confidence of the target detection bounding box is greater than the minimum prediction accuracy of the object category corresponding to the target detection bounding box.

[0116] In an embodiment of the present application, if the second size relationship is that the detection confidence of the target detection bounding box is greater than or equal to the confidence threshold (i.e., the minimum prediction accuracy of the object class corresponding to the target detection bounding box), it indicates that the target detection bounding box is not a hard example bounding box; if the second size relationship is that the detection confidence of the target detection bounding box is less than the confidence threshold (i.e., the minimum prediction accuracy of the object class corresponding to the target detection bounding box), it indicates that the target detection bounding box is a hard example bounding box.

[0117] In step 403, the attribution relationship between the predicted class and the true value class of each target detection bounding box is determined.

[0118] The attribution relationship includes two types, the first type is that the predicted class and the true value class of the target detection bounding box are the same, and the second type is that the predicted class and the true value class of the target detection bounding box are different. Therefore, when the attribution relationship between the predicted class and the true value class of each target detection bounding box is determined, it is only necessary to compare whether the predicted class and the true value class of each target detection bounding box are the same.

[0119] In an embodiment of the present application, if the attribution relationship is that the predicted class and the true value class of the target detection bounding box are the same, it indicates that the target detection bounding box is not a hard example bounding box; if the attribution relationship is that the predicted class and the true value class of the target detection bounding box are different, it indicates that the target detection bounding box is a hard example bounding box.

[0120] In step 404, at least one of the first size relationship, the second size relationship and the attribution relationship is used to determine the hard example bounding box from the target detection bounding boxes.

[0121] In an embodiment of the present application, if the hard example bounding box is determined from the target detection bounding boxes only according to the first size relationship, the target detection bounding box with the overlap degree less than the overlap threshold can be determined as the hard example bounding box according to the first size relationship.

[0122] In another embodiment of the present application, if the hard example bounding box is determined from the target detection bounding boxes only according to the second size relationship, the target detection bounding box with the detection confidence less than the confidence threshold can be determined as the hard example bounding box according to the second size relationship.

[0123] In still another embodiment of the present application, if the hard example bounding box is determined from the target detection bounding boxes only according to the attribution relationship, the target detection bounding box with the predicted class different from the true value class is determined as the hard example bounding box.

[0124] It should be noted that when the difficult example bounding box is determined, the target detection bounding box can be determined to meet the difficult example condition according to the first size relationship, the second size relationship and the attribution relationship, and the number of times that the target detection bounding box meets the difficult example condition is determined. If the number of times that the target detection bounding box meets the difficult example condition is greater than the number threshold, the target detection bounding box is determined to be the difficult example bounding box.

[0125] Specifically, when the difficult example bounding box is determined, the following content can be further included: for each target detection bounding box, if the first size relationship is that the overlap degree of the target detection bounding box is less than the overlap degree threshold, it is determined that the target detection bounding box meets the difficult example condition; if the second size relationship is that the detection confidence of the target detection bounding box is less than the confidence threshold, it is determined that the target detection bounding box meets the difficult example condition; if the attribution relationship is that the predicted category of the target detection bounding box is different from the true value category, it is determined that the target detection bounding box meets the difficult example condition; and if the number of times that the target detection bounding box meets the difficult example condition is greater than the number threshold, the target detection bounding box is determined to be the difficult example bounding box.

[0126] In an embodiment of the present application, if the number threshold is set to two, if the number of times that the target detection bounding box meets the difficult example condition is greater than or equal to two, the target detection bounding box is determined to be the difficult example bounding box. For example, whether the target detection bounding box n is determined to be the difficult example bounding box, the first size relationship of the overlap degree of the target detection bounding box n and the overlap degree threshold, the second size relationship of the detection confidence of the target detection bounding box n and the confidence threshold, and the attribution relationship of the predicted category of the target detection bounding box n and the true value category are determined; the first size relationship of the target detection bounding box n is that the overlap degree of the target detection bounding box is less than the overlap degree threshold, but the second size relationship of the target detection bounding box n is that the detection confidence of the target detection bounding box is greater than the confidence threshold, and the attribution relationship of the target detection bounding box n is that the predicted category of the target detection bounding box is the same as the true value category. Therefore, the number of times that the target detection bounding box n meets the difficult example condition is less than two, and the target detection bounding box n is not the difficult example bounding box.

[0127] The above-mentioned difficult example sample generation method determines the first size relationship, the second size relationship and the attribution relationship, and filters the overlap degree, the detection confidence and the predicted category of each target detection bounding box according to the first size relationship, the second size relationship and the attribution relationship, so as to obtain the difficult example bounding box with poor recognition accuracy, and ensures the accuracy of the difficult example bounding box determination, thereby providing a basic guarantee for subsequent smooth generation of difficult example samples.

[0128] In an embodiment, at least one candidate detection bounding box of each recognition object in the to-be-mined data can be obtained, and specifically as shown in Figure 5 The method comprises:

[0129] At step 501, object recognition is performed on the to-be-mined data by multiple neural networks to obtain at least one candidate detection bounding box of each candidate object in the to-be-mined data.

[0130] It should be noted that the candidate object refers to an object corresponding to different object categories in the to-be-mined data.

[0131] In an embodiment of the present application, when it is necessary to determine at least one candidate detection bounding box of each candidate object in the to-be-mined data, it can be determined whether a historical recognition database exists in the to-be-mined data. If the historical recognition database exists, at least one candidate detection bounding box of each candidate object can be obtained in the historical recognition database corresponding to the to-be-mined data.

[0132] In the historical recognition database, at least one candidate detection bounding box obtained after target recognition for different object categories in the to-be-mined data is stored. Therefore, each candidate object in the to-be-mined data and at least one candidate detection bounding box of each candidate object are stored in the historical recognition database.

[0133] Further, if the to-be-mined data does not have a historical recognition database, object recognition can be performed on each candidate object in the to-be-mined data according to the object recognition network, and the recognition result of the object recognition network is obtained. The recognition result is at least one candidate detection bounding box of each candidate object in the to-be-mined data.

[0134] In the training process of the object recognition network, the following contents can be included: generating an identification sample data corresponding to each object category according to each object category of all candidate objects, and a sample true value bounding box corresponding to the identification, inputting the identification sample data corresponding to each object category and the sample true value bounding box corresponding to the identification into the object recognition network, performing model training on the object recognition network, and obtaining the trained object recognition network.

[0135] At step 502, at least one recognition object is determined from the candidate objects according to the difficult example category requirement.

[0136] It should be noted that the difficult example category requirement is an object category corresponding to a pre-set difficult example sample. Therefore, when it is necessary to determine the recognition object, it is necessary to determine the object category corresponding to the pre-set difficult example sample according to the difficult example category requirement, and then determine the candidate object whose object category is the same as the object category of the difficult example category requirement. The candidate object is the recognition object.

[0137] For example, if the to-be-mined data contains three types of candidate objects, i.e., a category A candidate object, a category B candidate object, and a category C candidate object, and according to the difficult example category requirement, the object category corresponding to the difficult example sample is category C, the category C candidate object is taken as the recognition object.

[0138] At step 503, at least one candidate detection bounding box of each recognition object is extracted from at least one candidate detection bounding box of each candidate object.

[0139] It should be noted that, since the candidate object meeting the difficult example category requirement is taken as the recognition object, at least one candidate detection bounding box of the candidate object meeting the difficult example category requirement is at least one candidate detection bounding box of the recognition object.

[0140] The above method for generating a difficult example sample screens the recognition object from the candidate object according to the difficult example category requirement, ensures that the subsequently generated difficult example sample meets the use requirement of the worker, and further improves the convenience and quality of the user in obtaining the difficult example sample.

[0141] In one embodiment, a difficult example sample can be generated according to difficult example data belonging to the difficult example detection bounding box in the to-be-mined data, and specifically as shown in Figure 6 The method includes the following steps.

[0142] At step 601, the data corresponding to the ground truth bounding box corresponding to the difficult example detection bounding box in the to-be-mined data is extracted as the difficult example data.

[0143] It should be noted that, when it is necessary to determine the difficult example data, the data belonging to the ground truth bounding box corresponding to the difficult example detection bounding box can be determined from the to-be-mined data, and the data is the difficult example data.

[0144] In one embodiment of the present application, when it is necessary to determine the difficult example data, the following content can be specifically included: determining the bounding box position information of the ground truth bounding box corresponding to the difficult example detection bounding box in the to-be-mined data, and determining the candidate position information of each candidate data in the to-be-mined data; determining the difficult example data of the ground truth bounding box corresponding to the difficult example detection bounding box in the to-be-mined data according to the bounding box position information and the candidate position information.

[0145] Specifically, when the difficult example data of the ground truth bounding box corresponding to the difficult example detection bounding box in the to-be-mined data is determined according to the bounding box position information and the candidate position information, whether the candidate position information is located within the bounding box position information can be determined according to the bounding box position information and the candidate position information, if yes, it is determined that the candidate data is located within the ground truth bounding box corresponding to the difficult example detection bounding box, and if not, it is determined that the candidate data is not located within the ground truth bounding box corresponding to the difficult example detection bounding box.

[0146] In another embodiment of the present application, the output tensor point_indices corresponding to each candidate data in the data to be mined can be determined through relationship analysis network, and the dimension of the output tensor point_indices is: [npoints, nboxes], wherein npoints represents the number of points in the point cloud of this frame, and nboxes represents the number of difficult example bounding boxes. The positional relationship between the candidate data and the ground truth bounding box corresponding to the difficult example detection bounding box is determined according to the output tensor point_indices.

[0147] For example, if it is determined whether the Mth candidate data in the data to be mined is located in the ground truth bounding box corresponding to the Nth difficult example detection bounding box, if point_indices[m, n] = 1, it indicates that the Mth candidate data in the data to be mined is located in the ground truth bounding box corresponding to the Nth difficult example detection bounding box; if point_indices[m, n] = 0, it indicates that the Mth candidate data in the data to be mined is not located in the ground truth bounding box corresponding to the Nth difficult example detection bounding box.

[0148] In step 602, the difficult example sample is generated according to the difficult example data, the difficult example position information of the difficult example data in the data to be mined, the ground truth bounding box corresponding to the difficult example detection bounding box, the object category of the recognized object, and the number of difficult example data.

[0149] It should be noted that the coordinate system can be established in the data to be mined, and then the difficult example position information of the difficult example data in the data to be mined is read according to the established coordinate system, so that the difficult example sample can be stored as a difficult example sample according to the difficult example data, the difficult example position information of the difficult example data in the data to be mined, the ground truth bounding box corresponding to the difficult example detection bounding box, the object category of the recognized object, and the number of difficult example data. The difficult example sample can be a PKL (a kind of storage file of python) file.

[0150] The difficult example position information of the difficult example data in the data to be mined includes difficult example center point coordinates and difficult example corner point coordinates.

[0151] In an embodiment of the present application, when the perception model of the automatic driving is trained according to the difficult example sample, the difficult example data can be transferred from the to-be-mined data to the training sample data, and therefore, the coordinate conversion of the difficult example position information of the difficult example data in the to-be-mined data is required to obtain the sample position information of the difficult example data in the training sample data. Specifically, when the coordinate conversion of the difficult example position information of the difficult example data in the to-be-mined data is required, the difficult example center point coordinate and the difficult example corner point coordinate can be determined, the difficult example corner point relative coordinate of the difficult example data in the to-be-mined data is obtained by subtracting the difficult example center point coordinate from the difficult example corner point coordinate, the sample center point coordinate is determined in advance, and the sample corner point coordinate of the difficult example data in the sample position information is obtained by adding the difficult example corner point relative coordinate of the difficult example data in the to-be-mined data to the sample center point coordinate. Further, the sample position information of the difficult example data in the training sample data is determined according to the sample center point coordinate and the sample corner point coordinate of the sample position information.

[0152] For example, if the difficult example center point coordinate of the difficult example data in the difficult example position information is (x o , y o , z o ), the difficult example corner point coordinate is (x b , y b , z b ), and therefore, the difficult example corner point relative coordinate of the difficult example data in the to-be-mined data is (x b -x o , y b -y o , z b -z o ), if the sample center point coordinate of the sample position information is (x o ', y o ', z o '), the sample corner point coordinate of the difficult example data in the training sample data is (x b -x o +x o ', y b -y o +y o ', z b -z o +z o '), and therefore, the sample position information of the difficult example data in the training sample data is determined according to (x o ', y o ', z o ) and (x b -x o +x o ', y b -y o +y o ', z b -zo +z o ') determining sample position information of the difficult example data in the training sample data.

[0153] Further, according to the first size relationship, the second size relationship and the attribution relationship, the difficult example strength of the difficult example data can be determined, and the corner point relative coordinates of the difficult example data in the to-be-mined data can be determined, and the difficult example strength and the corner point relative coordinates are stored as a BIN (binary file) file, and then the difficult example data, the file path of the BIN file, the ground truth bounding box corresponding to the difficult example detection bounding box, the object category of the recognized object, and the number of difficult example data are stored as a PKL (a storage file of Python) file, that is, the difficult example sample.

[0154] The above-mentioned generation method of the difficult example sample determines the difficult example data belonging to the ground truth bounding box corresponding to the difficult example detection bounding box, guarantees the generation of the subsequent difficult example sample, and enables the working personnel to train the perception model of the automatic driving according to the difficult example sample, so as to improve the perception accuracy of the perception model of the automatic driving.

[0155] In one embodiment, when it is necessary to extract the difficult example sample from the to-be-mined data, specifically as shown in the method of claim 1, the method comprises the following steps. Figure 7

[0156] Step 701: acquiring at least one object detection result of each recognized object in the to-be-mined data; wherein the object detection result comprises a candidate detection bounding box of the recognized object, a confidence of the candidate detection bounding box and an object category.

[0157] Step 702: determining, for each recognized object, an overlap degree between each candidate detection bounding box of the recognized object and a ground truth bounding box.

[0158] Step 703: taking the candidate detection bounding box with the largest overlap degree as a target detection bounding box of the recognized object.

[0159] Step 704: determining a first size relationship between the overlap degree of each target detection bounding box and an overlap degree threshold value.

[0160] Step 705: determining a second size relationship between the detection confidence of each target detection bounding box and a confidence threshold value.

[0161] Step 706: determining an attribution relationship between the predicted category of each target detection bounding box and a ground truth category.

[0162] Step 707: for each target detection bounding box, if the first size relationship is that the overlap degree of the target detection bounding box is less than the overlap degree threshold value, it is determined that the target detection bounding box satisfies a difficult example condition.

[0163] ​Step 708, if the second size relationship is that the detection confidence of the target detection bounding box is less than the confidence threshold, it is determined that the target detection bounding box satisfies the difficult example condition.

[0164] Step 709, if the belonging relationship is that the predicted category of the target detection bounding box is different from the true value category, it is determined that the target detection bounding box satisfies the difficult example condition.

[0165] Step 710, if the number of times that the target detection bounding box satisfies the difficult example condition is greater than the number threshold, it is determined that the target detection bounding box is a difficult example bounding box.

[0166] Step 711, the edge frame position information of the true value bounding box corresponding to the difficult example detection bounding box in the to-be-mined data is determined, and the candidate position information of each candidate data in the to-be-mined data is determined.

[0167] Step 712, according to the edge frame position information and the candidate position information, the difficult example data of the true value bounding box corresponding to the difficult example detection bounding box in the to-be-mined data is determined.

[0168] Step 713, according to the difficult example data, the difficult example position information of the difficult example data in the to-be-mined data, the true value bounding box corresponding to the difficult example detection bounding box, the object category of the identified object, and the number of difficult example data, the difficult example sample is generated.

[0169] The above difficult example sample generation method realizes determining the target detection bounding box of each identified object by obtaining at least one object detection result of each identified object in the to-be-mined data, and determining the difficult example bounding box from each target detection bounding box. Further, according to the true value bounding box corresponding to the difficult example detection bounding box, the difficult example sample is extracted from the to-be-mined data. In the above process of obtaining the difficult example sample, first, the target detection bounding box with the highest overlap degree with the true value bounding box of the identified object is selected from each candidate detection bounding box of each identified object. Further, according to at least one of the overlap degree, the confidence and the object category corresponding to each target detection bounding box, the difficult example detection is performed on the target detection bounding box with the highest overlap degree. If there is still a difficult example bounding box that satisfies the difficult example condition, it means that even the target detection bounding box with the highest overlap degree with the true value bounding box still has a bounding box that does not meet the detection accuracy requirement corresponding to the overlap degree, the confidence and the object category. Therefore, the bounding box that does not meet the detection accuracy requirement is taken as the difficult example detection bounding box. At this time, the difficult example sample can be extracted from the true value bounding box corresponding to the difficult example detection bounding box. In this way, the operation of effectively screening the difficult example sample from the to-be-mined data is realized, and the efficiency and accuracy of obtaining the difficult example sample are guaranteed.

[0170] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0171] Based on the same inventive concept, the embodiments of the present application also provide a hard example sample generation device for implementing the hard example sample generation method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more hard example sample generation device embodiments provided below can refer to the limitations of the hard example sample generation method in the above text, which will not be repeated here.

[0172] In one embodiment, as shown in Figure 8 a hard example sample generation device is provided, comprising: an acquisition module 10, a first determination module 20, a second determination module 30 and an extraction module 40, wherein:

[0173] The acquisition module 10 is configured to acquire at least one object detection result of each identified object in the to-be-mined data; wherein the object detection result comprises: a candidate detection bounding box of the identified object, a confidence of the candidate detection bounding box and an object category.

[0174] The first determination module 20 is configured to determine a target detection bounding box of each identified object according to the overlap degree of each candidate detection bounding box of each identified object and the ground truth bounding box of each identified object.

[0175] The second determination module 30 is configured to determine a hard example bounding box from the target detection bounding boxes according to at least one of the overlap degree, the confidence and the object category corresponding to each target detection bounding box.

[0176] The extraction module 40 is configured to extract a hard example sample from the to-be-mined data according to the ground truth bounding box corresponding to the hard example detection bounding box.

[0177] The generation device of the above difficult example sample is used to obtain at least one object detection result of each recognition object in the to-be-mined data, determine a target detection bounding box of each recognition object, and determine a difficult example bounding box from each target detection bounding box. Then, the difficult example sample is extracted from the to-be-mined data according to a ground truth bounding box corresponding to the difficult example detection bounding box. In the process of obtaining the difficult example sample, first, the target detection bounding box with the highest overlap degree with the ground truth bounding box of each recognition object is selected from each candidate detection bounding box of the recognition object. Then, the target detection bounding box with the highest overlap degree is difficult example detected according to at least one of the overlap degree, the confidence and the object category corresponding to each target detection bounding box. If there is still a difficult example bounding box that meets the difficult example condition, it means that there is still a bounding box that does not meet the detection accuracy requirement of the overlap degree, the confidence and the object category in the target detection bounding box with the highest overlap degree with the ground truth bounding box. Therefore, the bounding box that does not meet the detection accuracy requirement is taken as the difficult example detection bounding box. At this time, the difficult example sample can be extracted from the ground truth bounding box corresponding to the difficult example detection bounding box. In this way, the operation of effectively screening the difficult example sample from the to-be-mined data is realized, and the efficiency and accuracy of obtaining the difficult example sample are ensured.

[0178] In one embodiment, as shown in FIG. 1, a generation device of a difficult example sample is provided, and the generation device of the difficult example sample comprises a first determination module 20. Figure 9 The first determination module 20 comprises a first determination unit 21 and a second determination unit 22.

[0179] The first determination unit 21 is configured to determine, for each recognition object, an overlap degree between each candidate detection bounding box of the recognition object and a ground truth bounding box.

[0180] The second determination unit 22 is configured to take the candidate detection bounding box with the largest overlap degree as a target detection bounding box of the recognition object.

[0181] In one embodiment, as shown in FIG. 2, a generation device of a difficult example sample is provided, and the generation device of the difficult example sample comprises a second determination module 30. Figure 10 The second determination module 30 comprises a third determination unit 31, a fourth determination unit 32, a fifth determination unit 33 and a sixth determination unit 34.

[0182] The third determination unit 31 is configured to determine a first size relationship between the overlap degree of each target detection bounding box and an overlap degree threshold.

[0183] The fourth determination unit 32 is configured to determine a second size relationship between the detection confidence of each target detection bounding box and a confidence threshold.

[0184] The fifth determination unit 33 is configured to determine a belonging relationship between the predicted category of each target detection bounding box and a ground truth category.

[0185] The sixth determining unit 34 is configured to determine the hard example bounding box from each target detection bounding box according to at least one of the first size relationship, the second size relationship, and the attribution relationship.

[0186] In one embodiment, as shown in FIG. 1, a hard example sample generation apparatus is provided, and the hard example sample generation apparatus comprises a sixth determining unit 34. Figure 11 The sixth determining unit 34 comprises a first determining sub-unit 341, a second determining sub-unit 342, a third determining sub-unit 343, and a fourth determining sub-unit 344.

[0187] The first determining sub-unit 341 is configured to determine, for each target detection bounding box, that the target detection bounding box satisfies the hard example condition if the first size relationship is that the overlap degree of the target detection bounding box is less than the overlap degree threshold.

[0188] The second determining sub-unit 342 is configured to determine that the target detection bounding box satisfies the hard example condition if the second size relationship is that the detection confidence of the target detection bounding box is less than the confidence threshold.

[0189] The third determining sub-unit 343 is configured to determine that the target detection bounding box satisfies the hard example condition if the attribution relationship is that the predicted category of the target detection bounding box is different from the true value category.

[0190] The fourth determining sub-unit 344 is configured to determine that the target detection bounding box is the hard example bounding box if the number of times that the target detection bounding box satisfies the hard example condition is greater than the number threshold.

[0191] In one embodiment, as shown in FIG. 1, a hard example sample generation apparatus is provided, and the hard example sample generation apparatus comprises a sixth determining unit 34. Figure 12 The sixth determining unit 34 comprises a first determining sub-unit 341, a second determining sub-unit 342, a third determining sub-unit 343, and a fourth determining sub-unit 344.

[0192] The seventh determining unit 11 is configured to perform object recognition on the to-be-mined data by using a plurality of neural networks to obtain at least one candidate detection bounding box of each candidate object in the to-be-mined data.

[0193] The eighth determining unit 12 is configured to determine at least one recognized object from the candidate objects according to a hard example category requirement.

[0194] The ninth determining unit 13 is configured to extract at least one candidate detection bounding box of each recognized object from the at least one candidate detection bounding box of each candidate object.

[0195] In one embodiment, as shown in FIG. 1, a hard example sample generation apparatus is provided, and the hard example sample generation apparatus comprises a sixth determining unit 34. Figure 13 The extraction module 40 comprises an extraction unit 41 and a generation unit 42.

[0196] Extraction unit 41 is used to extract the data corresponding to the ground truth bounding box of the hard case detection bounding box in the data to be mined as hard case data.

[0197] The generation unit 42 is used to generate difficult example samples based on the difficult example data, the difficult example location information of the difficult example data in the data to be mined, the ground truth bounding box corresponding to the difficult example detection bounding box, the object category of the identified object, and the number of difficult example data.

[0198] In one embodiment, such as Figure 14 As shown, a difficult example sample generation apparatus is provided. The extraction unit 41 of the difficult example sample generation apparatus includes: a fifth determining subunit 411 and a sixth determining subunit 412, wherein:

[0199] The fifth determining subunit 411 is used to determine the border position information of the true bounding box corresponding to the difficult case detection bounding box in the data to be mined, and to determine the candidate position information of each candidate data in the data to be mined.

[0200] The sixth determining subunit 412 is used to determine the hard case data in the data to be mined based on the border position information and the candidate position information, the ground truth bounding box corresponding to the hard case detection bounding box.

[0201] Each module in the aforementioned difficult example sample generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0202] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 15As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for wired or wireless communication with external terminals, and wireless communication can be realized through WIFI, mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by the processor to realize a generation method of a difficult example sample. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0203] Those skilled in the art can understand that, Figure 15 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0204] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the following steps:

[0205] Obtaining at least one object detection result of each identified object in the to-be-mined data; wherein the object detection result comprises: a candidate detection bounding box of the identified object, a confidence of the candidate detection bounding box and an object category;

[0206] According to the overlap degree of each candidate detection bounding box of each identified object and the ground truth bounding box of each identified object, determining a target detection bounding box of each identified object;

[0207] According to at least one of the overlap degree, the confidence and the object category corresponding to each target detection bounding box, determining a difficult example bounding box from each target detection bounding box;

[0208] According to the ground truth bounding box corresponding to the difficult example bounding box, a difficult example sample is extracted from the to-be-mined data.

[0209] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0210] For each identified object, a degree of overlap between each candidate detection bounding box of the identified object and the ground truth bounding box is determined.

[0211] The candidate detection bounding box with the largest degree of overlap is taken as a target detection bounding box of the identified object.

[0212] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0213] A first size relationship between the degree of overlap of each target detection bounding box and the degree of overlap threshold is determined.

[0214] A second size relationship between the detection confidence of each target detection bounding box and the confidence threshold is determined.

[0215] An attribution relationship between the predicted category of each target detection bounding box and the ground truth category is determined.

[0216] According to at least one of the first size relationship, the second size relationship, and the attribution relationship, a difficult example bounding box is determined from the target detection bounding boxes.

[0217] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0218] For each target detection bounding box, if the first size relationship is that the degree of overlap of the target detection bounding box is less than the degree of overlap threshold, it is determined that the target detection bounding box satisfies a difficult example condition.

[0219] If the second size relationship is that the detection confidence of the target detection bounding box is less than the confidence threshold, it is determined that the target detection bounding box satisfies the difficult example condition.

[0220] If the attribution relationship is that the predicted category of the target detection bounding box is different from the ground truth category, it is determined that the target detection bounding box satisfies the difficult example condition.

[0221] If the number of times that the target detection bounding box satisfies the difficult example condition is greater than a number threshold, it is determined that the target detection bounding box is a difficult example bounding box.

[0222] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0223] Object recognition is performed on the to-be-mined data by a plurality of neural networks, to obtain at least one candidate detection bounding box of each candidate object in the to-be-mined data.

[0224] determine at least one identified object from the candidate objects according to the difficult example category requirement;

[0225] extract at least one candidate detection bounding box of each identified object from at least one candidate detection bounding box of each candidate object.

[0226] In one embodiment, the processor, when executing the computer program, further implements the following steps:

[0227] extract data corresponding to the ground truth bounding box of the difficult example detection bounding box in the data to be mined as difficult example data;

[0228] generate difficult example samples according to the difficult example data, difficult example location information of the difficult example data in the data to be mined, the ground truth bounding box corresponding to the difficult example detection bounding box, the object category of the identified object, and the number of difficult example data.

[0229] In one embodiment, the processor, when executing the computer program, further implements the following steps:

[0230] determine the bounding box location information of the ground truth bounding box corresponding to the difficult example detection bounding box in the data to be mined, and determine the candidate location information of each candidate data in the data to be mined;

[0231] determine the difficult example data of the ground truth bounding box corresponding to the difficult example detection bounding box in the data to be mined according to the bounding box location information and the candidate location information.

[0232] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the following steps:

[0233] obtain at least one object detection result of each identified object in the data to be mined; wherein the object detection result includes: a candidate detection bounding box of the identified object, a confidence of the candidate detection bounding box, and an object category;

[0234] determine a target detection bounding box of each identified object according to the overlap degree of each candidate detection bounding box of each identified object and the ground truth bounding box of each identified object;

[0235] determine a difficult example bounding box from each target detection bounding box according to at least one of the overlap degree, the confidence, and the object category corresponding to each target detection bounding box;

[0236] extract difficult example samples from the data to be mined according to the ground truth bounding box corresponding to the difficult example detection bounding box.

[0237] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0238] For each identified object, determine an overlap degree between each candidate detection bounding box of the identified object and the ground truth bounding box;

[0239] determine a target detection bounding box of the identified object as the candidate detection bounding box with the largest overlap degree.

[0240] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0241] determine a first size relationship between the overlap degree of each target detection bounding box and an overlap degree threshold value;

[0242] determine a second size relationship between the detection confidence of each target detection bounding box and a confidence threshold value;

[0243] determine an attribution relationship between the predicted category of each target detection bounding box and the ground truth category;

[0244] determine a hard example bounding box from the target detection bounding boxes according to at least one of the first size relationship, the second size relationship, and the attribution relationship.

[0245] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0246] For each target detection bounding box, if the first size relationship is that the overlap degree of the target detection bounding box is less than the overlap degree threshold value, it is determined that the target detection bounding box satisfies a hard example condition;

[0247] if the second size relationship is that the detection confidence of the target detection bounding box is less than the confidence threshold value, it is determined that the target detection bounding box satisfies the hard example condition;

[0248] if the attribution relationship is that the predicted category of the target detection bounding box is different from the ground truth category, it is determined that the target detection bounding box satisfies the hard example condition;

[0249] if the number of times that the target detection bounding box satisfies the hard example condition is greater than a number threshold value, it is determined that the target detection bounding box is a hard example bounding box.

[0250] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0251] perform object recognition on the to-be-mined data through a plurality of neural networks to obtain at least one candidate detection bounding box of each candidate object in the to-be-mined data;

[0252] determine at least one identified object from the candidate objects according to a hard example category requirement;

[0253] extract at least one candidate detection bounding box of each identified object from the at least one candidate detection bounding box of each candidate object.

[0254] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0255] Extracting data corresponding to the ground truth bounding box of the difficult example detection bounding box in the to-be-mined data as difficult example data;

[0256] According to the difficult example data, the difficult example position information of the difficult example data in the to-be-mined data, the ground truth bounding box corresponding to the difficult example detection bounding box, the object category of the identified object, and the number of difficult example data, a difficult example sample is generated.

[0257] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0258] Determining the bounding box position information of the ground truth bounding box corresponding to the difficult example detection bounding box in the to-be-mined data, and determining the candidate position information of each candidate data in the to-be-mined data;

[0259] According to the bounding box position information and the candidate position information, the difficult example data of the ground truth bounding box corresponding to the difficult example detection bounding box in the to-be-mined data is determined.

[0260] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant national and regional laws, regulations and standards.

[0261] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0262] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0263] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for generating difficult sample examples, characterized in that, The method comprises: obtaining at least one object detection result of each identified object in the data to be mined; wherein the object detection result comprises: each candidate detection bounding box of the identified object, a confidence of the candidate detection bounding box, and an object category; for each identified object, determining an overlap degree between each candidate detection bounding box of the identified object and a ground truth bounding box; taking the candidate detection bounding box with the largest overlap degree as a target detection bounding box of the identified object; determining a difficult example bounding box from the target detection bounding boxes according to at least one of the overlap degree, the confidence, and the object category corresponding to each target detection bounding box; extracting a difficult example sample from the data to be mined according to the ground truth bounding box corresponding to the difficult example bounding box.

2. The method of claim 1, wherein, The confidence of the candidate detection bounding box refers to the prediction accuracy of the object category corresponding to the candidate detection bounding box.

3. The method of claim 1, wherein, The determination of the difficult example bounding box from the target detection bounding boxes according to at least one of the overlap degree, the confidence, and the object category corresponding to each target detection bounding box comprises determining a first size relationship between the overlap degree of each target detection bounding box and an overlap degree threshold value; determining a second size relationship between the detection confidence of each target detection bounding box and a confidence threshold value; determining an attribution relationship between the predicted category of each target detection bounding box and a ground truth category; determining the difficult example bounding box from the target detection bounding boxes according to at least one of the first size relationship, the second size relationship, and the attribution relationship.

4. The method of claim 3, wherein, The determination of the difficult example bounding box from the target detection bounding boxes according to the first size relationship, the second size relationship, and the attribution relationship comprises: for each target detection bounding box, if the first size relationship is that the overlap degree of the target detection bounding box is less than the overlap degree threshold value, it is determined that the target detection bounding box satisfies a difficult example condition; if the second size relationship is that the detection confidence of the target detection bounding box is less than the confidence threshold value, it is determined that the target detection bounding box satisfies the difficult example condition; if the attribution relationship is that the predicted category of the target detection bounding box is different from the ground truth category, it is determined that the target detection bounding box satisfies the difficult example condition; if the number of times that the target detection bounding box satisfies the difficult example condition is greater than a number threshold value, it is determined that the target detection bounding box is the difficult example bounding box.

5. The method of claim 1, wherein, The obtaining of the at least one candidate detection bounding box of each identified object in the data to be mined comprises: performing object recognition on the data to be mined through a plurality of neural networks to obtain at least one candidate detection bounding box of each candidate object in the data to be mined; determining at least one identified object from the candidate objects according to a difficult example category requirement; extracting at least one candidate detection bounding box of each identified object from the at least one candidate detection bounding box of each candidate object.

6. The method according to any one of claims 1 to 5, characterized in that, The generation of the difficult example sample according to the difficult example data belonging to the difficult example bounding box in the data to be mined comprises: extracting data corresponding to the ground truth bounding box corresponding to the difficult example bounding box in the data to be mined as difficult example data; generating the difficult example sample according to the difficult example data, difficult example position information of the difficult example data in the data to be mined, the ground truth bounding box corresponding to the difficult example bounding box, the object category of the identified object, and the number of the difficult example data.

7. The method of claim 6, wherein, The extracting the difficult example data comprises: determining the edge box position information of the ground-truth bounding box corresponding to the difficult example bounding box in the data to be mined, and determining candidate position information of each candidate data in the data to be mined; determining the difficult example data in the data to be mined according to the edge box position information and the candidate position information.

8. A device for generating difficult example samples, characterized in that, The device comprises: an acquisition module, configured to acquire at least one object detection result of each recognition object in the data to be mined; wherein the object detection result comprises a candidate detection bounding box of the recognition object, a confidence of the candidate detection bounding box, and an object category; a first determination module, configured to determine, for each recognition object, an overlap degree between each candidate detection bounding box of the recognition object and a ground-truth bounding box; and determine a target detection bounding box of the recognition object as the candidate detection bounding box with the maximum overlap degree; a second determination module, configured to determine a difficult example bounding box from the target detection bounding boxes according to at least one of the overlap degrees, the confidences, and the object categories corresponding to the target detection bounding boxes; an extraction module, configured to extract a difficult example sample from the data to be mined according to a ground-truth bounding box corresponding to the difficult example bounding box. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

Images