Target classification methods and systems

Abstract

A target classification method and system are disclosed. The method may include: upon receiving a detection result of a target item, classifying the detection result of the target item using two or more pre-set classification network models; the detection result of the target item includes: target item category information, target item location information, and confidence information; fusing the intermediate classification results output by each of the classification network models to obtain a classification result of the target item; the classification result of the target item includes: target item category information and confidence information; wherein the training samples corresponding to any two of the classification network models belong to different categories. Applying the above scheme can improve the accuracy of target detection.

Description

Technical Field

[0001] This invention relates to the field of target detection, and more specifically to a target classification method and system. Background Technology

[0002] The prototype of deep learning uses computer system architecture to simulate the human brain and is widely used in fields such as security inspection. In security inspection, deep learning is primarily used for object detection, that is, detecting the location and classification information of target objects in an input image.

[0003] Existing deep learning-based object detection solutions typically use a single convolutional neural network (CNN) or a fully connected deep neural network (DNN) for object detection.

[0004] However, the accuracy of target detection using the above method is poor. Summary of the Invention

[0005] The problem this invention aims to solve is to improve the accuracy of target detection.

[0006] To address the above problems, embodiments of the present invention provide a target classification method, the method comprising:

[0007] When the detection result of the target item is received, the detection result of the target item is classified using two or more preset classification network models; the detection result of the target item includes: target item category information, target item location information, and confidence information;

[0008] The intermediate classification results output by each of the classification network models are fused to obtain the classification result of the target item; the classification result of the target item includes: target item category information and confidence information;

[0009] Wherein, the training samples corresponding to any two of the classification network models belong to different categories.

[0010] Optionally, the training samples corresponding to any two of the classification network models may be of completely different or partially different categories.

[0011] Optionally, in the training samples corresponding to each of the classification network models, the difference in the number of training samples between any two categories is less than the second sample difference threshold.

[0012] Optionally, the training samples corresponding to the two or more classification network models are determined by the following method:

[0013] From all training samples, select training samples with the same visual features, and use the training samples with the same visual features as the training samples corresponding to the same classification network model.

[0014] From the remaining training samples selected according to visual features, training samples with the same item usage characteristics are selected, and these training samples with the same item usage characteristics are used as training samples corresponding to the same classification network model.

[0015] From the remaining training samples selected based on visual features and item usage characteristics, training samples with the same item material are selected, and these training samples with the same item material are used as training samples corresponding to the same classification network model.

[0016] Optionally, the training samples corresponding to the two or more classification network models are further determined by the following method:

[0017] Training samples from two categories with different visual features but whose difference in the number of training samples is less than a preset third sample difference threshold are used as training samples for the same classification network model; the third sample difference threshold is less than the second sample difference threshold.

[0018] Optionally, fusing the intermediate classification results output by each of the classification network models to obtain the classification result of the target item includes:

[0019] When the target item category information of the same target item exists only in one intermediate classification result, the intermediate classification result shall be used as the classification result of the target item.

[0020] When the target item category information of the same target item exists only in two intermediate classification results, the intermediate classification result with higher confidence among the two intermediate classification results shall be used as the classification result of the target item.

[0021] When the target item category information of the same target item exists in more than three intermediate classification results, a preset number of intermediate classification results are selected as the classification result of the target item based on the confidence information of the three or more intermediate classification results.

[0022] Optionally, the method further includes:

[0023] After receiving the input image, a preset detection network model is used to perform target detection on the input image to obtain the detection result of the target item.

[0024] Optionally, after receiving the input image, the step of using a preset detection network model to perform target detection on the input image to obtain the detection result of the target item includes:

[0025] After receiving the input image, the target object is detected in the input image using two or more preset detection network models.

[0026] The intermediate detection results output by each of the detection network models are fused to obtain the detection result of the target item; the intermediate detection result includes: target item category information, target item location information, and confidence information;

[0027] Wherein, the training samples corresponding to any two of the detection network models belong to different categories.

[0028] Optionally, the training samples corresponding to any two of the detection network models may be of completely different or partially different categories.

[0029] Optionally, the difference in the number of categories of training samples corresponding to any two detection network models is less than a preset first category difference; and the difference in the number of training samples of any two categories in the training samples corresponding to each detection network model is less than a first sample difference threshold.

[0030] Optionally, the two or more detection network models are trained using the following method:

[0031] Count the number of training samples in each category among all training samples, and determine the first category with the most training samples and the second category with the fewest training samples;

[0032] The training sample interval defined by the number of training samples in the first category to the number of training samples in the second category is evenly divided into K parts to obtain K-1 sample number separation values, where K is a positive integer greater than 1.

[0033] Based on K-1 sample count separators and the number of training samples for each category, the obtained training samples are separated into categories to obtain K training samples for the detection network model, and then trained.

[0034] Optionally, fusing the intermediate detection results output by each of the detection network models to obtain the detection result of the target item includes:

[0035] When the target item category information of the same target item exists only in one of the intermediate detection results, the intermediate detection result is used as the detection result of the target item.

[0036] When the target item category information of the same target item exists in two or more intermediate detection results, the detection result of the target item is obtained by combining the target item location information and confidence information in the two or more intermediate detection results.

[0037] Optionally, the step of combining the target item location information and confidence information from the two or more intermediate detection results to obtain the detection result of the target item includes:

[0038] For any intermediate detection result with a confidence level that is not the lowest among the two or more intermediate detection results, calculate the first area respectively, and perform the following operations on each of the first areas: calculate the overlapping area between the first area and each of the second areas, and perform a rejection operation on the corresponding intermediate detection result according to the size of the overlapping area, and take the intermediate detection result after the rejection operation among the two or more intermediate detection results as the detection result of the target item in this round;

[0039] The detection results of the target item in each round, and the intermediate detection result with the lowest confidence among the two or more intermediate detection results, are used together as the detection result of the target item.

[0040] Wherein, the first area is the target item area corresponding to the target item location information of the intermediate detection result, the second area is the target item area corresponding to the target item location information of the remaining intermediate detection results among the two or more intermediate detection results, and the confidence level of the intermediate detection result containing the first area is higher than the confidence level of the intermediate detection result containing the second area.

[0041] Optionally, the step of removing intermediate detection results based on the size of the overlapping area includes:

[0042] Based on the size of the overlapping area, determine the difference between the area of ​​each target item and the overlapping area in the two intermediate detection results.

[0043] When the difference between the area of ​​the target item and the overlapping area is less than the first area threshold, an elimination operation is performed on the intermediate detection results where the area of ​​the target item whose overlapping area difference is less than the first area threshold is located.

[0044] When the difference between the area of ​​the target item and the overlapping area is greater than the first area threshold, a rejection operation is performed based on the difference between the areas of the target items in the two intermediate detection results.

[0045] Optionally, the step of performing a rejection operation based on the difference between the areas of the target items in the two corresponding intermediate detection results includes:

[0046] Calculate the difference between the areas of the target item in the two corresponding intermediate detection results;

[0047] When the difference between the areas of the target items is less than the second area threshold, the intermediate detection results with lower confidence are removed.

[0048] Optionally, the step of performing the removal operation based on the difference between the areas of the target items in the two corresponding intermediate detection results further includes:

[0049] Use the target item location information from the removed intermediate detection results to update the target item location information from the retained intermediate detection results.

[0050] Optionally, the first area of ​​each intermediate detection result with a non-lowest confidence level is calculated in descending order of confidence level, and a removal operation is performed.

[0051] Optionally, the method further includes: using the classification result of the target item to determine whether the detection result of the target item is valid, and using the valid detection result of the target item as the identification result of the target item.

[0052] Optionally, determining whether the detection result of the target item is valid using the classification result of the target item includes:

[0053] Using the preset first training sample category mapping relationship information, the category information of the target item in the detection result of the target item is converted into a category, and based on the category conversion result, it is determined whether the detection result of the target item is valid, and the recognition result of the target item is obtained.

[0054] The first training sample category mapping information is the correspondence between the category of the same training sample when it is used as a training sample for the detection network model and the category of the same training sample when it is used as a training sample for the classification network model.

[0055] Optionally, determining whether the detection result of the target item is valid based on the category conversion result includes:

[0056] Based on the category conversion result, determine whether the target item category information in the detection result of the target item matches the target item category information in the classification result of the target item, and obtain the first judgment result;

[0057] Based on the first judgment result, and combined with the detection result of the target item and the confidence information in the classification result of the target item, it is determined whether the detection result of the target item is valid.

[0058] Optionally, determining whether the detection result of the target item is valid based on the first judgment result, combined with the detection result of the target item and the confidence information in the classification result of the target item, includes:

[0059] When the first judgment result is a match between the two, if the confidence level in the detection result of the target item is higher than the first confidence level threshold, or the confidence level in the classification result of the target item is higher than the second confidence level threshold, then the detection result of the target item is valid; otherwise, it is invalid.

[0060] When the first judgment result is that the two do not match, if the confidence level of the detection result of the target item is higher than the third confidence level threshold, or the confidence level of the classification result of the target item is higher than the fourth confidence level threshold, then the detection result of the target item is valid; otherwise, it is invalid.

[0061] Wherein, the third confidence threshold is greater than the first confidence threshold, and the fourth confidence threshold is greater than the second confidence threshold.

[0062] Optionally, the method further includes:

[0063] When the detection result of the target item is invalid, the target item category information in the detection result is converted using the preset second training sample category mapping relationship information, and the similar item identification result of the target item is obtained based on the category conversion result.

[0064] The second training sample category mapping information refers to the correspondence between the training sample category in which the same training sample is located when it is used as a training sample corresponding to the detection network model and the similar training sample category in which it is located when it is used as a training sample corresponding to the classification network model. The similar training sample category is a training sample category with the same visual features, or one or more training sample categories at the same level in the training sample category table.

[0065] This invention also provides a target classification system, the system comprising:

[0066] Two or more target classification units are provided, wherein the target classification units are adapted to classify the detection results of the target items when receiving the detection results of the target items; the detection results of the target items include: target item category information, target item location information, and confidence information;

[0067] The second fusion unit is adapted to fuse the intermediate classification results output by each of the classification network models to obtain the classification result of the target item; the classification result of the target item includes: target item category information and confidence information;

[0068] Wherein, the training samples corresponding to any two of the classification network models belong to different categories.

[0069] Optionally, the system further includes a detection unit, adapted to perform target detection on the input image using a preset detection network model after receiving the input image, and obtain the detection result of the target item.

[0070] Optionally, the detection unit includes:

[0071] Two or more target detection subunits, wherein the detection subunits are adapted to detect target objects in the input image by using a preset detection network model after receiving the input image;

[0072] The second fusion subunit is adapted to fuse the intermediate detection results output by each of the detection network models to obtain the detection result of the target item; the intermediate detection result includes: target item category information, target item location information, and confidence information;

[0073] Wherein, the training samples corresponding to any two of the classification network models belong to different categories.

[0074] Optionally, the system further includes:

[0075] The judgment unit is adapted to use the classification result of the target item to determine whether the detection result of the target item is valid, and to take the valid detection result of the target item as the identification result of the target item.

[0076] Compared with the prior art, the technical solution of the embodiments of the present invention has the following advantages:

[0077] By applying the solution of this invention, after obtaining the detection result of the target item, a classification network model is then used to classify the detection result of the target item. Compared with directly using the target detection result as the final result without using a classification network model, this can effectively reduce false detections. Furthermore, since the training samples corresponding to any two classification network models have different categories, different classification network models can be obtained by training on different training sample categories, which can further reduce the false detection rate. Attached Figure Description

[0078] Figure 1 A flowchart of a target classification method in an embodiment of the present invention;

[0079] Figure 2 This is a schematic diagram of the detection results of multiple target items in an embodiment of the present invention;

[0080] Figure 3 This is a flowchart of another target classification method in an embodiment of the present invention;

[0081] Figure 4 This is a flowchart of a method for obtaining detection results of a target item according to an embodiment of the present invention;

[0082] Figure 5 This is a schematic diagram of the detection results of the target item in an embodiment of the present invention;

[0083] Figure 6 This is a schematic diagram illustrating the calculation of the overlapping area between the frame to be inspected and other frames in an embodiment of the present invention;

[0084] Figure 7 This is a flowchart of another target classification method in an embodiment of the present invention;

[0085] Figure 8 This is a schematic diagram of the structure of a target classification system in an embodiment of the present invention. Detailed Implementation

[0086] Existing deep learning-based object detection solutions typically use a single CNN or DNN for object detection, resulting in poor accuracy.

[0087] To address the aforementioned problems, embodiments of the present invention provide a target classification method. By applying the scheme of this embodiment, after obtaining the detection results of the target item, a classification network model is then used to classify the detection results of the target item. Compared to directly using the target detection results as the final result without using a classification network model, this method can effectively reduce false detections. Furthermore, since the training samples corresponding to any two classification network models belong to different categories, different classification network models can be obtained by training on different training sample categories, further reducing the false detection rate.

[0088] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0089] Reference Figure 1 This invention provides a target classification method, which may include the following steps:

[0090] Step 11: When the detection result of the target item is received, the detection result of the target item is classified using two or more preset classification network models.

[0091] The detection results of the target item include: target item category information, target item location information, and confidence level information. The training samples corresponding to any two of the detection network models belong to different categories.

[0092] In practical applications, new images are constantly emerging, meaning that various unknown objects and stacked object patterns may appear. Therefore, the detection results of target objects obtained through detection network models may still contain false positives. Figure 2As shown, metal bottle bottoms and metal strips may be mistaken for knives.

[0093] To reduce false positives and improve detection accuracy, embodiments of this invention utilize a pre-defined classification network to further determine the category of the target item. For a given target item, all its positive sample images reflect the item's correlation, which reveals the target item's characteristics. Therefore, the classification network model can learn the correlation features between these characteristic items and determine the detection result of the target item. The classification network model does not need to focus on the target item's location; it only needs to learn classification features to accurately determine the target item's category.

[0094] In one embodiment of the present invention, in order to further reduce false detections, the number of classification network models can be two or more.

[0095] In practice, the number of classification network models can be two, three, or more; there is no specific limitation, as long as the categories of the training samples corresponding to any two classification network models are different. The training samples corresponding to the two classification network models are the training samples used to train the classification network models.

[0096] Each classification network model can be obtained by training the same network model with training samples from different categories, or by training different network models with training samples from different categories. That is, the model structures corresponding to each detection network model can be the same or different, without any specific restrictions. For example, each classification network model can be obtained by training a convolutional neural network model with training samples from different categories.

[0097] Understandably, different detection network models can use the same or different detection methods to detect the input image, depending on their model structure. If the model structures are the same, the detection methods will be the same; conversely, if the model structures are different, the detection methods will differ.

[0098] In practice, the categories of training samples corresponding to any two classification network models can be completely different or partially different, which makes the training samples corresponding to each detection network model not completely the same.

[0099] Taking a classification network model with L components as an example, the labels of each classification network model are H1, H2, ..., HL (L≥2). The total number of categories in the training samples corresponding to the L classification network models is C2, which is usually different from the total number of categories in the training samples corresponding to the detection network model, and can include some categories that are prone to false positives. There can be overlap between the categories in the training samples corresponding to the classification network models. In other words, for any category n, n∈C2, category n can appear simultaneously in the training samples corresponding to classification network models H1, H2, ..., and HL.

[0100] In practice, the training of the classification network model and the detection network model are independent of each other, and each has its own training sample set. The training sample set of the detection network model consists of a large number of images, while the training sample set of the classification network can consist of a large number of detection boxes.

[0101] To improve the accuracy of category determination by the classification model, the difference in the number of training samples for any two categories within the training samples corresponding to each classification network model should be less than a second sample difference threshold. This second sample difference threshold can be set to ensure that the difference in the number of training samples for any two categories within the same classification network model is small.

[0102] For example, in the training sample set of a classification network model, the category with the fewest training samples is category 1, and the number of training samples for category 1 is Mcmin. In this case, if category 1 is used as the training network for classification network model H1, then the number of training samples for other categories corresponding to classification network model H1 is at most βMCmin.

[0103] If the number of training samples for other categories in the training sample set exceeds βMCmin, then βMCmin training samples are used to train the classification network model H1. If the number of training samples for other categories in the training sample set does not exceed βMCmin, then all training samples for those other categories can be used to train the classification network model H1.

[0104] In one embodiment of the present invention, the training samples corresponding to the two or more classification network models can be determined by the following method:

[0105] First, from all training samples (i.e., the training set corresponding to the classification network model), training samples with the same visual features are selected, and training samples with the same visual features are used as training samples corresponding to the same classification network model.

[0106] In practical implementation, models should share similar visual characteristics, i.e., similar shapes. For example, the shapes of a metal bottle bottom, a metal strip, and a fruit knife are similar; in this case, the metal bottle bottom, the metal strip, and the fruit knife can be used as training samples for training the classification network model A1. Similarly, the shapes of a metal bottle and a metal gas canister are similar; in this case, the metal bottle and the metal gas canister can be used as training samples for training the classification network model A2.

[0107] Next, from the remaining training samples selected according to visual features, training samples with the same item usage characteristics are selected, and these training samples with the same item usage characteristics are used as training samples corresponding to the same classification network model.

[0108] In practice, items with the same usage characteristics, such as wrenches, pliers, and saws, can be used as training samples for the same classification network model. In this case, the difference between the number of training samples with the same usage characteristics and the number of training samples for other categories already in the classification network model should be less than the second sample difference threshold.

[0109] Finally, from the remaining training samples selected according to visual features and item usage characteristics, training samples with the same item material are selected, and these training samples with the same item material are used as training samples corresponding to the same classification network model.

[0110] In practice, items of the same material, such as mobile phones, laptops, and monitors, can be used as training samples for the same classification network model. In this case, the difference between the number of samples with the same material and the number of training samples for other categories in the classification network model should be less than the second sample difference threshold.

[0111] In one embodiment of the present invention, training samples of two categories with different visual features but a difference in the number of training samples less than a preset third sample difference threshold can be used as training samples corresponding to the same classification network model; the third sample difference threshold is less than the second sample difference threshold.

[0112] In other words, training samples from two categories with very different visual features but similar numbers of training samples can be used as training samples for the same classification network model.

[0113] Step 12: Fuse the intermediate classification results output by each of the classification network models to obtain the classification result of the target item.

[0114] In practice, the detection results of the target item can be output in the form of detection boxes, as shown in Table 1: Detection results Boxn for the target item:

[0115] Table 1

[0116] Boxn Network Qn Category C1_n Confidence level PQ Xmin Ymin Xmax Ymax

[0117] After passing through the classification network model Hi, the intermediate classification results output by the classification network model Hi may be as shown in Table 2:

[0118] Table 2

[0119]

[0120] As can be seen from Table 2, the classification network model Hi determines that the target item in the detection result Boxn belongs to category C2_1, category C2_2 and category C2_i.

[0121] After passing through the classification network model Hj, the intermediate classification results output by the classification network model Hj may be as shown in Table 3:

[0122] Table 3

[0123]

[0124] As can be seen from Table 3, the target items in Boxn, determined by the classification network model Hj, belong to categories C2_1, C2_2, and C2_j.

[0125] Different classification network models output different intermediate classification results.

[0126] In practice, various methods can be used to fuse the intermediate classification results output by each of the classification network models to obtain the classification result of the target item.

[0127] In one embodiment of the present invention, when the target item category information of the same target item exists only in one intermediate classification result, the intermediate classification result is used as the classification result of the target item.

[0128] When the target item category information of the same target item exists only in two intermediate classification results, the intermediate classification result with higher confidence among the two intermediate classification results shall be used as the classification result of the target item.

[0129] When the target item category information of the same target item exists in more than three intermediate classification results, a preset number of intermediate classification results are selected as the classification result of the target item based on the confidence information of the three or more intermediate classification results.

[0130] For example, assuming the classification result of the target item is OH, if category C2_i only exists in the intermediate classification results of the classification network model Hi, then category C2_i and its corresponding confidence PH3 will be retained in the classification result OH of the target item.

[0131] If category C2_1 exists only in classification network models Hi and Hj, and PH1 < PH4, then the confidence level PH4 corresponding to C2_1 will be retained in the classification result OH of the target item.

[0132] If category C2_2 exists in more than three intermediate classification results, and if the confidence level corresponding to category C2_2 in multiple intermediate classification results is extremely high, for example, if the confidence level corresponding to category C2_2 is greater than GP1, then a small number (e.g., three) of the intermediate classification results with the highest confidence level can be selected and retained in the classification result OH of the target item.

[0133] If the confidence scores for all intermediate classification results for category C2_2 are low (e.g., all confidence scores for category C2_2 are less than GP2), then a small number (e.g., one) of the intermediate classification results with the highest confidence scores can be selected and retained in the target item's classification result OH. Here, GP2 is less than GP1.

[0134] If the highest confidence level for category C2_2 is less than GP1 and the lowest confidence level is greater than GP2 among all intermediate classification results, then the highest confidence levels (e.g., five) of the intermediate classification results can be selected and retained in the classification result OH of the target item.

[0135] As can be seen from the above, the target classification method in this embodiment of the invention can further improve the accuracy of target detection by using a classification network to further determine the category of the target item after obtaining the detection result of the target item.

[0136] Figure 3 Another target classification method provided by an embodiment of the present invention may include the following steps:

[0137] Step 31: After receiving the input image, the target detection is performed on the input image using a preset detection network model to obtain the detection result of the target item.

[0138] Step 32: When the detection result of the target item is received, the detection result of the target item is classified using two or more preset classification network models.

[0139] The detection results of the target item include: target item category information, target item location information, and confidence level information.

[0140] Step 33: Fuse the intermediate classification results output by each of the classification network models to obtain the classification result of the target item.

[0141] The classification result of the target item includes: target item category information and confidence information. The training samples corresponding to any two of the classification network models belong to different categories.

[0142] For steps 32 and 33, please refer to the description of steps 11 and 12 above for implementation. They will not be repeated here.

[0143] Regarding step 31, in practice, a detection network model can be used to perform target detection on the input image to obtain the detection results of the target item.

[0144] In one embodiment of the present invention, in order to further improve the accuracy of target detection, reference is made to... Figure 4 The following methods can be used to obtain the detection results of the target item:

[0145] Step 41: After receiving the input image, the target object can be detected in the input image by using two or more preset detection network models.

[0146] The intermediate detection results include: target item category information, target item location information, and confidence level information. The training samples corresponding to any two of the detection network models must belong to different categories.

[0147] In practice, the number of the two or more detection network models is not limited; for example, it can be two, three, or more, as long as the categories of the training samples corresponding to any two detection network models are different. The training samples corresponding to the detection network models are the training samples used to train the network models to obtain the detection network models.

[0148] Each detection network model can be obtained by training the same network model using training samples of different categories. That is, the model structures corresponding to each classification network model can be the same or different, or they can be obtained by training different network models using training samples of different categories. There is no specific limitation. For example, each detection network model can be obtained by training a convolutional neural network model using training samples of different categories.

[0149] Understandably, different classification network models can use the same classification method to categorize the detection results of target items, or they can use different classification methods, depending on the model structure of each network model. If the model structures of different classification network models are the same, the way they categorize the detection results of target items will also be the same; conversely, if the model structures of different classification network models are different, the way they categorize the detection results of target items will also be different.

[0150] In practice, the categories of training samples corresponding to any two of the aforementioned detection network models can be completely different or partially different. The training methods for each classification network model can be the same or different.

[0151] Taking two detection network models as an example, namely detection network model Q1 and detection network model Q2, the total number of categories in the training samples of the two detection network models is N, where N is an integer ≥ 2, and the set of N categories is C1. Specifically, the set of all categories in the training samples of detection network model Q1 is CQ1, and the set of all categories in the training samples of detection network model Q2 is CQ2, where CQ1 ∈ C1 and CQ2 ∈ C1. Sets CQ1 and CQ2 can contain the same category; that is, the elements of sets CQ1 and CQ2 can be partially the same. In other words, some categories can appear in both sets CQ1 and CQ2. However, at least one element in sets CQ1 and CQ2 must be different.

[0152] In practical applications, the distribution of actual training data is often uneven. Strictly speaking, any dataset is imbalanced, which causes detection network models to tend to learn features from a wider range of training samples. In the field of object detection, negative samples (i.e., images that do not contain the target object, such as images that do not contain contraband) are often far more numerous than positive samples (images that do contain the target object, such as images that contain contraband), resulting in a severe imbalance between positive and negative samples. This, in turn, leads to increased training costs and network model degradation.

[0153] For example, in subway security checks, prohibited items such as knives and guns are detected far less frequently than regular items like containers and umbrellas, resulting in unsatisfactory detection effectiveness. Taking the Shanghai Metro as an example, the average daily passenger flow exceeds 11 million. Shanghai Metro has 415 stations, equipped with over 6,000 security personnel and X-ray security scanners, conducting an average of 3.2 million package checks daily. A total of over 35,000 dangerous items are detected annually, averaging only 1.09 prohibited items per million packages. Moreover, prohibited items include many types, such as controlled knives and firearms, demonstrating a highly unbalanced distribution of positive and negative samples. Similarly, in satellite remote sensing, single images often have very high resolutions, such as 30,000*40,000, but the actual effective foreground is very limited, indicating an imbalance between foreground and background.

[0154] In embodiments of the present invention, in order to make the training samples more balanced, the difference in the number of categories of training samples corresponding to any two detection network models is less than a preset first category difference; and in the training samples corresponding to each detection network model, the difference in the number of training samples of any two categories is less than a first sample difference threshold.

[0155] By adjusting the first class difference, the training samples used for each detection network model can be more balanced across classes. By adjusting the first sample difference threshold, the number of training samples for each class can be more balanced. This avoids the impact of sample imbalance on the accuracy of target detection and improves the target detection effect.

[0156] For imbalanced training samples, in one embodiment of the present invention, the following method can be used for training to obtain the two or more detection network models:

[0157] First, we can count the number of training samples in each category across all training samples (i.e., the training sample set) and determine the first category with the most training samples and the second category with the fewest training samples.

[0158] Next, the training sample interval defined by the number of training samples in the first category to the number of training samples in the second category is evenly divided into K parts, resulting in K-1 sample quantity separation values, where K is a positive integer greater than 1.

[0159] Finally, based on the K-1 sample count separation values ​​and the number of training samples in each category, the obtained training samples are separated into categories to obtain K training samples for the detection network model, and then the model is trained.

[0160] Assume the training sample set includes N categories, with M1, M2, ..., MN training samples for each category. The first category, having the most training samples out of the N categories, has Mmax training samples. The second category, having the fewest training samples out of the N categories, has Mmin training samples.

[0161] To distinguish [Mmin, Mmax], divide it evenly into K parts, resulting in K sub-intervals. The value range of each sub-interval is as follows: [Mmin, Mmin+△], (Mmin+△, Mmin+2*△], ..., (Mmin+(K-1)*△, Mmax]. Where △=(Mmax-Mmin) / K.

[0162] Training samples from all categories whose training sample counts fall within the same sub-interval are used as training samples for a detection network model. For example, when K*2, training samples from all categories whose training sample counts fall within [Mmin, (Mmin+Mmax) / 2] are used as training samples for detection network model Q1. Training samples from all categories whose training sample counts fall within ((Mmin+Mmax) / 2, Mmax] are used as training samples for detection network model Q2.

[0163] In specific implementation, category adjustment can also be made in combination with the specific features of each category. For example, for categories that are prone to confusion and similarity, such as scissors and knives, they should be grouped into the same network as much as possible to facilitate the network to learn the differences between the two items. Suppose the numbers of two categories with the same visual features are M1 and M2 respectively, and the detection network models are Q1 and Q2 respectively. At this time, there may be two situations:

[0164] 1) The difference between M1 and M2 is too large, such as M1 / M2 > β, where β is a constant greater than 1 (such as set to 3). At this time, M1 can be updated to βM2 (for example, select βM2 samples from the M1 samples using a random function), and M2 remains unchanged. The M1 training samples and the M2 training samples are used as the training samples for training to obtain the same detection network model (such as the detection network model Q1).

[0165] Meanwhile, in order to avoid a significant impact on the performance of detecting the category to which the M1 training samples belong, according to the average value Mavg' of all the training samples of the detection network model Q2, and then extract Mavg' samples from the M1 training samples as the training samples for training to obtain the detection network model Q2.

[0166] If M1 / M2 < 1 / β, it is also implemented in a similar way as above, which will not be elaborated here.

[0167] 2) The difference between M1 and M2 is not obvious, such as 1 / β < M1 / M2 < β, then the M1 training samples and the M2 training samples are used as the training samples for training to obtain the same detection network model.

[0168] In specific implementation, the detection network model can output the intermediate detection result in the form of a detection box. Each detection box has the target item category information, the target item position information and the confidence information, such as: the intermediate detection result output by the detection network model Qn: {network Qn, category n, confidence, Xnmin, Ynmin, Xnmax, Ynmax}, the category of the target item is n, (Xnmin, Ynmin) is the lower left vertex coordinate of the detection box output by the detection network model Qn, and (Xnmax, Ynmax) is the upper right vertex coordinate of the detection box output by the detection network model Qn, as Figure 5 shown in the rectangular box.

[0169] For the same input image, each detection network model will output an intermediate detection result. Taking the number of detection network models as M as an example, the identifiers of each detection network model are Q1, Q2,..., QM respectively. All the results output by the M detection network models are shown in Table 4:

[0170] Table 4

[0171]

[0172] In Table 3, Boxn is the identifier of the intermediate detection result, n∈{1,N}, category C1_n is the category identifier of the target item in the intermediate detection result Boxn, and PQn is the confidence value in the intermediate detection result Boxn.

[0173] Step 42: Fuse the intermediate detection results output by each of the detection network models to obtain the detection result of the target item.

[0174] The intermediate detection results include: target item category information, target item location information, and confidence level information.

[0175] In practice, various methods can be used to fuse the intermediate detection results output by each detection network model to obtain the detection result of the target item, and no specific method is limited.

[0176] In one embodiment of the present invention, when the target item category information of the same target item exists only in one of the intermediate detection results, the intermediate detection result can be used as the detection result of the target item.

[0177] For example, if the target item category C1_n in the intermediate detection result Boxn only exists as the intermediate detection result output by the detection network model Qn, then the intermediate detection result Boxn can be directly used as the detection result of the target item.

[0178] When the target item category information of the same target item exists in two or more intermediate detection results, the detection result of the target item is obtained by combining the target item location information and confidence information in the two or more intermediate detection results.

[0179] In specific implementation, when the target item category information of the same target item exists in two or more intermediate detection results, for any intermediate detection result with a non-lowest confidence level among the two or more intermediate detection results, a first area is calculated, and the following operations are performed on each of the first areas: the overlapping area between the first area and each second area is calculated, and the corresponding intermediate detection result is eliminated according to the size of the overlapping area. The intermediate detection result after the elimination operation among the two or more intermediate detection results is taken as the detection result of the target item in this round; the detection results of the target item in each round and the intermediate detection result with the lowest confidence level among the two or more intermediate detection results are jointly taken as the detection result of the target item.

[0180] Wherein, the first area is the target item area corresponding to the target item location information of the intermediate detection result, the second area is the target item area corresponding to the target item location information of the remaining intermediate detection results among the two or more intermediate detection results, and the confidence level of the intermediate detection result containing the first area is higher than the confidence level of the intermediate detection result containing the second area.

[0181] In one embodiment of the present invention, the step of removing intermediate detection results based on the size of the overlapping area may include:

[0182] Based on the size of the overlapping area, determine the difference between the area of ​​each target item and the overlapping area in the two intermediate detection results.

[0183] When the difference between the area of ​​the target item and the overlapping area is less than the first area threshold, an elimination operation is performed on the intermediate detection results where the area of ​​the target item whose overlapping area difference is less than the first area threshold is located.

[0184] When the difference between the area of ​​the target item and the overlapping area is greater than the first area threshold, a rejection operation is performed based on the difference between the areas of the target items in the two intermediate detection results.

[0185] For example, when the category C1_n of the target item in the intermediate detection result Boxn exists in the intermediate detection results output by M detection network models, the intermediate detection results for category C1_n in each detection network model can be extracted first.

[0186] The intermediate detection results for category C1_n are stored in the temporary storage space TOn. For example, the temporary storage space for category C1_n is TO1, and the temporary storage space for category C1_2 is TO2, ultimately resulting in the storage spaces {TO1, ..., TOn, ..., TON} for all categories. The storage space TOn for each category is then processed separately. At this point, the content stored in TOn (assuming a total of n' detection boxes) is as shown in Table 5:

[0187] Table 5

[0188]

[0189] Where Boxn' represents the intermediate detection result of the detection network model Qn for category C1_n, n∈{1,N}.

[0190] Taking the fusion of intermediate detection results for category C1_n as an example, the specific fusion process is as follows:

[0191] a) Sort all detection boxes in descending order of confidence level;

[0192] b) Take the detection box with the highest confidence as the box to be detected. Calculate the overlapping area S between the box to be detected and another box (whose confidence is lower than that of the box to be detected) in sequence, and obtain the detection result of the target item in this round.

[0193] Take the operation performed on the box to be detected and one of the other boxes as an example. As Figure 6 shown, the area of the box to be detected is the first area S1, and the area of one of the other boxes is the second area S2. At this time, the overlapping coefficient α of the two boxes can be calculated.

[0194] When S1 > γS2, where γ is a constant greater than 1, at this time S1 is much larger than S2, and the overlapping coefficient α = S / S2. If α exceeds the first preset overlapping threshold G1, it indicates that the overlapping area S is close to the second area S2, and the other box can be considered to be included in the box to be detected. Therefore, the other box can be removed from TOn.

[0195] When S1 < S2 / γ, at this time S1 is much smaller than S2, and α = S / S1. If α exceeds the second preset overlapping threshold G2, it indicates that the overlapping area S is close to the first area S1, and the box to be detected can be considered to be included in the other box. Therefore, the box to be detected can be removed from TOn.

[0196] When S2 / γ ≤ S1 ≤ γS2, at this time the first area S1 is comparable to the second area S2, and α = S / (S1 + S2 - S). If α exceeds the third preset overlapping threshold G3, it indicates that the first area S1 is close to the second area S2, and the detection effects of the box to be detected and the other box are comparable. At this time, only one detection box needs to be retained. For example, the box to be detected with higher confidence can be selected, and the other box can be removed from TOn.

[0197] Among them, γ, the first preset overlapping threshold G1, the second preset overlapping threshold G2, and the third preset overlapping threshold G3 are all preset constants, and the specific values corresponding to different categories may vary.

[0198] In an embodiment of the present invention, when the detection effects of the box to be detected and the other box are comparable, if the area of the target item in the removed intermediate detection result is larger than the area of the target item in the retained intermediate detection result, the target item position information of the removed intermediate detection result can be used to update the target item position information of the retained intermediate detection result.

[0199] For example, when the other box is removed, if S2 > S1, in order to minimize the influence of eliminating the box to be detected, the confidence of the box to be detected can be kept unchanged, and the target item position information in the other box can be used to update the target item position information in the box to be detected, so that the area of the box to be detected is larger and the detection output covers more. If S2 ≤ S1, it indicates that the area and confidence of the box to be detected are both the largest, and no update is required.

[0200] After performing the above operation on the target box and another box, the detection result of the target item in this round is obtained. Using the detection result of the target item in this round, the storage space Ton is updated to obtain the updated storage space TOn', which is used as the output of category C1_n.

[0201] c) After one round of traversal, start traversing from the second highest confidence box until the lowest confidence box, which requires a total of n'-1 rounds, and then obtain the fused output of all categories {TO1', ..., TOn', ..., TON'}, as the detection result of the target item.

[0202] In practical implementation, based on the target item's location information (the (Xmin, Ymin) and (Xmax, Ymax) parameters of the detection box) in the detection results, a complete image can be extracted, such as... Figure 2 As shown, images 21 and 22 are extracted images of the bottle bottom, image 23 is an extracted image of the metal strip, and images 24 and 25 are extracted images of the fruit knife. The actual detection result for the target object may be only one image of the target object, or it may be multiple images of the target object, for example, ... Figure 2 As shown, when the target item is the bottom of the bottle, the detection result of the target item may include both Image 21 and Image 22.

[0203] As can be seen from the above, the target classification method in this embodiment of the invention can make the training samples of the detection network model more balanced, and by using two or more detection network models to perform target detection simultaneously, the intermediate results output by each detection network model are finally fused to obtain the detection result of the target item, which can make the target detection result more accurate.

[0204] Reference Figure 7 The present invention also provides another target classification method, which may include the following steps:

[0205] Step 71: After receiving the input image, the input image is subjected to target detection using a preset detection network model to obtain the detection result of the target item.

[0206] For step 71, please refer to the description of step 31 above for implementation, and it will not be repeated here.

[0207] Step 72: When the detection result of the target item is received, the detection result of the target item is classified using two or more preset classification network models.

[0208] The detection results of the target item include: target item category information, target item location information, and confidence level information. The training samples corresponding to any two of the detection network models belong to different categories.

[0209] Step 73: Fuse the intermediate classification results output by each of the classification network models to obtain the classification result of the target item.

[0210] The classification results of the target items include: target item category information and confidence level information.

[0211] For steps 72 and 73, please refer to the descriptions of steps 11 and 12 for implementation details, which will not be repeated here.

[0212] Step 74: Using the classification result of the target item, determine whether the detection result of the target item is valid, and take the valid detection result of the target item as the identification result of the target item.

[0213] In practice, the classification result of the target item can be directly used as the identification result of the target item, or the classification result of the target item can be used to further determine whether the detection result of the target item is valid, and the valid detection result of the target item can be used as the identification result of the target item, thereby further improving the accuracy of target detection.

[0214] Since the detection results of the target item include not only the category information of the target item, but also the location information of the target item, the classification results of the target item are used to remove invalid target item detection results from the numerous target item detection results, and the remaining target item detection results are used as the target item identification results.

[0215] In practice, various methods can be used to determine whether the detection results of the target item are valid, based on the classification results of the target item.

[0216] In one embodiment of the present invention, the target item category information in the detection result of the target item can be converted by using the preset first training sample category mapping relationship information, and the detection result of the target item can be determined based on the category conversion result to obtain the identification result of the target item.

[0217] The first training sample category mapping information is the correspondence between the training sample category when the same training sample is used as a training sample corresponding to the detection network model and the training sample category when it is used as a training sample corresponding to the classification network model.

[0218] In practice, a training sample can be used as a training sample for both a detection network model and a classification network model. When the training sample is used to train the detection network model, it belongs to category C1_i. When the training sample is used to train the classification network model, it belongs to category C2_j. The correspondence between categories C1_i and C2_j is the category mapping information of the training sample. This first training sample category mapping information may include category mapping information for multiple training samples.

[0219] In practice, the first training sample category mapping information can be obtained in advance, and the specific method of obtaining it is not limited.

[0220] In one embodiment of the present invention, based on the category conversion result, the following method can be used to determine whether the detection result of the target item is valid:

[0221] First, based on the category conversion result, it is determined whether the target item category information in the detection result of the target item matches the target item category information in the classification result of the target item, and a first judgment result is obtained. Then, based on the first judgment result, combined with the confidence information in the detection result of the target item and the classification result of the target item, it is determined whether the detection result of the target item is valid.

[0222] When the first judgment result is a match between the two, if the confidence level of either the detection result or the classification result of the target item is higher than the first confidence threshold, then the detection result of the target item is valid; otherwise, it is invalid.

[0223] When the first judgment result is that the two do not match, if the confidence level of either the detection result or the classification result of the target item is higher than the second confidence threshold, then the detection result of the target item is valid; otherwise, it is invalid. The second confidence threshold is greater than the first confidence threshold.

[0224] In practice, the intermediate detection result Boxn output by any detection network model Qn is used as the detection result Boxn of the target item. The specific content of Boxn is shown in Table 3. Table 6 shows the classification results of the target item.

[0225] Table 6

[0226] Network Hi Category C2_i Confidence level PH3 Network Hj Category C2_j Confidence level PH6

[0227] In the embodiments of the present invention, for ease of description, the category conversion result for any training sample category C1_n corresponding to any detection network model is f(C1_n).

[0228] If f(C1_n)∈{C2_i, C2_j}, that is, the result after category conversion belongs to {C2_i, C2_j}, it indicates that the detection result of the target item and the classification result of the target item both have the same prediction result, and the two match.

[0229] When the two match, if the confidence level PQ in the detection result of the target item is higher than the first confidence threshold, it indicates that the confidence level of the detection result Boxn of the target item is very high. In this case, the detection result Boxn of the target item can be directly determined to be valid; otherwise, the detection result Boxn of the target item is determined to be invalid. Alternatively, if the confidence level (PH3) of the corresponding category (e.g., category C2_i) in the classification result of the target item is higher than the second confidence threshold, it indicates that the target item largely belongs to category C2_i. In this case, the detection result Boxn of the target item can be directly determined to be valid; otherwise, the detection result Boxn of the target item is determined to be invalid.

[0230] When the two do not match, if the confidence level PQ in the detection result of the target item is higher than the third confidence threshold, it indicates that the confidence level of the detection result Boxn of the target item is extremely high. In this case, the detection result Boxn of the target item can be directly determined to be valid; otherwise, the detection result Boxn of the target item is determined to be invalid. Alternatively, if the confidence level (PH3) of the corresponding category (e.g., category C2_i) in the classification result of the target item is higher than the fourth confidence threshold, it indicates that the target item largely belongs to category C2_i. In this case, the detection result Boxn of the target item can be directly determined to be valid; otherwise, the detection result Boxn of the target item is determined to be invalid.

[0231] In practice, when the detection result of the target item is invalid, the category information of the target item in the detection result can be converted by using the preset second training sample category mapping relationship information, and the similar item identification result of the target item can be obtained based on the category conversion result.

[0232] The second training sample category mapping information refers to the correspondence between the training sample category in which the same training sample is located when it is used as a training sample corresponding to the detection network model and the similar training sample category in which it is located when it is used as a training sample corresponding to the classification network model. The similar training sample category is a training sample category with the same visual features, or one or more training sample categories at the same level in the training sample category table.

[0233] For example, when the training sample category corresponding to the detection network model is scissors, the similar training sample categories corresponding to the classification network model can be fruit knives and daggers. The correspondence between scissors and fruit knives and daggers is a second training sample category mapping relationship. After classifying the scissors category, f(C1_n) represents fruit knives and daggers. At this point, it is determined whether the fruit knife or dagger matches the classification result of the target item.

[0234] When the two match, if the confidence level PQ in the detection result of the target item is higher than the fifth confidence threshold, it indicates that the confidence level of the detection result Boxn of the target item is very high. In this case, the detection result Boxn of the target item can be directly determined to be valid. Otherwise, the detection result Boxn of the target item is determined to be invalid. Here, the fifth confidence threshold is higher than the first confidence threshold. Alternatively, if the confidence level (PH3) of the corresponding category (e.g., category C2_i) in the classification result of the target item is higher than the sixth confidence threshold, it indicates that the target item largely belongs to category C2_i. In this case, the detection result Boxn of the target item can be directly determined to be valid. Otherwise, the detection result Boxn of the target item is determined to be invalid. Here, the sixth confidence threshold is higher than the second confidence threshold.

[0235] When the two do not match, if the confidence level PQ in the detection result of the target item is higher than the seventh confidence threshold, it indicates that the confidence level of the detection result Boxn of the target item is extremely high. In this case, the detection result Boxn of the target item can be directly determined to be valid. Otherwise, the detection result Boxn of the target item is determined to be invalid. Here, the seventh confidence threshold is higher than the third confidence threshold and is higher than the fifth confidence threshold. Alternatively, if the confidence level (PH3) of the corresponding category (e.g., category C2_i) in the classification result of the target item is higher than the eighth confidence threshold, it indicates that the target item largely belongs to category C2_i. In this case, the detection result Boxn of the target item can be directly determined to be valid. Otherwise, the detection result Boxn of the target item is determined to be invalid. Here, the eighth confidence threshold is higher than the fourth confidence threshold and is higher than the sixth confidence threshold.

[0236] It should be noted that, in actual implementation, the first confidence threshold to the eighth confidence threshold can be set according to the actual situation, which will not be elaborated here.

[0237] As can be seen from the above, the target classification method in this embodiment of the invention can further improve the accuracy of target detection by using the classification result of the target item to determine whether the detection result of the target item is valid, and taking the valid detection result of the target item as the identification result of the target item.

[0238] To enable those skilled in the art to better understand and implement the present invention, the target classification system corresponding to the above method is described in detail below.

[0239] Figure 8 This is a schematic diagram of the structure of a target classification system according to an embodiment of the present invention. (Refer to...) Figure 8 The target classification system 80 may include: two or more target classification units 81 and a first fusion unit 82. Wherein:

[0240] The target classification unit 81 is adapted to classify the detection result of the target item when it receives the detection result of the target item; the detection result of the target item includes: target item category information, target item location information and confidence information;

[0241] The first fusion unit 82 is adapted to fuse the intermediate classification results output by each of the classification network models to obtain the classification result of the target item; the classification result of the target item includes: target item category information and confidence information;

[0242] Wherein, the training samples corresponding to any two of the classification network models belong to different categories.

[0243] In one embodiment of the present invention, the target classification system 80 may further include: a detection unit 83. Wherein:

[0244] The detection unit 83 is adapted to perform target detection on the input image using a preset detection network model after receiving the input image, and obtain the detection result of the target item.

[0245] In one embodiment of the present invention, reference is made to... Figure 8 The detection unit 8383 may include: two or more target detection subunits 831 and a second fusion subunit 832. Wherein:

[0246] The classification subunit 831 is adapted to classify the detection result of the target item using a preset classification network model when the detection result of the target item is received.

[0247] The second fusion subunit 832 is adapted to fuse the intermediate detection results output by each of the detection network models to obtain the detection result of the target item; the intermediate detection result includes: target item category information, target item location information and confidence information;

[0248] Wherein, the training samples corresponding to any two of the classification network models belong to different categories.

[0249] In one embodiment of the present invention, the target classification system 80 may further include: a judgment unit 84. Wherein:

[0250] The judgment unit 84 is adapted to use the classification result of the target item to determine whether the detection result of the target item is valid, and to use the valid detection result of the target item as the identification result of the target item.

[0251] For each unit in the target classification system 80, please refer to the description of the corresponding steps in the target classification method above for specific implementation, which will not be repeated here.

[0252] As can be seen from the above, the target classification system 80 in this embodiment of the invention can effectively improve the accuracy of target detection compared to using a single network model for target detection.

[0253] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A target classification method, characterized in that, include: After receiving the input image, a preset detection network model is used to perform target detection on the input image to obtain the detection result of the target item; When the detection result of the target item is received, the detection result of the target item is classified using two or more preset classification network models; The detection results of the target item include: target item category information, target item location information, and confidence level information; The intermediate classification results output by each of the classification network models are fused to obtain the classification result of the target item; the classification result of the target item includes: target item category information and confidence information; Using the classification results of the target item, determine whether the detection result of the target item is valid, and take the valid detection result of the target item as the identification result of the target item; The step of determining whether the detection result of the target item is valid using the classification result of the target item includes: using preset first training sample category mapping relationship information to perform category conversion on the target item category information in the detection result of the target item, and based on the category conversion result, determining whether the detection result of the target item is valid, and obtaining the identification result of the target item; when the detection result of the target item is invalid, using preset second training sample category mapping relationship information to perform category conversion on the target item category information in the detection result of the target item, and based on the category conversion result, obtaining the identification result of similar items of the target item; Wherein, the first training sample category mapping information is the correspondence between the category in which the same training sample is located when it is used as a training sample corresponding to the detection network model and the category in which it is located when it is used as a training sample corresponding to the classification network model; the second training sample category mapping information is the correspondence between the training sample category in which the same training sample is located when it is used as a training sample corresponding to the detection network model and the similar training sample category in which it is located when it is used as a training sample corresponding to the classification network model; the similar training sample category is a training sample category with the same visual features, or one or more training sample categories at the same level in the training sample category table; The step of determining whether the detection result of the target item is valid based on the category conversion result includes: determining whether the target item category information in the detection result of the target item matches the target item category information in the classification result of the target item, to obtain a first determination result; when the first determination result indicates a match, if the confidence level in the detection result of the target item is higher than a first confidence threshold, or the confidence level in the classification result of the target item is higher than a second confidence threshold, then the detection result of the target item is valid, otherwise invalid; when the first determination result indicates a mismatch, if the confidence level in the detection result of the target item is higher than a third confidence threshold, or the confidence level in the classification result of the target item is higher than a fourth confidence threshold, then the detection result of the target item is valid, otherwise invalid; the third confidence threshold is greater than the first confidence threshold, and the fourth confidence threshold is greater than the second confidence threshold.

2. The target classification method as described in claim 1, characterized in that, The training samples corresponding to any two of the classification network models are in completely different categories, or partially different categories.

3. The target classification method as described in claim 1, characterized in that, In the training samples corresponding to each of the classification network models, the difference in the number of training samples between any two categories is less than the second sample difference threshold.

4. The target classification method as described in claim 3, characterized in that, The training samples corresponding to the two or more classification network models are also determined by the following method: Training samples of two categories with different visual features but a difference in the number of training samples less than a preset third sample difference threshold are used as training samples corresponding to the same classification network model. The third sample difference threshold is less than the second sample difference threshold.

5. The target classification method as described in claim 1, characterized in that, The process of fusing the intermediate classification results output by each of the classification network models to obtain the classification result of the target item includes: When the target item category information of the same target item exists only in one intermediate classification result, the intermediate classification result shall be used as the classification result of the target item. When the target item category information of the same target item exists only in two intermediate classification results, the intermediate classification result with higher confidence among the two intermediate classification results shall be used as the classification result of the target item. When the target item category information of the same target item exists in more than three intermediate classification results, a preset number of intermediate classification results are selected as the classification result of the target item based on the confidence information of the three or more intermediate classification results.

6. The target classification method as described in claim 1, characterized in that, The step of receiving an input image and then using a preset detection network model to perform target detection on the input image to obtain the detection result of the target item includes: After receiving the input image, the target object is detected in the input image using two or more preset detection network models. The intermediate detection results output by each of the detection network models are fused to obtain the detection result of the target item; the intermediate detection result includes: target item category information, target item location information, and confidence information; Wherein, the training samples corresponding to any two of the detection network models belong to different categories.

7. The target classification method as described in claim 6, characterized in that, The training samples corresponding to any two of the aforementioned detection network models are of completely different categories, or partially different categories.

8. The target classification method as described in claim 6, characterized in that, The difference in the number of categories of training samples corresponding to any two of the detection network models is less than a preset first category difference; the difference in the number of training samples of any two categories in the training samples corresponding to each detection network model is less than a first sample difference threshold.

9. The target classification method as described in claim 6, characterized in that, The two or more detection network models were trained using the following method: Count the number of training samples in each category among all training samples, and determine the first category with the most training samples and the second category with the fewest training samples; The training sample interval defined by the number of training samples in the first category to the number of training samples in the second category is evenly divided into K parts to obtain K-1 sample number separation values, where K is a positive integer greater than 1. Based on K-1 sample count separators and the number of training samples for each category, the obtained training samples are separated into categories to obtain K training samples for the detection network model, and then trained.

10. The target classification method as described in claim 6, characterized in that, The process of fusing the intermediate detection results output by each of the detection network models to obtain the detection result of the target item includes: When the target item category information of the same target item exists only in one of the intermediate detection results, the intermediate detection result is used as the detection result of the target item. When the target item category information of the same target item exists in two or more intermediate detection results, the detection result of the target item is obtained by combining the target item location information and confidence information in the two or more intermediate detection results.

11. The target classification method as described in claim 10, characterized in that, The step of combining the target item location information and confidence information from the two or more intermediate detection results to obtain the detection result of the target item includes: For any intermediate detection result with a confidence level that is not the lowest among the two or more intermediate detection results, calculate the first area respectively, and perform the following operations on each of the first areas: calculate the overlapping area between the first area and each of the second areas, and perform a rejection operation on the corresponding intermediate detection result according to the size of the overlapping area, and take the intermediate detection result after the rejection operation among the two or more intermediate detection results as the detection result of the target item in this round; The detection results of the target item in each round, and the intermediate detection result with the lowest confidence among the two or more intermediate detection results, are used together as the detection result of the target item. Wherein, the first area is the target item area corresponding to the target item location information of the intermediate detection result, the second area is the target item area corresponding to the target item location information of the remaining intermediate detection results among the two or more intermediate detection results, and the confidence level of the intermediate detection result containing the first area is higher than the confidence level of the intermediate detection result containing the second area.

12. The target classification method as described in claim 11, characterized in that, The step of removing intermediate detection results based on the size of the overlapping area includes: Based on the size of the overlapping area, determine the difference between the area of ​​each target item and the overlapping area in the two intermediate detection results. When the difference between the area of ​​the target item and the overlapping area is less than the first area threshold, an elimination operation is performed on the intermediate detection results where the area of ​​the target item whose overlapping area difference is less than the first area threshold is located. When the difference between the area of ​​the target item and the overlapping area is greater than the first area threshold, a rejection operation is performed based on the difference between the areas of the target items in the two intermediate detection results.

13. The target classification method as described in claim 12, characterized in that, The step of performing a removal operation based on the difference between the areas of the target items in the two intermediate detection results includes: Calculate the difference between the areas of the target item in the two corresponding intermediate detection results; When the difference between the areas of the target items is less than the second area threshold, the intermediate detection results with lower confidence are removed.

14. The target classification method as described in claim 12, characterized in that, The step of performing a removal operation based on the difference between the areas of the target items in the two intermediate detection results further includes: Use the target item location information from the removed intermediate detection results to update the target item location information from the retained intermediate detection results.

15. The target classification method as described in claim 11, characterized in that, Calculate the first area of ​​each intermediate detection result that is not the lowest confidence level in descending order of confidence level, and perform a removal operation.

16. The target classification method as described in claim 1, characterized in that, The training samples corresponding to the two or more classification network models are determined by the following method: From all training samples, select training samples with the same visual features, and use the training samples with the same visual features as the training samples corresponding to the same classification network model. From the remaining training samples selected according to visual features, training samples with the same item usage characteristics are selected, and these training samples with the same item usage characteristics are used as training samples corresponding to the same classification network model. From the remaining training samples selected according to visual features and item usage characteristics, training samples with the same item material are selected, and these training samples with the same item material are used as training samples corresponding to the same classification network model. Wherein, the training samples corresponding to any two of the classification network models belong to different categories.

17. A target classification system, characterized in that, include: The detection unit is adapted to perform target detection on the input image using a preset detection network model after receiving the input image, and obtain the detection result of the target item; Two or more target classification units, wherein when a detection result of a target item is received, the target classification unit is adapted to classify the detection result of the target item using two or more preset classification network models respectively; The detection results of the target item include: target item category information, target item location information, and confidence level information; The second fusion unit is adapted to fuse the intermediate classification results output by each of the classification network models to obtain the classification result of the target item; the classification result of the target item includes: target item category information and confidence information; The judgment unit is adapted to use the classification result of the target item to determine whether the detection result of the target item is valid, and to take the valid detection result of the target item as the recognition result of the target item; The judgment unit is adapted to use preset first training sample category mapping relationship information to perform category conversion on the target item category information in the detection result of the target item, and based on the category conversion result, determine whether the detection result of the target item is valid, and obtain the recognition result of the target item; when the detection result of the target item is invalid, it uses preset second training sample category mapping relationship information to perform category conversion on the target item category information in the detection result of the target item, and based on the category conversion result, obtain the recognition result of similar items of the target item. Wherein, the first training sample category mapping information is the correspondence between the category in which the same training sample is located when it is used as a training sample corresponding to the detection network model and the category in which it is located when it is used as a training sample corresponding to the classification network model; the second training sample category mapping information is the correspondence between the training sample category in which the same training sample is located when it is used as a training sample corresponding to the detection network model and the similar training sample category in which it is located when it is used as a training sample corresponding to the classification network model; the similar training sample category is a training sample category with the same visual features, or one or more training sample categories at the same level in the training sample category table; The step of determining whether the detection result of the target item is valid based on the category conversion result includes: determining whether the target item category information in the detection result of the target item matches the target item category information in the classification result of the target item, to obtain a first determination result; when the first determination result indicates a match, if the confidence level in the detection result of the target item is higher than a first confidence threshold, or the confidence level in the classification result of the target item is higher than a second confidence threshold, then the detection result of the target item is valid, otherwise invalid; when the first determination result indicates a mismatch, if the confidence level in the detection result of the target item is higher than a third confidence threshold, or the confidence level in the classification result of the target item is higher than a fourth confidence threshold, then the detection result of the target item is valid, otherwise invalid; the third confidence threshold is greater than the first confidence threshold, and the fourth confidence threshold is greater than the second confidence threshold.

18. The target classification system as described in claim 17, characterized in that, The detection unit includes: Two or more target detection subunits, wherein the detection subunits are adapted to detect target objects in the input image by using a preset detection network model after receiving the input image; The second fusion subunit is adapted to fuse the intermediate detection results output by each of the detection network models to obtain the detection result of the target item; the intermediate detection result includes: target item category information, target item location information, and confidence information; Wherein, the training samples corresponding to any two of the classification network models belong to different categories.

Images