Open set target detection method and device based on multi-label classification, equipment and medium

Abstract

The invention provides an open set target detection method and device based on multi-label classification, equipment and a medium. The method comprises the steps that a multi-label classification layer is embedded in any one of fusion layers of an open set target detection model; converting the multi-label classification layer and the front K layers of the open set target detection model into a first reasoning layer, and converting the rear N-K layers of the open set target detection model into a second reasoning layer; inputting the target image and preset M target category texts into a first reasoning layer to obtain image features, M text features and confidence coefficients of the M target category texts; filtering the M text features based on the confidence of the M target category texts to obtain m filtered text features; and under the condition that the value of m is not zero, inputting the filtered m text features and image features into a second reasoning layer to obtain a classification result of the target image. The method is good in reasoning acceleration effect, does not need extra training cost, does not affect the positioning accuracy, and is high in universality.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to an open set target detection method, device, equipment and medium based on multi-label classification. Background Art

[0002] As scenes such as urban governance, security monitoring, and smart transportation become increasingly complex, how to improve the inference speed of open-set target detection in complex scenes is a key problem that needs to be solved urgently in open-set target detection models. In existing technologies, the model distillation solution is used to accelerate inference, but the positioning accuracy decreases and requires additional training costs; the model quantization solution is used to accelerate inference, but the versatility is poor; the model pruning solution is used to accelerate inference, but specific equipment is required. Summary of the invention

[0003] The present invention provides an open-set target detection method, device, equipment and medium based on multi-label classification, so as to solve the defects in the prior art that the solution of model distillation is used to accelerate reasoning, the positioning accuracy is reduced, and additional training costs are required; the solution of model quantization is used to accelerate reasoning, the versatility is poor; the solution of model pruning is used to accelerate reasoning, and specific equipment is required.

[0004] In a first aspect, the present invention provides an open set target detection method based on multi-label classification, comprising: Embedding a multi-label classification layer in any one of the fusion layers of the open set target detection model, wherein the open set target detection model includes N layers, where N is a natural number greater than or equal to 1; Convert the multi-label classification layer and the first K layers before the multi-label classification layer in the open set target detection model into a first inference layer, and convert the NK layer after the multi-label classification layer in the open set target detection model into a second inference layer, where K is a natural number greater than or equal to 1; Inputting a target image and M preset target category texts into the first reasoning layer, obtaining image features and M text features corresponding to the target image output by the first reasoning layer, and confidences of the M target category texts, where M is a natural number greater than or equal to 1; Based on the confidences of the M target category texts, the M text features are filtered to obtain m filtered text features, where m is a non-negative integer; When the value of m is not zero, the filtered m text features and the image features are input into the second reasoning layer to obtain a classification result of the target image output by the second reasoning layer; The open set target detection model is trained based on sample images, multiple sample category texts, and classification labels of the sample images.

[0005] In some embodiments, the inputting the target image and the preset M target category texts into the first reasoning layer, obtaining the image features and M text features corresponding to the target image output by the first reasoning layer, and the confidence of the M target category texts, includes: Input the target image and M preset target category texts into the first K layers, and obtain the region-level image features and M text features corresponding to the target image output by the first K layers; Based on the multi-label classification layer, the similarity between each text feature and the image feature of the corresponding area is calculated, and based on the similarity between each text feature and the image feature of the corresponding area, the category confidence of each text feature is calculated.

[0006] In some embodiments, filtering the M text features based on the confidences of the M target category texts to obtain the filtered m text features includes: Based on the confidence levels of the M target category texts and a preset threshold, the M text features are filtered to obtain the filtered m text features.

[0007] In some embodiments, after obtaining the filtered m text features, the method further includes: When the value of m is zero, the open set target detection model directly outputs the classification result of the target image as empty.

[0008] In some embodiments, the training process of the open set object detection model includes: Obtain a sample image and multiple sample category texts, and determine the classification label of the sample image; Using the sample image and multiple sample category texts as training samples, and using the classification labels of the sample images as sample labels to train an initial open set object detection model; The parameters of the initial open set target detection model are iteratively optimized to obtain the open set target detection model.

[0009] In some embodiments, the initial open set target detection model includes a first initial reasoning layer and a second initial reasoning layer, the first initial reasoning layer includes an initial multi-label classification layer and an initial front K layer, the initial front K layer refers to the K layers in the initial open set target detection model that are located before the initial multi-label classification layer, and the initial open set target detection model includes N layers; Correspondingly, the training of the initial open set target detection model includes: Input the sample image and multiple sample category texts into the initial front K layers, and obtain the region-level sample image features and multiple sample text features corresponding to the sample image output by the initial front K layers; Based on the initial multi-label classification layer, calculating the similarity between each sample text feature and the sample image feature of the corresponding area; Calculating a loss function value based on the similarity between each text feature and the image feature of the corresponding area and the classification label of the sample image; Based on the loss function value, the initial multi-objective classification layer is trained.

[0010] In a second aspect, the present invention further provides an open set target detection device based on multi-label classification, comprising: An embedding unit, configured to embed a multi-label classification layer in any one of the fusion layers of the open set target detection model, wherein the open set target detection model includes N layers, where N is a natural number greater than or equal to 1; a conversion unit, configured to convert the multi-label classification layer and the first K layers before the multi-label classification layer in the open set target detection model into a first inference layer, and convert the NK layer after the multi-label classification layer in the open set target detection model into a second inference layer, where K is a natural number greater than or equal to 1; An inference unit, configured to input a target image and M preset target category texts into the first inference layer, and obtain image features and M text features corresponding to the target image output by the first inference layer, as well as confidences of the M target category texts, where M is a natural number greater than or equal to 1; A filtering unit, used for filtering the M text features based on the confidences of the M target category texts to obtain m text features after filtering, where m is a non-negative integer; A classification unit, used for inputting the filtered m text features and the image features into the second reasoning layer when the value of m is not zero, to obtain a classification result of the target image output by the second reasoning layer; The open set target detection model is trained based on sample images, multiple sample category texts, and classification labels of the sample images.

[0011] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an open set target detection method based on multi-label classification as described above is implemented.

[0012] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the open-set target detection methods based on multi-label classification as described above.

[0013] In a fifth aspect, the present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the open set target detection methods based on multi-label classification as described above.

[0014] The open-set target detection method, device, equipment and medium based on multi-label classification provided by the present invention embed a multi-label classification layer in any layer of the fusion layer of the open-set target detection model, convert the multi-label classification layer and the first K layers in the open-set target detection model into the first reasoning layer, convert the last NK layers in the open-set target detection model into the second reasoning layer, input the target image and preset M target category texts into the first reasoning layer, obtain the image features and M text features corresponding to the target image, and the confidence of the M target category texts, filter the M text features based on the confidence of the M target category texts to obtain the filtered m text features, and when the m value is not zero, input the filtered m text features and image features into the second reasoning layer to obtain the classification result of the target image, with good reasoning acceleration effect, no additional training cost, no impact on positioning accuracy, and strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 This is one of the flow charts of the open set target detection method based on multi-label classification provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of the process of deploying an open set target detection model provided by an embodiment of the present invention; Figure 3 is a flow chart of inputting a target image and M preset target category texts into a first reasoning layer provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of a flow chart of filtering multiple text features based on the confidence of multiple target category texts provided by an embodiment of the present invention; Figure 5 This is a second flow chart of an open set target detection method based on multi-label classification provided by an embodiment of the present invention; Figure 6 is a schematic diagram of the structure of an open set target detection device based on multi-label classification provided by an embodiment of the present invention; Figure 7 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] At present, a single small target detection model can no longer meet the detection needs of complex scenes. The large open set detection model that has undergone large-scale pre-training has universal detection capabilities by combining multimodal information of text and images, and can adapt to complex detection tasks in various scenes. However, due to the large number of parameters and slow reasoning speed, the large open set detection model is restricted from being widely used in production. The following are the more popular open set target detection reasoning acceleration solutions: 1) Model distillation: construct a relatively small model, transfer the positioning and classification capabilities of the large open set detection model to the small model through transfer learning, and use the small model instead of the large model during deployment to achieve acceleration. This process requires additional training costs, and there is knowledge loss in the distillation process. The distilled small model is inferior to the original large model in positioning accuracy. 2) Model quantization scheme: Pre-trained models are usually inferred with 32-bit floating-point numbers, and model quantization is the process of converting the weight values ​​and activation values ​​of the trained model from high precision to low precision, such as converting 32-bit floating-point numbers into 8-bit integers, which greatly reduces the amount of calculation in the inference process. However, the value range of some operations in the open set detection model is large, and there is overflow or underflow after quantization. After conversion, some models have a significant decrease in positioning accuracy, making this acceleration scheme not universal. 3) Model pruning scheme: There is randomness in the model training process, and the weight coefficients of some parameters are small, which has little impact on the output results. Therefore, by setting pruning rules to avoid the calculation of these redundant parameters in the model, the amount of calculation can be reduced during inference and acceleration can be achieved. However, non-structured pruning requires special hardware support, and structured pruning has a greater impact on the model positioning accuracy.

[0019] In addition, traditional image classification methods use spatial dimension squeezing operations such as global average pooling and global maximum pooling, which can easily lose fine-grained information in the image, such as smaller objects.

[0020] To this end, an embodiment of the present invention provides an open set target detection method, device, equipment and medium based on multi-label classification, by embedding a multi-label classification layer in any layer of the fusion layer of the open set target detection model, converting the multi-label classification layer and the first K layers in the open set target detection model into a first reasoning layer, converting the last NK layers in the open set target detection model into a second reasoning layer, inputting the target image and preset M target category texts into the first reasoning layer, obtaining image features and M text features corresponding to the target image, and the confidence of the M target category texts, filtering the M text features based on the confidence of the M target category texts to obtain filtered m text features, and when the m value is not zero, inputting the filtered m text features and image features into the second reasoning layer to obtain the classification result of the target image, the present invention has good reasoning acceleration effect, does not require additional training cost, does not affect the positioning accuracy, and has strong versatility.

[0021] Figure 1 One of the flow charts of the open set target detection method based on multi-label classification provided by the embodiment of the present invention. Figure 1 As shown, an open set target detection method based on multi-label classification is provided, comprising the following steps: steps 110 to 150. The steps of the method flow are only used as a possible implementation mode of the present invention.

[0022] Step 110: embed a multi-label classification layer in any layer of the fusion layer of the open set target detection model, where the open set target detection model includes N layers, where N is a natural number greater than or equal to 1.

[0023] Among them, the open set object detection model includes an image feature encoder, a fusion layer and a decoder.

[0024] It should be noted that the image features in the decoder remain unchanged. Embedding the multi-label classification layer in the decoder is equivalent to embedding it at the end of the fusion layer, which has less acceleration and is not recommended as an embedding position in applications. The image feature encoder is used to encode image features and implicitly aligns them with text features. Embedding the multi-label classification layer in the image feature encoder will result in poor similarity between the image features and text features input in the multi-label classification layer, and reduced classification reliability. Therefore, in the open-set object detection model, the embedding position can be selected in any layer of the fusion layer.

[0025] Preferably, a multi-label classification layer is embedded in the first layer of the fusion layer, which has good inference acceleration effect and strong classification reliability.

[0026] Step 120: convert the multi-label classification layer and the first K layers before the multi-label classification layer in the open set target detection model into the first inference layer, and convert the NK layer after the multi-label classification layer in the open set target detection model into the second inference layer, where K is a natural number greater than or equal to 1.

[0027] Figure 2 A schematic diagram of the process of deploying an open set target detection model provided by an embodiment of the present invention. Figure 2 As shown, the multi-label classification layer and the first K layers of the open set target detection model are converted to the first inference layer, corresponding to Figure 2 Engine1; convert the NK layer after the open set target detection model to the second reasoning layer, corresponding to Figure 2 MediumEngine2.

[0028] Step 130: Input the target image and M preset target category texts into the first reasoning layer, and obtain the image features and M text features corresponding to the target image output by the first reasoning layer, as well as the confidence of the M target category texts, where M is a natural number greater than or equal to 1.

[0029] Optionally, the original M target category texts in the open set target detection model are input into the text encoder to obtain M text features.

[0030] Optionally, the image features corresponding to the target image are region-level image features, including image features of each region of the target image.

[0031] Figure 3 A schematic diagram of a process of inputting a target image and M preset target category texts into a first reasoning layer provided in an embodiment of the present invention, such as Figure 3 As shown, in some embodiments, the target image and the preset M target category texts are input into the first reasoning layer, and the image features and M text features corresponding to the target image output by the first reasoning layer, as well as the confidence of the M target category texts, are obtained, including: Input the target image and the preset M target category texts into the first K layers, and obtain the region-level image features and M text features corresponding to the target image output by the first K layers; Based on the multi-label classification layer, the similarity between each text feature and the image feature of the corresponding area is calculated, and based on the similarity between each text feature and the image feature of the corresponding area, the category confidence of each text feature is calculated.

[0032] Optionally, preset M target category texts are determined according to user input.

[0033] like Figure 3As shown in the figure, after the target image passes through the first K layers, the image features with dimensions of H×W×C are obtained, where H and W represent the height and width of the image, and C represents the number of features of the image. The image features are flattened and transposed to obtain the region-level image features (HW×C); the text encoder is used to process the preset M target category texts to obtain text features (M×C); the region-level image features and text features are matrix multiplied to obtain a similarity matrix (M×HW); the aggregation function is used to aggregate the region-level single-label classification results into the image-level multi-label classification results to obtain the confidence of each target category text.

[0034] It should be noted that the similarity matrix contains fine-grained regional image classification information. In order to aggregate this fine-grained information, the embodiment of the present invention adopts the maximum pooling operation along the HW dimension to obtain global multi-label classification information of size M×1. Due to the use of the maximum pooling operation, as long as there is a region in the target image that is similar to the target category text, the corresponding target category text can be retained during the reasoning process, thereby avoiding missed detection and ensuring that the detection accuracy of reasoning and training is consistent.

[0035] Step 140: Based on the confidences of the M target category texts, the M text features are filtered to obtain m filtered text features, where m is a non-negative integer.

[0036] It can be understood that by filtering the M text features, the amount of data processing can be reduced, thereby improving the detection efficiency.

[0037] Optionally, the value range of m is [0,M].

[0038] Figure 4 A schematic diagram of a process for filtering multiple text features based on the confidence of multiple target category texts provided by an embodiment of the present invention. Figure 4 As shown, in some embodiments, based on the confidence of M target category texts, M text features are filtered to obtain m filtered text features, including: Based on the confidence of the M target category texts and a preset threshold, the M text features are filtered to obtain m filtered text features.

[0039] Optionally, a category filtering mechanism is used to filter the M text features.

[0040] For example, M is 5, and the confidence levels of the M target category texts are 0.8, 0.1, 0.7, 0.1, and 0.8, respectively. Combined with the preset threshold (such as 0.6), a bool mask is generated. The mask indicates whether the corresponding text category exists in the target image. The mask is used to filter the 5 text features corresponding to the 5 target category texts to obtain 3 text features.

[0041] Step 150: when the value of m is not zero, the filtered m text features and image features are input into the second reasoning layer to obtain the classification result of the target image output by the second reasoning layer; Among them, the open set target detection model is trained based on sample images and multiple sample category texts, as well as the classification labels of the sample images.

[0042] It can be understood that by inputting the filtered m text features and image features into the second reasoning layer to obtain the classification result of the target image output by the second reasoning layer, the computational overhead of the second reasoning layer can be reduced and reasoning acceleration can be achieved.

[0043] In some embodiments, after obtaining the filtered m text features, the method further includes: When the value of m is zero, the open set object detection model directly outputs the classification result of the target image as empty.

[0044] It can be understood that when the m value is zero, it indicates that there is no corresponding object category in any text in the target image, and the processing of the second inference layer can be bypassed and returned directly, and the result is empty; in this case, the single calculation cost of reasoning is reduced from "first inference layer + second inference layer" to "first inference layer", achieving reasoning acceleration.

[0045] Figure 5 The second flow chart of the open set target detection method based on multi-label classification provided by the embodiment of the present invention. Figure 5 As shown in the figure, the open-set target detection method based on multi-label classification includes two parts: embedded multi-label classification layer training and inference acceleration; a multi-label classification layer is inserted in the pre-training process of the open-set target detection model to avoid additional training overhead; in the multi-label classification loss layer, the similarity between the image region features and the target category text features is used as the classification score, so the multi-label classification layer inserted after pre-training has fine-grained classification capabilities at the region level; in the inference process, when the model runs to the multi-label classification layer, the similarity between the image region features and the target category text features is calculated, and the multi-label confidence is obtained through global aggregation. According to the set threshold, the target categories below the threshold are deleted in the subsequent inference process, thereby achieving inference acceleration without affecting the accuracy of the detection results.

[0046] Embodiment 1: The typical application scenario of the present invention is the open-set target detection task that needs to be deployed on a regular basis. For example, in an urban governance scenario, it is necessary to deploy the detection function of 10 specific behaviors of personnel in specific monitoring. In actual scenarios, the proportion of time that the 10 behaviors appear at the same time is extremely low. The traditional open-set target detection large model acceleration solution needs to run the entire model to obtain the result, regardless of whether there is a corresponding behavior in the scene, which results in a huge waste of computing power.

[0047] In Example 1, a multi-label classification layer is embedded in the open set target detection model, and during deployment, the multi-label classification layer and the first K layers of the open set target detection model are converted to the first inference layer Engine1, and the last NK layers of the open set target detection model are converted to the second inference layer Engine2. No specific deployment equipment is required, and it is compatible with any open set target detection model structure. Assuming that the embedded multi-label classification layer is located in the first layer of the fusion layer, Engine1 includes an image feature encoder, Engine2 includes a fusion layer and a decoder, and Engine1 is always running during the inference process. Automatic processing can be divided into three cases: 1) When there is no corresponding behavior in the scene, an empty result is automatically returned; 2) When some categories exist in the scene, Engine2 is called to continue reasoning after reducing text features; 3) When all categories exist in the scene, the image features and all text features output by Engine1 are input into Engine2 to continue reasoning.

[0048] In this embodiment 1, the reasoning acceleration effect can be obtained in both case 1) and case 2). Since case 1) and case 2) account for a large proportion in actual applications, the average throughput of the model can be effectively improved without modifying the model parameters and without compromising the accuracy of model detection.

[0049] In an embodiment of the present invention, a multi-label classification layer is embedded in any layer of the fusion layer of an open set target detection model, the multi-label classification layer and the first K layers in the open set target detection model are converted into a first reasoning layer, and the last NK layers in the open set target detection model are converted into a second reasoning layer. The target image and the preset M target category texts are input into the first reasoning layer to obtain image features and M text features corresponding to the target image, as well as the confidences of the M target category texts. Based on the confidences of the M target category texts, the M text features are filtered to obtain the filtered m text features. When the m value is not zero, the filtered m text features and image features are input into the second reasoning layer to obtain the classification result of the target image. The reasoning acceleration effect is good, no additional training cost is required, the positioning accuracy is not affected, and the versatility is strong.

[0050] In some embodiments, the training process of the open set object detection model includes: Obtain a sample image and multiple sample category texts, and determine the classification label of the sample image; Using sample images and multiple sample category texts as training samples, and using the classification labels of sample images as sample labels to train an initial open set object detection model; The parameters of the initial open set target detection model are iteratively optimized to obtain the open set target detection model.

[0051] Optionally, the initial open set object detection model includes an initial image feature encoder, an initial fusion layer, and an initial decoder.

[0052] In some embodiments, the initial open set target detection model includes a first initial reasoning layer and a second initial reasoning layer, the first initial reasoning layer includes an initial multi-label classification layer and an initial first K layer, the initial first K layer refers to the K layers in the initial open set target detection model that are located before the initial multi-label classification layer, and the initial open set target detection model includes N layers; Correspondingly, the initial open set target detection model is trained, including: Input the sample image and multiple sample category texts into the initial first K layers, and obtain the region-level sample image features and multiple sample text features corresponding to the sample image output by the initial first K layers; Based on the initial multi-label classification layer, calculate the similarity between each sample text feature and the sample image feature of the corresponding area; Calculate the loss function value based on the similarity between each text feature and the image feature of the corresponding area and the classification label of the sample image; Based on the loss function value, train the initial multi-target classification layer.

[0053] It should be noted that the initial multi-label classification layer is a parameter-free layer and no additional training parameters are introduced.

[0054] Optionally, in the process of training the initial open set target detection model, a multi-label classification loss function is used to calculate the loss function value.

[0055] Among them, the calculation formula of the multi-label classification loss function is as follows: ; Where N represents the number of sample category texts, p(yi) represents the probability of the existence of sample category text i after maximum pooling aggregation, yi represents the true value of sample category text i, yi is 0 or 1, 0 means that sample category text i does not exist in the sample image, and 1 means that sample category text i exists in the sample image.

[0056] It can be understood that by introducing a multi-label loss function in the training process of the open-set object detection model, the training of the multi-label classification layer can be completed simultaneously without any additional training cost.

[0057] The following is a description of an open-set target detection device based on multi-label classification provided by an embodiment of the present invention. The open-set target detection device based on multi-label classification described below and the open-set target detection method based on multi-label classification described above can correspond to each other.

[0058] Figure 6 A schematic diagram of the structure of an open set target detection device based on multi-label classification provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the open set target detection device 600 based on multi-label classification includes: An embedding unit 610, configured to embed a multi-label classification layer in any one of the fusion layers of the open set object detection model, where the open set object detection model includes N layers, where N is a natural number greater than or equal to 1; a conversion unit 620, configured to convert the multi-label classification layer and the first K layers before the multi-label classification layer in the open set target detection model into a first inference layer, and convert the NK layers after the multi-label classification layer in the open set target detection model into a second inference layer, where K is a natural number greater than or equal to 1; The reasoning unit 630 is used to input the target image and the preset M target category texts into the first reasoning layer, and obtain the image features and M text features corresponding to the target image output by the first reasoning layer, as well as the confidences of the M target category texts, where M is a natural number greater than or equal to 1; A filtering unit 640 is used to filter the M text features based on the confidences of the M target category texts to obtain m filtered text features, where m is a non-negative integer; A classification unit 650, for inputting the filtered m text features and image features into the second reasoning layer when the value of m is not zero, to obtain a classification result of the target image output by the second reasoning layer; Among them, the open set target detection model is trained based on sample images and multiple sample category texts, as well as the classification labels of the sample images.

[0059] Optionally, the target image and the preset M target category texts are input into the first reasoning layer, and the region-level image features and M text features corresponding to the target image output by the first reasoning layer, as well as the confidence of the M target category texts, are obtained, including: Input the target image and the preset M target category texts into the first K layers, and obtain the image features and M text features corresponding to the target image output by the first K layers; Based on the multi-label classification layer, the similarity between each text feature and the image feature of the corresponding area is calculated, and based on the similarity between each text feature and the image feature of the corresponding area, the category confidence of each text feature is calculated.

[0060] Optionally, based on the confidences of the M target category texts, the M text features are filtered to obtain the filtered m text features, including: Based on the confidence of the M target category texts and a preset threshold, the M text features are filtered to obtain m filtered text features.

[0061] Optionally, the open set target detection device 600 based on multi-label classification further includes: The output unit is used to directly output the classification result of the target image as empty when the value of m is zero.

[0062] Optionally, the training process of the open set object detection model includes: Obtain a sample image and multiple sample category texts, and determine the classification label of the sample image; Using sample images and multiple sample category texts as training samples, and using the classification labels of sample images as sample labels to train an initial open set object detection model; The parameters of the initial open set target detection model are iteratively optimized to obtain the open set target detection model.

[0063] Optionally, the initial open set target detection model includes a first initial reasoning layer and a second initial reasoning layer, the first initial reasoning layer includes an initial multi-label classification layer and an initial first K layer, the initial first K layer refers to K layers before the initial multi-label classification layer in the initial open set target detection model, and the initial open set target detection model includes N layers; Correspondingly, the initial open set target detection model is trained, including: Input the sample image and multiple sample category texts into the initial first K layers, and obtain the region-level sample image features and multiple sample text features corresponding to the sample image output by the initial first K layers; Based on the initial multi-label classification layer, calculate the similarity between each sample text feature and the sample image feature of the corresponding area; Calculate the loss function value based on the similarity between each text feature and the image feature of the corresponding area and the classification label of the sample image; Based on the loss function value, train the initial multi-target classification layer.

[0064] It should be noted here that the open-set target detection device based on multi-label classification provided in an embodiment of the present invention can implement all the method steps implemented in the above-mentioned open-set target detection method embodiment based on multi-label classification, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0065] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute an open set target detection method based on multi-label classification, the method comprising: embedding a multi-label classification layer in any layer of the fusion layer of the open set target detection model, the open set target detection model comprising N layers, N being a natural number greater than or equal to 1; converting the multi-label classification layer and the first K layers before the multi-label classification layer in the open set target detection model into a first reasoning layer, converting the NK layers after the multi-label classification layer in the open set target detection model into a second reasoning layer, K being a natural number greater than or equal to 1; inputting the target image and the preset M target category texts into the first reasoning layer, obtaining the image features and M text features corresponding to the target image output by the first reasoning layer, and the confidence of the M target category texts, M is a natural number greater than or equal to 1; based on the confidence of the M target category texts, the M text features are filtered to obtain m filtered text features, where m is a non-negative integer; when the value of m is not zero, the m filtered text features and image features are input into the second reasoning layer to obtain the classification result of the target image output by the second reasoning layer; wherein the open set target detection model is trained based on sample images and multiple sample category texts, as well as the classification labels of the sample images.

[0066] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0067] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the open set target detection method based on multi-label classification provided by the above methods, the method comprising: embedding a multi-label classification layer in any layer of the fusion layer of the open set target detection model, the open set target detection model includes N layers, N is a natural number greater than or equal to 1; converting the multi-label classification layer and the first K layers before the multi-label classification layer in the open set target detection model into a first inference layer, and converting the NK layers after the multi-label classification layer in the open set target detection model into a second inference layer, K is a natural number greater than or equal to 1; inputting the target image and the preset M target category texts into the first inference layer, obtaining the image features and M text features corresponding to the target image output by the first inference layer, and the confidence of the M target category texts, M is a natural number greater than or equal to 1; based on the confidence of the M target category texts, the M text features are filtered to obtain m filtered text features, where m is a non-negative integer; when the value of m is not zero, the m filtered text features and image features are input into the second reasoning layer to obtain the classification result of the target image output by the second reasoning layer; wherein the open set target detection model is trained based on sample images and multiple sample category texts, as well as the classification labels of the sample images.

[0068] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the open set target detection method based on multi-label classification provided by the above methods, the method comprising: embedding a multi-label classification layer in any layer of the fusion layer of the open set target detection model, the open set target detection model comprising N layers, N being a natural number greater than or equal to 1; converting the multi-label classification layer and the first K layers before the multi-label classification layer in the open set target detection model into a first inference layer, converting the NK layers after the multi-label classification layer in the open set target detection model into a second inference layer, K being a natural number greater than or equal to 1; inputting the target image and the preset M target category texts into the first inference layer, obtaining the image features and M text features corresponding to the target image output by the first inference layer, as well as the confidence of the M target category texts, M is a natural number greater than or equal to 1; based on the confidence of the M target category texts, the M text features are filtered to obtain m filtered text features, where m is a non-negative integer; when the value of m is not zero, the m filtered text features and image features are input into the second reasoning layer to obtain the classification result of the target image output by the second reasoning layer; wherein the open set target detection model is trained based on sample images and multiple sample category texts, as well as the classification labels of the sample images.

[0069] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0070] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An open set target detection method based on multi-label classification, characterized in that: include: Embedding a multi-label classification layer in any one of the fusion layers of the open set target detection model, wherein the open set target detection model includes N layers, where N is a natural number greater than or equal to 1; Convert the multi-label classification layer and the first K layers before the multi-label classification layer in the open set target detection model into a first inference layer, and convert the NK layer after the multi-label classification layer in the open set target detection model into a second inference layer, where K is a natural number greater than or equal to 1; Inputting a target image and M preset target category texts into the first reasoning layer, obtaining image features and M text features corresponding to the target image output by the first reasoning layer, and confidences of the M target category texts, where M is a natural number greater than or equal to 1; Based on the confidences of the M target category texts, the M text features are filtered to obtain m filtered text features, where m is a non-negative integer; When the value of m is not zero, the filtered m text features and the image features are input into the second reasoning layer to obtain a classification result of the target image output by the second reasoning layer; The open set target detection model is trained based on sample images, multiple sample category texts, and classification labels of the sample images.

2. The open set target detection method based on multi-label classification according to claim 1, characterized in that: The step of inputting a target image and M preset target category texts into the first reasoning layer, and obtaining image features and M text features corresponding to the target image output by the first reasoning layer, as well as confidences of the M target category texts, includes: Input the target image and M preset target category texts into the first K layers, and obtain the region-level image features and M text features corresponding to the target image output by the first K layers; Based on the multi-label classification layer, the similarity between each text feature and the image feature of the corresponding area is calculated, and based on the similarity between each text feature and the image feature of the corresponding area, the category confidence of each text feature is calculated.

3. The open set target detection method based on multi-label classification according to claim 1, characterized in that: The filtering of the M text features based on the confidence of the M target category texts to obtain the filtered m text features includes: Based on the confidence levels of the M target category texts and a preset threshold, the M text features are filtered to obtain the filtered m text features.

4. The open set target detection method based on multi-label classification according to claim 1, characterized in that: After obtaining the filtered m text features, the method further includes: When the value of m is zero, the open set target detection model directly outputs the classification result of the target image as empty.

5. The open set target detection method based on multi-label classification according to claim 1, characterized in that: The training process of the open set object detection model includes: Obtain a sample image and multiple sample category texts, and determine the classification label of the sample image; Using the sample image and multiple sample category texts as training samples, and using the classification labels of the sample images as sample labels to train an initial open set object detection model; The parameters of the initial open set target detection model are iteratively optimized to obtain the open set target detection model.

6. The open set target detection method based on multi-label classification according to claim 5, characterized in that: The initial open set target detection model includes a first initial reasoning layer and a second initial reasoning layer, the first initial reasoning layer includes an initial multi-label classification layer and an initial front K layer, the initial front K layer refers to the K layers in the initial open set target detection model that are located before the initial multi-label classification layer, and the initial open set target detection model includes N layers; Correspondingly, the training of the initial open set target detection model includes: Input the sample image and multiple sample category texts into the initial front K layers, and obtain the region-level sample image features and multiple sample text features corresponding to the sample image output by the initial front K layers; Based on the initial multi-label classification layer, calculating the similarity between each sample text feature and the sample image feature of the corresponding area; Calculating a loss function value based on the similarity between each text feature and the image feature of the corresponding area and the classification label of the sample image; Based on the loss function value, the initial multi-objective classification layer is trained.

7. An open set target detection device based on multi-label classification, characterized in that: include: An embedding unit, configured to embed a multi-label classification layer in any one of the fusion layers of the open set target detection model, wherein the open set target detection model includes N layers, where N is a natural number greater than or equal to 1; a conversion unit, configured to convert the multi-label classification layer and the first K layers before the multi-label classification layer in the open set target detection model into a first inference layer, and convert the NK layer after the multi-label classification layer in the open set target detection model into a second inference layer, where K is a natural number greater than or equal to 1; An inference unit, configured to input a target image and M preset target category texts into the first inference layer, and obtain image features and M text features corresponding to the target image output by the first inference layer, as well as confidences of the M target category texts, where M is a natural number greater than or equal to 1; A filtering unit, used for filtering the M text features based on the confidences of the M target category texts to obtain m text features after filtering, where m is a non-negative integer; A classification unit, used for inputting the filtered m text features and the image features into the second reasoning layer when the value of m is not zero, to obtain a classification result of the target image output by the second reasoning layer; The open set target detection model is trained based on sample images, multiple sample category texts, and classification labels of the sample images.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the open set target detection method based on multi-label classification as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the open set target detection method based on multi-label classification as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the open set target detection method based on multi-label classification as described in any one of claims 1 to 6 is implemented.

Images