New class mining and classification method, device and equipment under open world and medium

Abstract

The application relates to a new class mining and classification method, device, equipment and medium under an open world, wherein the method comprises the following steps: collecting object detection data of a target, and generating at least one region proposal network of the target based on a centrality regression loss function and a preset region proposal strategy; fine-tuning a ViT structure by using a preset contrast learning algorithm, performing feature extraction on each region proposal network of the at least one region proposal network according to the fine-tuned result, obtaining at least one classification feature of each region proposal network, classifying target detection image data based on a preset clustering algorithm and each classification feature of the at least one classification feature, and obtaining a final classification result. Therefore, the problems that a target detection algorithm can only recognize and detect a single data set, is difficult to be extended to multiple data sets or real scenes, and the target detection algorithm cannot judge or detect unknown objects are solved.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a novel class discovery and classification method, apparatus, equipment and medium in an open world. Background Technology

[0002] Current mainstream object detection algorithms can only identify and detect objects on a single dataset and cannot be extended to multiple datasets or real-world scenarios. This makes it a key research focus in open-world object detection how to handle new objects that are not labeled on a single dataset in complex images with a large number of objects and categories.

[0003] Currently, related technologies consider "open set" recognition in visual applications that are close to displays. In open set recognition, it is necessary to identify known classes and reject unknown classes. It defines open set recognition as a problem of minimizing empirical risk + open space risk. The system associates the object to be identified with known classes (determines whether it is a class that appears in the training set), and marks the identified class as "unknown". When the number of unknown classes reaches a certain number, the system uses incremental learning and other methods to mark the unknown classes as "known". Some related technologies further discuss the open set recognition problem and use statistical extreme value theory to evaluate the posterior probability.

[0004] Meanwhile, based on the research topic of open-world detection, some related technologies have proposed the following ORE solutions:

[0005] 1) In complex images, use an energy-based classification head and RPN to identify unknown images, identifying images without complete annotations as "unknown";

[0006] 2) When corresponding unknown categories are continuously received during training, contrastive learning is used in the feature space to learn clusters, learning these identified unknown categories without forgetting the previously learned categories.

[0007] During training, the technique can divide COCO into 20 pre-learned VOC categories and 60 unlearned categories, and then divide the 60 categories into three consecutive tasks with semantic drift. By dividing them into three tasks, the remaining 60 categories are gradually opened up for incremental learning.

[0008] The related art can also introduce incremental learning in the definition of OWOD, and through a two-stage method, first use an open set detector to detect unknown categories, and then use an unsupervised method to predict unknown categories and define new categories; in addition, some related research finds that for large unlabeled data sets, there will be problems of recognizing and locating new objects, so by using prior knowledge about certain object types, two memory modules (i.e., working memory and semantic memory) are used to discover new categories.

[0009] However, the prior art cannot learn all categories in advance in the real scene, and when the model cannot judge or detect unknown objects, the target is classified as a background category or an unknown category, which greatly affects the performance of open world object detection, and needs to be solved urgently. SUMMARY

[0010] The present application provides a new class mining and classification method, device, equipment and medium under an open world, to solve the problem that the current target detection algorithm can only recognize and detect a single data set, and is difficult to extend to multiple data sets or real scenes, so that the target detection algorithm cannot judge or detect unknown objects.

[0011] The first aspect of the present application provides a new class mining and classification method under an open world, comprising the following steps: collecting object detection data of a target, and generating at least one region proposal network of the target based on a center degree regression loss function and a preset region proposal strategy; using a preset contrast learning algorithm to fine-tune the ViT structure, extracting features of each region proposal network of the at least one region proposal network according to the fine-tuned result, obtaining at least one classification feature of each region proposal network, and classifying the target detection image data based on a preset clustering algorithm and each classification feature of the at least one classification feature, to obtain a final classification result.

[0012] Optionally, in an embodiment of the present application, collecting object detection data of a target, and generating at least one region proposal network of the target based on a center degree regression loss function and a preset region proposal strategy, comprises: optimizing a class-agnostic detector based on the center degree regression loss function and an IOU loss function, to obtain a class-agnostic detector that meets a preset condition; generating a first anchor box and a second anchor box of the object detection data according to a class-aware detector and the optimized class-agnostic detector.

[0013] Optionally, in an embodiment of the present application, the ViT structure is fine-tuned using a preset contrast learning algorithm, and each region proposal network of the at least one region proposal network is subjected to feature extraction according to a result of the fine-tuning, to obtain at least one classification feature of each region proposal network, including: cropping image data in a VOC dataset based on a preset cropping strategy to obtain at least one data block, and generating a label set based on the at least one first data block; training the first anchor frame and the second anchor frame based on a preset COCO dataset, and processing a preset unlabeled dataset according to a training result to generate at least one second data block; and constructing a total loss function according to the at least one first data block, the label set and the at least one second data block.

[0014] Optionally, in an embodiment of the present application, the target detection image data is classified based on a preset clustering algorithm and each classification feature of the at least one classification feature, including: classifying the object detection data based on a preset over-clustering strategy, a K-means algorithm meeting a preset requirement and each classification feature of each region proposal network.

[0015] Optionally, in an embodiment of the present application, a mathematical expression of the total loss function is as follows:

[0016]

[0017] wherein λ is a to-be-determined coefficient, is a contrast loss function under unsupervised learning, is a contrast loss function under supervised learning, is the unlabeled dataset, is the label set.

[0018] The second aspect embodiment of the present application provides a new class mining and classification device in an open world, including: a collection module configured to collect object detection data of a target, and generate at least one region proposal network of the target based on a centrality regression loss function and a preset region proposal strategy; an extraction module configured to fine-tune a ViT structure using a preset contrast learning algorithm, and perform feature extraction on each region proposal network of the at least one region proposal network according to a result of the fine-tuning, to obtain at least one classification feature of each region proposal network; and a classification module configured to classify the target detection image data based on a preset clustering algorithm and each classification feature of the at least one classification feature, to obtain a final classification result.

[0019] Optionally, in an embodiment of the present application, the collecting module comprises: a first generating unit configured to optimize a class-agnostic detector based on the centrality regression loss function and the IOU loss function to obtain a class-agnostic detector satisfying a preset condition; and a second generating unit configured to generate first anchor boxes and second anchor boxes of the object detection data based on a class-aware detector and the optimized class-agnostic detector.

[0020] Optionally, in an embodiment of the present application, the extracting module comprises: a cropping unit configured to crop image data in a VOC dataset based on a preset cropping strategy to obtain at least one data block, and generate a label set based on the at least one first data block; a training unit configured to train the first anchor boxes and the second anchor boxes based on a preset COCO dataset, and process a preset unlabeled dataset according to a training result to generate at least one second data block; and a function unit configured to construct a total loss function based on the at least one first data block, the label set and the at least one second data block.

[0021] Optionally, in an embodiment of the present application, the classification module is specifically configured to classify the object detection data based on a preset over-clustering strategy, a K-means algorithm satisfying a preset requirement and each classification feature of each region proposal network.

[0022] Optionally, in an embodiment of the present application, a mathematical expression of the total loss function is as follows:

[0023]

[0024] wherein λ is a to-be-determined coefficient, is a contrast loss function under unsupervised learning, is a contrast loss function under supervised learning, is the unlabeled dataset, is the label set.

[0025] An embodiment of the third aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the new class mining and classification method in an open world as described in the above embodiments.

[0026] An embodiment of the fourth aspect of the present application provides a computer readable storage medium, which stores a computer program executable by a processor to implement the new class mining and classification method in an open world as described above.

[0027] Therefore, the embodiments of the present application have the following beneficial effects:

[0028] The embodiments of the present application can collect object detection data of a target, generate at least one region proposal network of the target based on a centrality regression loss function and a preset region proposal strategy, fine-tune a ViT structure by using a preset contrast learning algorithm, perform feature extraction on each region proposal network of the at least one region proposal network according to a result after fine-tuning, obtain at least one classification feature of each region proposal network, classify target detection image data based on a preset clustering algorithm and each classification feature of the at least one classification feature, and obtain a final classification result. The present application is aimed at a new class classification problem in an open world, generates a better region proposal network by fusing different object detection methods, obtains different class information by using a contrast learning method and a clustering method, thereby effectively improving the accuracy and the number of classes of new class discovery on a public dataset, enhancing the generalization ability of the model in multiple environments, improving the accuracy of the model in a real environment, reducing the time loss in the training of the entire model, and having a wide application prospect. Thus, the problem that a target detection algorithm can only recognize and detect a single dataset and is difficult to be extended to multiple datasets or real scenes, so that the target detection algorithm cannot judge or detect unknown objects, and other problems are solved.

[0029] Additional aspects and advantages of the present application will be made apparent by the following description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0030] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings, wherein:

[0031] Figure 1 A flowchart of a new class mining and classification method in an open world according to an embodiment of the present application;

[0032] Figure 2 A logical comparison diagram of a new class mining and classification method in an open world according to an embodiment of the present application and a traditional detection method;

[0033] Figure 3 A logical architecture diagram of a new class mining and classification method in an open world according to an embodiment of the present application;

[0034] Figure 4 An example diagram of a new class mining and classification device in an open world according to an embodiment of the present application;

[0035] Figure 5 A structural diagram of an electronic device according to an embodiment of the present application.

[0036] Wherein, 10 - new class mining and classification device under open world, 100 - acquisition module, 200 - extraction module, 300 - classification module, 501 - memory, 502 - processor, 503 - communication interface. DETAILED DESCRIPTION

[0037] The embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0038] The new class mining and classification method, device, equipment and medium under open world of the embodiments of the present application are described below with reference to the accompanying drawings. In view of the problems mentioned in the above background art, the present application provides a new class mining and classification method under open world, in which the object detection data of the target is acquired, and at least one region proposal network of the target is generated based on the centrality regression loss function and the preset region proposal strategy; the preset contrast learning algorithm is used to fine-tune the ViT structure, and according to the fine-tuned result, each region proposal network of the at least one region proposal network is subjected to feature extraction to obtain at least one classification feature of each region proposal network, and the target detection image data is classified based on the preset clustering algorithm and each classification feature of the at least one classification feature to obtain a final classification result. The present application is directed to the new class classification problem under open world, and through the fusion of different object detection methods, a better region proposal network is generated, and through the contrast learning method and the clustering method, different class information is obtained, so that the accuracy and the number of classes of new class discovery can be effectively improved on the public dataset, the generalization ability of the model in multiple environments is enhanced, the accuracy of the model in the real environment is improved, the time loss in the whole model training is reduced, and the application prospect is wide. Thus, the problems that the current target detection algorithm can only recognize and detect a single dataset and is difficult to be extended to multiple datasets or real scenes, so that the target detection algorithm cannot judge or detect unknown objects are solved.

[0039] Specifically, Figure 1 A flowchart of the new class mining and classification method under open world provided by the embodiments of the present application.

[0040] As Figure 1 shown, the new class mining and classification method under open world includes the following steps:

[0041] In step S101, the object detection data of the target is acquired, and at least one region proposal network of the target is generated based on the centrality regression loss function and the preset region proposal strategy.

[0042] As can be appreciated by those skilled in the art, to a great extent, the discovery of new classes in object detection relies on whether the detection network accurately mines and captures potential unknown objects, and the region proposal network (RPN) trained according to the 20 classes in the PASCAL VOC dataset can be biased and ignore unknown classes in COCO, therefore, embodiments of the present application can collect object detection data of a target, and obtain a class-agnostic RPN based on a centerness regression loss function and the RPN in the existing network, which can generate more generalizable objectness scores and detect more objects, as shown in Figure 2 .

[0043] Optionally, in an embodiment of the present application, object detection data of a target is collected, and a target region proposal network is generated based on a centerness regression loss function and a preset region proposal strategy, including: optimizing a class-agnostic detector based on a centerness regression loss function and an IOU loss function to obtain a class-agnostic detector satisfying a preset condition; and generating a first anchor box and a second anchor box of the object detection data according to the class-aware detector and the optimized class-agnostic detector.

[0044] It should be noted that in an embodiment of the present application, the class-aware detector in Faster-RCNN and the class-related loss function are replaced by a class-agnostic loss function, which only estimates the objectness of a region, therefore, embodiments of the present application can optimize the class-agnostic detector, thereby combining with the class-aware detector, and the optimization of the class-agnostic detector in Faster-RCNN includes the following two steps:

[0045] 1) the classification loss of the class-agnostic detector is replaced by a centerness regression loss function;

[0046] 2) the classification loss of the ROI (Region of Interest) in the class-agnostic detector is replaced by an IOU (Intersection Of Union) regression loss function.

[0047] Therefore, the embodiments of the present application can effectively reduce the bias and generate more potential unknown objects by forcing the class-aware detector, i.e., the class-aware detector, to only focus on the positioning-aware IOU loss or the centerness loss, thereby generating a classification diagnostic RPN.

[0048] Meanwhile, the embodiments of the present application also adopt the region proposal method of the standard RPN. For the collected object detection data, the class-agnostic detector combined with the class-aware detector after optimization can obtain two different anchor frames.

[0049] It can be understood that since the anchor frame obtained by the class-aware detector has good detection accuracy for known classes, but its generalization is poor and performs poorly on unknown classes; and the anchor frame obtained by the class-agnostic detector although performs worse than the anchor frame obtained by the class-aware detector on known classes, but it can be well generalized to unknown classes. Therefore, the embodiments of the present application combine the two anchor frames to obtain a new set of anchor frames, which has the advantages of the above two anchor frames, as shown in FIG. 4, thereby effectively ensuring the recognition accuracy of the model for known classes. Figure 3

[0050] In step S102, the ViT structure is fine-tuned by using a preset contrast learning algorithm, and at least one classification feature of each region proposal network of the at least one region proposal network is extracted according to the fine-tuned result, to obtain at least one classification feature of each region proposal network.

[0051] After obtaining the above region proposal network, so that the model has better region proposal for the positioning of the object, the embodiments of the present application can also fine-tune the ViT (Vision Transformer, vision transformer) initialized by DINO based on a contrast learning method, and use unsupervised learning on VOC data and self-supervised learning on COCO data, to obtain a feature extractor, and further improve and improve the detection performance of the model.

[0052] ​Optionally, in an embodiment of the present application, the ViT structure is fine-tuned by using a preset contrast learning algorithm, and each region proposal network of the at least one region proposal network is subjected to feature extraction according to a result of the fine-tuning, to obtain at least one classification feature of each region proposal network, comprising: cropping image data in a VOC dataset based on a preset cropping strategy to obtain at least one data block, and generating a label set based on the at least one first data block; training a first anchor box and a second anchor box based on a preset COCO dataset, and processing a preset unlabeled dataset according to a training result to generate at least one second data block; and constructing a total loss function according to the at least one first data block, the label set and the at least one second data block.

[0053] Specifically, in an embodiment of the present application, the image in the VOC dataset can be first cropped into data blocks based on actual background to form a label set B L ; then, the two anchor boxes obtained above are trained in the COCO dataset, and the unlabeled dataset B U is processed to obtain corresponding data blocks; thereafter, two different views x, x' are generated by random enhancement for the same block, and the mathematical expression of the unsupervised contrast loss function in the embodiment of the present application is as follows:

[0054]

[0055] wherein z, z' are respectively corresponding extracted features, and τ is a hyperparameter.

[0056] For the image in the VOC dataset, the mathematical expression of the self-supervised contrast loss is as follows:

[0057]

[0058] wherein N i represents an index of x with the same label.

[0059] Therefore, the embodiment of the present application can combine the unsupervised contrast loss and the self-supervised contrast loss to obtain a total loss function.

[0060] Optionally, in an embodiment of the present application, the mathematical expression of the total loss function is as follows:

[0061]

[0062] wherein λ is a to-be-determined coefficient, is a contrast loss function under unsupervised learning, is a contrast loss function under supervised learning, is an unlabeled dataset, is a label set.

[0063] It should be noted that in the embodiments of the present application, by combining the unsupervised contrast loss and the self-supervised contrast loss, a total loss function is obtained, and the mathematical expression of the loss function is as follows:

[0064]

[0065] wherein λ is a to-be-determined coefficient, is a contrast loss function under unsupervised learning, is a contrast loss function under supervised learning, is an unlabeled data set, is a labeled set.

[0066] Therefore, the embodiments of the present application use unsupervised learning on VOC data, use self-supervised learning on COCO data, and construct a total loss function, so as to obtain a better feature extractor to obtain the features of the target object, thereby effectively improving the performance of target detection and recognition.

[0067] In step S103, the target detection image data is classified based on the preset clustering algorithm and each classification feature of the at least one classification feature, to obtain a final classification result.

[0068] After obtaining the features of the target object, further, the embodiments of the present application can classify the target object to divide it into known classes and different unknown classes, so that in the object detection process, not only new classes can be mined, but also different unknown classes can be distinguished.

[0069] Optionally, in an embodiment of the present application, classifying the target detection image data based on the preset clustering algorithm and each classification feature of the at least one classification feature includes: classifying the object detection data based on each classification feature of the preset over-clustering strategy, the K-means algorithm satisfying the preset requirement, and each region proposal network.

[0070] It should be noted that after obtaining the features of the target object, the embodiments of the present application can use an over-clustering strategy to increase the estimated number of clusters to improve the purity of clustering; further, the embodiments of the present application can also use a small-batch K-means algorithm with online update of cluster center to classify the object detection data.

[0071] Therefore, the embodiments of the present application classify the target object by using the related clustering algorithm, thereby improving the detection performance, improving the scalability of object detection in an open world, enhancing the generalization ability of the model in multiple environments, improving the accuracy of the model in a real environment, and reducing the time loss in the training of the entire model.

[0072] The novel class discovery and classification method proposed in this application for open world environments involves collecting object detection data of the target and generating at least one region proposal network for the target based on a centrality regression loss function and a preset region proposal strategy. A preset contrastive learning algorithm is used to fine-tune the ViT structure. Based on the fine-tuning results, features are extracted from each region proposal network to obtain at least one classification feature for each region proposal network. The target detection image data is then classified based on a preset clustering algorithm and each classification feature of the at least one classification feature to obtain the final classification result. This application addresses the novel class classification problem in open world environments by fusing different object detection methods to generate a better region proposal network. Simultaneously, contrastive learning and clustering methods are used to obtain different category information, thereby effectively improving the accuracy and number of categories in novel class discovery on public datasets, enhancing the model's generalization ability in multiple environments, improving the model's accuracy in real-world environments, and reducing the time consumption in the overall model training process. It has broad application prospects.

[0073] Secondly, a novel class mining and classification apparatus in an open world, according to embodiments of this application, is described with reference to the accompanying drawings.

[0074] Figure 4 This is a block diagram of a novel class discovery and classification device in an open world according to an embodiment of this application.

[0075] like Figure 4 As shown, the new class mining and classification device 10 in the open world includes: a collection module 100, an extraction module 200, and a classification module 300.

[0076] The acquisition module 100 is used to acquire object detection data of the target and generate at least one region proposal network of the target based on the centrality regression loss function and a preset region proposal strategy.

[0077] The extraction module 200 is used to fine-tune the ViT structure using a preset contrastive learning algorithm, and extract features from each region proposal network of at least one region proposal network based on the fine-tuning results, so as to obtain at least one classification feature of each region proposal network.

[0078] The classification module 300 is used to classify target detection image data based on a preset clustering algorithm and each classification feature with at least one classification feature to obtain the final classification result.

[0079] Optionally, in one embodiment of this application, the acquisition module 100 includes: a first generation unit and a second generation unit.

[0080] The first generation unit is configured to optimize the class-agnostic detector based on a centrality regression loss function and an IOU loss function to obtain a class-agnostic detector satisfying a preset condition.

[0081] The second generation unit is configured to generate first anchor boxes and second anchor boxes of the object detection data according to the class-aware detector and the optimized class-agnostic detector.

[0082] Optionally, in an embodiment of the present application, the extraction module 200 comprises a clipping unit, a training unit and a function unit.

[0083] The clipping unit is configured to clip image data in the VOC dataset based on a preset clipping strategy to obtain at least one data block, and generate a label set based on the at least one first data block.

[0084] The training unit is configured to train the first anchor boxes and the second anchor boxes based on a preset COCO dataset, and process a preset unlabeled dataset according to a training result to generate at least one second data block.

[0085] The function unit is configured to construct a total loss function according to the at least one first data block, the label set and the at least one second data block.

[0086] Optionally, in an embodiment of the present application, the classification module 300 is specifically configured to classify the object detection data based on a preset over-clustering strategy, a preset K-means algorithm and each classification feature of each region proposal network.

[0087] Optionally, in an embodiment of the present application, a mathematical expression of the total loss function is as follows:

[0088]

[0089] wherein λ is a to-be-determined coefficient, is a contrast loss function under unsupervised learning, is a contrast loss function under supervised learning, is an unlabeled dataset, is a label set.

[0090] It should be noted that the foregoing explanation and description of the embodiment of the method for new class mining and classification under an open world also apply to the embodiment of the device for new class mining and classification under an open world, which will not be described herein again.

[0091] According to the new class mining and classification device under an open world provided by the embodiment of the application, object detection data of a target is collected, at least one region proposal network of the target is generated based on a centrality regression loss function and a preset region proposal strategy; a preset contrast learning algorithm is used to fine-tune a ViT structure, feature extraction is performed on each region proposal network of the at least one region proposal network according to a result of the fine-tuning, at least one classification feature of each region proposal network is obtained, a target detection image data is classified based on a preset clustering algorithm and each classification feature of the at least one classification feature, and a final classification result is obtained. The application is aimed at the new class classification problem under an open world, better region proposal networks are generated by fusing different object detection methods, different class information is obtained through a contrast learning method and a clustering method, so that the accuracy and the number of classes of new class discovery can be effectively improved on a public data set, the generalization ability of a model in multiple environments is enhanced, the accuracy of the model in a real environment is improved, the time loss in the training of the entire model is reduced, and the application prospect is wide.

[0092] Figure 5 The electronic device provided by the embodiment of the application is shown in a structural schematic diagram. The electronic device can include:

[0093] The memory 501, the processor 502, and the computer program stored in the memory 501 and executable on the processor 502.

[0094] The processor 502 implements the new class mining and classification method under an open world provided by the above-mentioned embodiments when executing the program.

[0095] Further, the electronic device further includes:

[0096] The communication interface 503 is used for communication between the memory 501 and the processor 502.

[0097] The memory 501 is used to store the computer program executable on the processor 502.

[0098] The memory 501 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0099] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected with each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 5 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0100] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can complete communication between each other through an internal interface.

[0101] The processor 502 can be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0102] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the above new class mining and classification method in an open world.

[0103] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0104] Moreover, the terms "first", "second", "third", etc. are used herein only to describe different steps or categories of steps. Thus, the use of the term "first" does not imply that different steps must be in a time sequence. Nor is it implied that a "first" step must precede a "second" step, that a "second" step, etc. must follow a "first" step, etc. Furthermore, when a process or method is described herein with several steps or several categories of steps, it should be understood that these are merely illustrative of the steps that can be employed in the process or method. Not all of the steps can be required, and in some cases, additional steps can be employed. The order of the steps can be varied, and some of the steps can be performed simultaneously. The steps can be performed in an order different than that described herein. The steps can be performed in any order, unless otherwise specified.

[0105] Any process or method described in flowcharts or otherwise described herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for performing specific logic functions (or steps) or portions thereof, and the various embodiments of the application can include additional or fewer steps performing the same or equivalent functions. In some embodiments, the order of steps can be varied, and / or some steps can be performed simultaneously, unless otherwise specifically noted. The various embodiments of the application can be embodied in a number of different forms, all of which have been contemplated to be within the scope of the applicable patent princi¬ ples described herein.

[0106] Logic and / or steps represented in flowcharts or otherwise described herein, for example, can be embodied in computer-readable media, which can be any available media that can be accessed by a general purpose or special purpose computing system, device, or apparatus to execute instructions stored in the media. By way of example, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general purpose or special purpose computing system, device, or apparatus. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or combination thereof) to a general purpose or special purpose computing system, device, or apparatus, the computer-readable media used for this purpose can be any available media that can be accessed by this system, device, or apparatus. Combinations of the above should also be included within the scope of the computer-readable media.

[0107] It should be understood that parts of the present application can be realized in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution system. If realized in hardware and in another embodiment, any one or a combination of the following technologies known in the art can be used: discrete logic circuit with logic gate circuit for implementing logic functions on data signals, application specific integrated circuit with suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA), etc.

[0108] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiments can be completed by a program instructing the relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0109] In addition, each functional unit in each embodiment of the present application can be integrated into one processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software functional module. The integrated module, if realized in the form of a software functional module and sold or used as an independent product, can also be stored in a computer readable storage medium.

[0110] The above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

Claims

1. A new class mining and classification method under open world, characterized in that, The method comprises the following steps: collecting object detection data of a target, and generating at least one region proposal network of the target based on a centrality regression loss function and a preset region proposal strategy; initializing a ViT structure based on a preset DINO, fine-tuning the initialized ViT structure by using a preset contrast learning algorithm, performing feature extraction on each region proposal network of the at least one region proposal network according to a result of the fine-tuning, and obtaining at least one classification feature of each region proposal network; classifying the object detection data based on a preset clustering algorithm and each classification feature of the at least one classification feature, and obtaining a final classification result; The collecting object detection data of a target, and generating at least one region proposal network of the target based on a centrality regression loss function and a preset region proposal strategy comprises: optimizing a class-agnostic detector based on the centrality regression loss function and an IOU loss function, and obtaining a class-agnostic detector satisfying a preset condition; generating a first anchor box and a second anchor box of the object detection data according to a class-aware detector and the optimized class-agnostic detector.

2. The method of claim 1, wherein, The initializing the ViT structure based on the preset DINO, and fine-tuning the initialized ViT structure by using the preset contrast learning algorithm, and performing feature extraction on each region proposal network of the at least one region proposal network according to a result of the fine-tuning, and obtaining at least one classification feature of each region proposal network comprises: cropping image data in a VOC data set based on a preset cropping strategy to obtain at least one data block, and generating a label set based on at least one first data block; training the first anchor box and the second anchor box based on a preset COCO data set respectively, and processing a preset unlabeled data set according to a training result to generate at least one second data block; constructing a total loss function according to the at least one first data block, the label set and the at least one second data block.

3. The method of claim 2, wherein, The classifying the object detection data based on the preset clustering algorithm and each classification feature of the at least one classification feature comprises: classifying the object detection data based on a preset excessive clustering strategy, a K-means algorithm satisfying a preset requirement and each classification feature of each region proposal network, wherein the K-means algorithm satisfying the preset requirement is a small-batch K-means algorithm with online update of cluster center.

4. The method of claim 3, wherein, The mathematical expression of the total loss function is as follows: wherein λ is a to-be-determined coefficient, is a contrast loss function under unsupervised learning, is a contrast loss function under supervised learning, is the unlabeled data set, is the label set.

5. A new class mining and classifying apparatus under an open world, characterized by, comprise: a collecting module configured to collect object detection data of a target, and generate at least one region proposal network of the target based on a centrality regression loss function and a preset region proposal strategy; The extraction module is configured to initialize a ViT structure based on a preset DINO, fine-tune the initialized ViT structure by using a preset contrast learning algorithm, perform feature extraction on each region proposal network of the at least one region proposal network based on a result of the fine-tuning, and obtain at least one classification feature of each region proposal network. The classification module is configured to classify the object detection data based on a preset clustering algorithm and each classification feature of the at least one classification feature, and obtain a final classification result. The collection module includes: The first generation unit is configured to optimize a class-agnostic detector based on the center degree regression loss function and the IOU loss function, and obtain a class-agnostic detector that meets a preset condition. The second generation unit is configured to generate a first anchor box and a second anchor box of the object detection data based on a class-aware detector and the optimized class-agnostic detector.

6. The apparatus of claim 5, wherein, The extraction module includes: The cropping unit is configured to crop image data in a VOC dataset based on a preset cropping strategy, obtain at least one data block, and generate a label set based on at least one first data block. The training unit is configured to train the first anchor box and the second anchor box based on a preset COCO dataset, and process a preset unlabeled dataset based on a training result to generate at least one second data block. The function unit is configured to construct a total loss function based on the at least one first data block, the label set, and the at least one second data block.

7. The apparatus of claim 6, wherein, The classification module is specifically configured to classify the object detection data based on a preset over-clustering strategy, a K-means algorithm that meets a preset requirement, and each classification feature of each region proposal network, wherein the K-means algorithm that meets the preset requirement is a small-batch K-means algorithm with online update of clustering cluster centers. The mathematical expression of the total loss function is as follows:

8. The apparatus of claim 7, wherein, The computer program is executed by the processor to implement the new class mining and classification method in an open world as claimed in any one of claims 1-4. wherein λ is a to-be-determined coefficient, is a contrast loss function under unsupervised learning, is a contrast loss function under supervised learning, is the unlabeled data set, is the label set.

9. An electronic device, comprising: The program is executed by the processor to implement the new class mining and classification method in an open world as claimed in any one of claims 1-4. ​ 10. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images