Multi-scale and multi-stage open semantic target detection method and device and electronic equipment

Abstract

The invention relates to the technical field of target detection, in particular to a multi-scale and multi-stage open semantic target detection method and device and electronic equipment, and the method comprises the steps: obtaining a training image data set; performing multi-scale feature alignment on the training image data set to obtain a vision and language alignment result; generating a new class of pseudo labels according to the alignment result of the vision and the language, training a vision and language pre-training model according to the new class of pseudo labels and the real labels of the classes, and performing multi-stage iterative optimization on the pseudo labels by the vision and language pre-training model in the training process; and performing open semantic target detection on the target image based on the trained vision and language pre-training model. Therefore, the problems of how to better understand the local features and improve the object detection precision are solved.

Description

Technical Field

[0001] The present application relates to the technical field of target detection, and in particular to a multi-scale and multi-stage open semantic target detection method, device and electronic equipment. Background Art

[0002] In the field of computer vision, object detection is a key technical problem that involves accurately locating and classifying objects from images. OVD (Open-Vocabulary Object Detection) has received widespread attention. It can identify objects in images without additional annotations for new categories. CLIP (Contrastive Language-Image Pre-training Large Visual-Language Pre-training Model) is designed to learn the association between images and text. CLIP is trained on a large-scale image-text dataset, which can effectively link visual information with natural language descriptions, providing a new idea for solving OVD.

[0003] However, in the pre-training stage, CLIP mainly performs feature alignment based on the image description task. This process mainly focuses on the understanding of global features. For example, small objects may be ignored and the details of large objects may not be accurately captured, while target detection relies more on the understanding of local features. Secondly, the sizes of objects in the images to be detected are different, and CLIP only considers image understanding of image scale when training on image-text pairs. The model itself lacks the ability to understand multi-scale features. Finally, the existing methods of using CLIP for open semantic target detection have low detection accuracy and fail to give full play to the advantages of pre-training models. Summary of the invention

[0004] The present application provides a multi-scale and multi-stage open semantic target detection method, device and electronic device to solve the problem of how to better understand local features and improve object detection accuracy.

[0005] The first aspect of the present application provides a multi-scale and multi-stage open semantic object detection method, comprising the following steps: obtaining a training image dataset; performing multi-scale feature alignment on the training image dataset to obtain a vision and language alignment result; generating a pseudo-label for a new class based on the vision and language alignment result, and training a vision and language pre-training model based on the pseudo-label for the new class and the true annotation of the class, wherein the vision and language pre-training model performs multi-stage iterative optimization on the pseudo-label during the training process; and performing open semantic object detection on the target image based on the trained vision and language pre-training model.

[0006] Optionally, multi-scale feature alignment is performed on the training image dataset to obtain a vision and language alignment result, including: performing multi-scale feature alignment on the training image dataset on the target layer of the residual network to obtain an output result corresponding to the target layer; performing region of interest alignment and pooling processing on the output result corresponding to the target layer to obtain multiple groups of bounding box-level multi-scale features; and generating vision and language alignment results based on the multiple groups of bounding box-level multi-scale features.

[0007] Optionally, the target layer includes: a second layer, a third layer, and a fourth layer, and the output results corresponding to the target layer are aligned with the region of interest and pooled to obtain multiple groups of bounding box-level multi-scale features, including:

[0008] After the output results of the second layer are aligned with the regions of interest and pooled, the first group of bounding box-level features are generated through the third and fourth layers; after the output results of the third layer are aligned with the regions of interest and pooled, the second group of bounding box-level features are generated through the fourth layer; the output results of the fourth layer are aligned with the regions of interest and pooled to generate the third group of bounding box-level features; multiple groups of bounding box-level multi-scale features are generated based on the first group of bounding box-level features, the second group of bounding box-level features and the third group of bounding box-level features.

[0009] Optionally, a pseudo label of a new class is generated based on the alignment result of vision and language, including: locating the target area in the training image data set to generate a positioning score, and classifying the target area in the training image data set based on a target classifier to generate a classification score; determining the classification result based on the classification score, the positioning score and preset conditions, and using the area with the classification result of the new class as the pseudo label of the new class, wherein the new class is a category that does not appear in the training image data set.

[0010] Optionally, the vision and language pre-training model is trained according to the pseudo-labels of the new class and the true annotations of the class, including: iteratively optimizing the vision and language pre-training model according to the pseudo-labels of the new class and the true annotations of the class, and iteratively updating the pseudo-labels of the new class using the confidence scores of the outputs of the region of interest during the training process; if the number of iterations of the iterative optimization training reaches the target number or the detection accuracy of the vision and language pre-training model reaches the target accuracy, stopping the iterative optimization training, wherein the confidence score of the output of the region of interest is greater than a preset threshold.

[0011] Optionally, before classifying the target area in the training image dataset based on the target classifier to generate a classification score, it also includes: creating multiple position embedding branches in the attention classification layer of the visual language pre-training model; based on a fixed main architecture of the visual language pre-training model, adjusting the position encoding of the multiple position embedding branches to obtain the target classifier.

[0012] The second aspect of the present application provides a multi-scale and multi-stage open semantic object detection device, including: an acquisition module for acquiring a training image data set; an alignment module for performing multi-scale feature alignment on the training image data set to obtain a vision and language alignment result; a training module for generating a pseudo-label of a new class according to the vision and language alignment result, and training a vision and language pre-training model according to the pseudo-label of the new class and the real annotation of the class, wherein the vision and language pre-training model performs multi-stage iterative optimization on the pseudo-label during the training process; and a detection module for performing open semantic object detection on a target image based on the trained vision and language pre-training model.

[0013] Optionally, the alignment module is further used to perform multi-scale feature alignment on the training image dataset on the target layer of the residual network to obtain an output result corresponding to the target layer; perform region of interest alignment and pooling processing on the output result corresponding to the target layer to obtain multiple groups of bounding box-level multi-scale features; and generate vision and language alignment results based on multiple groups of bounding box-level multi-scale features.

[0014] Optionally, the target layer includes: a second layer, a third layer and a fourth layer, and the alignment module is further used to perform interest region alignment and pooling processing on the output result of the second layer, and then generate a first group of bounding box level features through the third and fourth layers; perform interest region alignment and pooling processing on the output result of the third layer, and then generate a second group of bounding box level features through the fourth layer; perform interest region alignment and pooling processing on the output result of the fourth layer to generate a third group of bounding box level features; generate multiple groups of bounding box-level multi-scale features based on the first group of bounding box-level features, the second group of bounding box-level features and the third group of bounding box-level features.

[0015] Optionally, the training module is further used to locate the target area in the training image data set to generate a positioning score, and classify the target area in the training image data set based on the target classifier to generate a classification score; determine the classification result according to the classification score, the positioning score and preset conditions, and use the area with the classification result as a new class as a pseudo-label for the new class, wherein the new class is a class that does not appear in the training image data set.

[0016] Optionally, the training module is further used to perform iterative optimization training on the vision and language pre-training model based on the pseudo-labels of the new class and the real labels of the class, and during the training process, the pseudo-labels of the new class are iteratively updated using the confidence scores of the outputs of the region of interest; if the number of iterations of the iterative optimization training reaches the target number or the detection accuracy of the vision and language pre-training model reaches the target accuracy, the iterative optimization training is stopped, wherein the confidence score of the output of the region of interest is greater than a preset threshold.

[0017] Optionally, the multi-scale and multi-stage open semantic target detection device also includes: an adjustment module, which is used to create multiple position embedding branches in the attention classification layer of the visual language pre-training model before classifying the target area in the training image data set based on the target classifier to generate a classification score; on the basis of fixing the main architecture of the visual language pre-training model, adjust the position encoding of the multiple position embedding branches to obtain the target classifier.

[0018] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a multi-scale and multi-stage open semantic target detection method as described in the above embodiment.

[0019] The fourth aspect of the present application provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, it is used to implement a multi-scale and multi-stage open semantic object detection method as described in the above embodiment.

[0020] The fifth aspect of the present application provides a computer program product, including: a computer program or instructions, which, when executed, implement the multi-scale and multi-stage open semantic object detection method as described in the above embodiments.

[0021] Therefore, this application has at least the following beneficial effects:

[0022] The embodiment of the present application can obtain rich feature representations at different scales by performing multi-scale feature alignment on the training image data set, thereby improving the detection capability of objects of different sizes, generating pseudo labels of new classes based on the alignment results, and training the visual and language pre-training model based on the pseudo labels of the new classes and the real annotations of the classes, which can improve the model's detection capability for objects of different scales and enhance the generalization capability of the model. Open semantic target detection is performed on the target image based on the trained model, which not only improves the detection accuracy of new classes of objects, but also can identify the detailed features of small objects. Thus, the problem of how to better understand local features and improve the accuracy of multi-scale object detection is solved.

[0023] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A flowchart of a multi-scale and multi-stage open semantic object detection method provided according to an embodiment of the present application;

[0026] Figure 2 This is an example diagram of multi-scale and multi-stage open semantic object detection according to an embodiment of the present application;

[0027] Figure 3 This is an example diagram of a multi-scale and multi-stage open semantic object detection device according to an embodiment of the present application;

[0028] Figure 4 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0030] The following describes the multi-scale and multi-stage open semantic target detection method, device, electronic device, and storage medium of the embodiment of the present application with reference to the accompanying drawings. In view of the above background technology, the present application provides a multi-scale and multi-stage open semantic target detection method. In this method, by performing multi-scale feature alignment on the training image data set, rich feature representations can be obtained at different scales, thereby improving the detection ability of objects of different sizes. According to the alignment results, pseudo labels of new classes are generated, and the visual and language pre-training model is trained based on the pseudo labels of the new classes and the real annotations of the classes, which can improve the model's detection ability for objects of different scales and enhance the generalization ability of the model. Open semantic target detection is performed on the target image based on the trained model, which not only improves the detection accuracy of new class objects, but also can identify the detailed features of small objects.

[0031] Specifically, Figure 1 A flowchart of a multi-scale and multi-stage open semantic object detection method provided in an embodiment of the present application.

[0032] like Figure 1 As shown, the multi-scale and multi-stage open semantic object detection method includes the following steps:

[0033] In step S101, a training image dataset is obtained.

[0034] The embodiments of the present application can collect a large number of images containing different scenes, objects and backgrounds. These images can be selected from public data sets or proprietary data sets, aiming to cover as wide a range of target categories as possible. In addition, in order to enhance the generalization ability of the model to new categories, the image data set selected by the embodiments of the present application can contain a variety of visual features, including but not limited to different perspectives, lighting conditions, and occlusion conditions.

[0035] In step S102, multi-scale feature alignment is performed on the training image dataset to obtain a vision and language alignment result.

[0036] In one embodiment of the present application, multi-scale feature alignment is performed on a training image dataset to obtain a vision and language alignment result, including: performing multi-scale feature alignment on the training image dataset on a target layer of a residual network to obtain an output result corresponding to the target layer; performing region of interest alignment and pooling processing on the output result corresponding to the target layer to obtain multiple groups of bounding box-level multi-scale features; and generating vision and language alignment results based on the multiple groups of bounding box-level multi-scale features.

[0037] It is understandable that the embodiments of the present application can process the training image data set on the target layer of the residual network (ResNet) using the real annotations of the class. The target layer generally refers to a specific layer in ResNet, such as the second layer, the third layer, and the fourth layer. These layers are selected as key points because they can provide feature representations at different levels of abstraction. By processing the outputs of these layers, the embodiments of the present application can ensure that the model can capture features of different scales from coarse to fine. In this process, each image passes through the residual network and generates corresponding features at the selected target layer.

[0038] like Figure 2 As shown, unlike the traditional method that only focuses on the output features of a single layer (the output result of the third layer is aligned with the region of interest and pooled before passing through the fourth layer), the embodiment of the present application performs region of interest alignment and pooling operations at different levels of the residual network (ResNet). In addition to the conventional third layer, the second and fourth layers are added as additional branches, with a total of 3 branches, each of which is responsible for processing object features within a specific scale range, and can obtain rich feature representations at different scales, thereby improving the detection ability of objects of different sizes.

[0039] Furthermore, the output results corresponding to the target layer are subjected to interest region alignment and pooling processing to obtain multiple groups of bounding box-level multi-scale features, including: after the output results of the second layer are subjected to interest region alignment and pooling processing, a first group of bounding box-level features are generated through the third and fourth layers; after the output results of the third layer are subjected to interest region alignment and pooling processing, a second group of bounding box-level features are generated through the fourth layer; the output results of the fourth layer are subjected to interest region alignment and pooling processing to generate a third group of bounding box-level features; and multiple groups of bounding box-level multi-scale features are generated according to the first group of bounding box-level features, the second group of bounding box-level features, and the third group of bounding box-level features.

[0040] It can be understood that the embodiment of the present application can perform region of interest alignment and pooling operations on the feature map generated by the target layer. Specifically, the output result of the third layer is subjected to region of interest alignment and pooling and then passed through the fourth layer to generate a second group of bounding box-level features, the output result of the second layer is subjected to region of interest alignment and pooling and then passed through the third and fourth layers to generate a first group of bounding box-level features, and the output result of the fourth layer is subjected to region of interest alignment and pooling to generate a third group of bounding box-level features, thereby generating multiple groups of bounding box-level features, and adding multiple groups of position encodings corresponding to the multiple groups of bounding box-level features obtained in the final attention layer of CLIP, and aligning the region-level (bounding box-level) features with the category text embedding, thereby achieving the effect of image-text alignment of CLIP to feature-text (bounding box-text) alignment.

[0041] It should be noted that the three branches are aligned separately in the training stage, and the average results of the three branches are taken as the classification result in the inference stage. This not only retains the advantage of CLIP global feature alignment, but also enhances its ability to understand local details, thereby improving the overall detection accuracy.

[0042] In step S103, a pseudo label of a new class is generated according to the vision and language alignment result, and a vision and language pre-training model is trained according to the pseudo label of the new class and the real annotation of the class, wherein the vision and language pre-training model performs multi-stage iterative optimization on the pseudo label during the training process.

[0043] In one embodiment of the present application, a pseudo label of a new class is generated based on the alignment result of vision and language, including: locating the target area in the training image data set to generate a positioning score, and classifying the target area in the training image data set based on a target classifier to generate a classification score; determining the classification result based on the classification score, the positioning score and preset conditions, and using the area with the classification result of the new class as the pseudo label of the new class, wherein the new class is a category that does not appear in the training image data set.

[0044] For example, in the embodiment of the present application, for a certain area numbered i, the positioning score given by the former is L i, the latter gives a classification score of C for a certain class (basic class or new class) i , then the embodiment of the present application can set the weight λ and the threshold thre, and select the preset condition λL i +(1-λ)C i >thre and the classification result is the region of the new class as the pseudo label of the new class.

[0045] Before generating classification scores by classifying target areas in a training image data set based on a target classifier, an embodiment of the present application may create multiple position embedding branches in the attention classification layer of a visual language pre-training model; based on a fixed main architecture of the visual language pre-training model, the position encodings of the multiple position embedding branches may be adjusted to obtain a target classifier.

[0046] Specifically, the embodiment of the present application can train a category-independent bounding box extractor based on CLIP (a detector obtained by fully supervised training of the basic class, and capable of extracting bounding boxes of new classes), and at the same time, by creating multiple position embedding branches in the last attention classification layer of CLIP and only fine-tuning the multiple position embeddings while fixing the weights of other parts of CLIP, a classifier capable of classifying regions of interest (capable of classifying new classes) is obtained.

[0047] Furthermore, the embodiments of the present application can generate pseudo labels for new classes based on a class-independent bounding box extractor (also a detector of the basic class) and a target classifier capable of classifying new classes.

[0048] In one embodiment of the present application, a vision and language pre-training model is trained based on pseudo-labels of new classes and true annotations of the classes, including: iteratively optimizing the vision and language pre-training model based on the pseudo-labels of new classes and true annotations of the classes, and iteratively updating the pseudo-labels of new classes using confidence scores of outputs of regions of interest during the training process; stopping the iterative optimization training if the number of iterations of the iterative optimization training reaches a target number or the detection accuracy of the vision and language pre-training model reaches a target accuracy, wherein the confidence score of the output of the region of interest is greater than a preset threshold.

[0049] Among them, the target number and the preset threshold can be set according to the situation and are not specifically limited. The embodiment of the present application can adopt a multi-stage iterative optimization mechanism, based on the obtained pseudo-labels of the new class, add the real annotations of the basic class, fine-tune the fully supervised detector of the basic class mentioned above, and use the scores output by the region of interest head to continuously iteratively update the pseudo-labels of the new class, which helps to continuously improve the model's ability to recognize new categories. The entire training process may include multiple such iterative cycles, and can also improve the overall robustness and generalization ability of the model while ensuring the quality of training.

[0050] Specifically, assuming that the step size is N, after fine-tuning N iterations, set the threshold t, select the prediction results of the interest region head with score>t in the prediction results of the training set, and the prediction results of the new class replace the original pseudo-label of the new class. Continue fine-tuning, and after N iterations, repeat the above steps until the detection accuracy of the model on the basic class and the new class meets the requirements, so that the vision and language pre-training model can better adapt to the open vocabulary object detection task while maintaining the original performance.

[0051] In step S104, open semantic object detection is performed on the target image based on the trained vision and language pre-training model.

[0052] The target image may contain objects of known categories and new categories that have not been seen. Based on the trained vision and language pre-training model, the embodiment of the present application can realize accurate and efficient open semantic object detection of objects in the target image. This method can not only identify objects of known categories, but also discover and correctly classify newly appeared unknown categories, thereby greatly expanding the application scope of target detection.

[0053] According to the multi-scale and multi-stage open semantic target detection method proposed in the embodiment of the present application, by performing multi-scale feature alignment on the training image data set, it is possible to obtain rich feature representations at different scales, thereby improving the detection capability of objects of different sizes, generating pseudo labels of new classes based on the alignment results, and training the visual and language pre-training model based on the pseudo labels of the new classes and the real annotations of the classes, which can improve the model's detection capability for objects of different scales and enhance the generalization capability of the model. Open semantic target detection is performed on the target image based on the trained model, which not only improves the detection accuracy of new classes of objects, but also can identify the detailed features of small objects. Thus, the problem of how to better understand local features and improve the accuracy of multi-scale object detection is solved.

[0054] Next, a multi-scale and multi-stage open semantic object detection device proposed according to an embodiment of the present application is described with reference to the accompanying drawings.

[0055] Figure 3 It is a block diagram of a multi-scale and multi-stage open semantic object detection device according to an embodiment of the present application.

[0056] like Figure 3 As shown, the multi-scale and multi-stage open semantic object detection device 10 includes:

[0057] Among them, the acquisition module 100 is used to acquire the training image data set; the alignment module 200 is used to perform multi-scale feature alignment on the training image data set to obtain the vision and language alignment results; the training module 300 is used to generate pseudo-labels of new classes according to the vision and language alignment results, and train the vision and language pre-training model according to the pseudo-labels of the new classes and the real annotations of the classes, wherein the vision and language pre-training model performs multi-stage iterative optimization on the pseudo-labels during the training process; the detection module 400 is used to perform open semantic target detection on the target image based on the trained vision and language pre-training model.

[0058] In one embodiment of the present application, the alignment module 200 is further used to perform multi-scale feature alignment on the training image data set on the target layer of the residual network to obtain an output result corresponding to the target layer; perform region of interest alignment and pooling processing on the output result corresponding to the target layer to obtain multiple groups of bounding box-level multi-scale features; and generate visual and language alignment results based on the multiple groups of bounding box-level multi-scale features.

[0059] In one embodiment of the present application, the target layer includes: a second layer, a third layer and a fourth layer, and the alignment module 200 is further used to perform interest region alignment and pooling processing on the output result of the second layer, and then generate a first group of bounding box level features through the third layer and the fourth layer; perform interest region alignment and pooling processing on the output result of the third layer, and then generate a second group of bounding box level features through the fourth layer; perform interest region alignment and pooling processing on the output result of the fourth layer to generate a third group of bounding box level features; generate multiple groups of bounding box-level multi-scale features according to the first group of bounding box level features, the second group of bounding box level features and the third group of bounding box level features.

[0060] In one embodiment of the present application, the training module 300 is further used to locate the target area in the training image data set to generate a positioning score, and to classify the target area in the training image data set based on the target classifier to generate a classification score; determine the classification result according to the classification score, the positioning score and preset conditions, and use the area with the classification result as a new class as a pseudo label of the new class, wherein the new class is a class that does not appear in the training image data set.

[0061] In one embodiment of the present application, the training module 300 is further used to perform iterative optimization training on the vision and language pre-training model based on the pseudo-labels of the new class and the real labels of the class, and to iteratively update the pseudo-labels of the new class using the confidence scores of the outputs of the region of interest during the training process; if the number of iterations of the iterative optimization training reaches the target number or the detection accuracy of the vision and language pre-training model reaches the target accuracy, the iterative optimization training is stopped, wherein the confidence score of the output of the region of interest is greater than a preset threshold.

[0062] In one embodiment of the present application, the multi-scale and multi-stage open semantic object detection device 10 also includes: an adjustment module for creating multiple position embedding branches in the attention classification layer of the visual language pre-training model before classifying the target area in the training image data set based on the target classifier to generate a classification score; on the basis of fixing the main architecture of the visual language pre-training model, adjusting the position encoding of the multiple position embedding branches to obtain the target classifier.

[0063] It should be noted that the aforementioned explanation of the embodiment of the multi-scale and multi-stage open semantic object detection method is also applicable to the multi-scale and multi-stage open semantic object detection device of this embodiment, and will not be repeated here.

[0064] According to the multi-scale and multi-stage open semantic target detection device proposed in the embodiment of the present application, by performing multi-scale feature alignment on the training image data set, it is possible to obtain rich feature representations at different scales, thereby improving the detection capability of objects of different sizes, generating pseudo labels of new classes based on the alignment results, and training the visual and language pre-training model based on the pseudo labels of the new classes and the real annotations of the classes, which can improve the model's detection capability for objects of different scales and enhance the generalization capability of the model. Open semantic target detection is performed on the target image based on the trained model, which not only improves the detection accuracy of new classes of objects, but also can identify the detailed features of small objects. Thus, the problem of how to better understand local features and improve the accuracy of multi-scale object detection is solved.

[0065] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0066] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0067] When the processor 402 executes the program, the multi-scale and multi-stage open semantic object detection method provided in the above embodiment is implemented.

[0068] Furthermore, the electronic device further comprises:

[0069] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0070] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0071] The memory 401 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0072] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0073] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0074] The processor 402 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0075] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multi-scale and multi-stage open semantic object detection method.

[0076] An embodiment of the present application also provides a computer program product, including: a computer program or instructions, which, when executed, implements the above multi-scale and multi-stage open semantic object detection method.

[0077] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0078] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0079] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0080] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0081] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0082] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A multi-scale and multi-stage open semantic object detection method, characterized in that: The following steps are involved: Get a training image dataset; Performing multi-scale feature alignment on the training image dataset to obtain a vision and language alignment result; Generate a pseudo label of a new class according to the alignment result of the vision and language, and train a vision and language pre-training model according to the pseudo label of the new class and the real annotation of the class, wherein the vision and language pre-training model performs multi-stage iterative optimization on the pseudo label during the training process; Perform open semantic object detection on target images based on the trained vision and language pre-trained model.

2. The multi-scale and multi-stage open semantic object detection method according to claim 1, characterized in that: The performing multi-scale feature alignment on the training image dataset to obtain a vision and language alignment result includes: Performing multi-scale feature alignment on the training image data set on the target layer of the residual network to obtain an output result corresponding to the target layer; Performing region of interest alignment and pooling processing on the output results corresponding to the target layer to obtain multiple groups of bounding box-level multi-scale features; The vision and language alignment results are generated according to the multiple groups of bounding box-level multi-scale features.

3. The multi-scale and multi-stage open semantic object detection method according to claim 2, characterized in that: The target layer includes: a second layer, a third layer and a fourth layer, and the output results corresponding to the target layer are aligned with the regions of interest and pooled to obtain the multi-scale features of the multiple groups of bounding box levels, including: After performing region of interest alignment and pooling processing on the output result of the second layer, a first set of bounding box-level features is generated through the third layer and the fourth layer; After performing region of interest alignment and pooling processing on the output result of the third layer, a second set of bounding box-level features is generated through the fourth layer; Performing region of interest alignment and pooling processing on the output result of the fourth layer to generate a third set of bounding box level features; The multiple groups of bounding box-level multi-scale features are generated according to the first group of bounding box-level features, the second group of bounding box-level features, and the three groups of bounding box-level features.

4. The multi-scale and multi-stage open semantic object detection method according to claim 1, characterized in that: Generating a pseudo label of a new class according to the alignment result of the vision and language includes: Positioning the target area in the training image data set to generate a positioning score, and classifying the target area in the training image data set based on a target classifier to generate a classification score; The classification result is determined according to the classification score, the positioning score and a preset condition, and the area whose classification result is a new class is used as a pseudo label of the new class, wherein the new class is a class that does not appear in the training image data set.

5. The multi-stage open semantic object detection method according to claim 4, characterized in that: The step of training the vision and language pre-training model according to the pseudo-label of the new class and the real annotation of the class includes: Iteratively optimizing the vision and language pre-training model according to the pseudo-label of the new class and the real label of the class, and iteratively updating the pseudo-label of the new class using the confidence score of the output of the region of interest during the training process; If the number of iterations of the iterative optimization training reaches the target number or the detection accuracy of the vision and language pre-training model reaches the target accuracy, the iterative optimization training is stopped, wherein the confidence score of the output of the region of interest is greater than a preset threshold.

6. The multi-scale and multi-stage open semantic object detection method according to claim 4, characterized in that: Before classifying the target area in the training image dataset based on the target classifier to generate a classification score, the method further includes: Creating multiple position embedding branches in the attention classification layer of the vision-language pre-trained model; On the basis of fixing the main structure of the visual language pre-training model, the position encodings of the multiple position embedding branches are adjusted to obtain the target classifier.

7. A multi-scale and multi-stage open semantic object detection device, characterized in that: include: An acquisition module, used to acquire a training image dataset; An alignment module, used for performing multi-scale feature alignment on the training image dataset to obtain a visual and language alignment result; A training module, used for generating a pseudo label of a new class according to the alignment result of the vision and language, and training a vision and language pre-training model according to the pseudo label of the new class and the real annotation of the class, wherein the vision and language pre-training model performs multi-stage iterative optimization on the pseudo label during the training process; The detection module is used to perform open semantic object detection on the target image based on the trained vision and language pre-trained model.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-scale and multi-stage open semantic object detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed, the multi-scale and multi-stage open semantic object detection method described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed, it implements the multi-scale and multi-stage open semantic object detection method described in any one of claims 1-6.

Images