Open-world object detection training method based on dictionary creation and domain adaptation

Abstract

The present invention discloses an open-world target detection training method based on dictionary creation and domain adaptation, relates to the technical field of communication software, solves the problem that the existing open-world target detection is limited to a single scene, firstly introduces an image description data set and a multimodal feature extraction network, aligns the text modality and visual modality features output by the two, and simultaneously introduces a multimodal Transformer network to perform text matching learning and text mask learning of image text; the parameters of the regional visual feature extraction model and the visual mapping text layer learned in the pre-training stage are migrated to the target detection model, and pictures from two domain data sets are input, wherein the source domain pictures participate in the target detection training, and the target domain pictures only participate in the global domain adaptation training, and during the training process, the classifier weights of the detection head are replaced with fixed word vectors of known categories to perform target detection training.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and more specifically to the technical field of open-world target detection training methods based on dictionary creation and domain adaptation. Background Art

[0002] The main purpose of target detection is to detect and locate multiple specific targets in an image. The core issue of target detection is to locate and classify the detected content. It is necessary to determine the shape, size and position of the target in the image according to the influence of different lighting and masking conditions on the detected object, and ensure higher accuracy and shorter detection time. Open world target detection methods are methods for identifying new categories in real complex scenes.

[0003] Traditional target detection methods are mainly limited to fixed category data sets in fixed scenes. The trained classifiers only have the ability to identify labeled categories, but in non-fixed scenes, they do not have the ability to efficiently identify known and unknown categories. It is unrealistic to label all information in all scenes. Traditional methods for detecting new categories only learn the implicit relationship between categories, which seriously ignores the intrinsic connection between text features and visual features, resulting in very low recognition accuracy. At the same time, due to the lack of cross-scene data sets, the model is difficult to generalize to bad weather after training under a fixed data set, which further reduces the ability to detect new categories in bad weather.

[0004] Existing open-world target detection is limited to a single scene, such as indoors or in normal weather. However, in the real open world, the scenes are complex and face various bad weather conditions. Existing methods are trained in normal weather and are difficult to generalize to bad weather conditions, resulting in low recognition accuracy. Although there are related literatures that disclose domain-adaptive target detection models to solve this problem, such as Yuhua Chen, Wen Li, ChristosSakaridis, Dengxin Dai, and Luc Van Gool. 2018.Domain adaptive faster r-cnnfor object detection in the wild. In Proceedings of the IEEE conference onComputer Vision and Pattern Recognition. 3339–3348, they do not have the ability to recognize new classes in the open world. The reason is that simple domain adaptation methods only generalize global scene features and do not increment category features. Summary of the invention

[0005] The purpose of the present invention is to solve the problem in the prior art that open-world target detection is limited to a single scene. In order to solve the above technical problem, the present invention provides an open-world target detection training method based on dictionary creation and domain adaptation.

[0006] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0007] The open-world object detection method based on dictionary creation and domain adaptation includes the following steps:

[0008] Step 1, obtaining image sample data and picture description data corresponding to the image sample data;

[0009] Step 2: Build a regional visual feature extraction model, a visual mapping text model, a BERT word vector extraction model, and a multimodal Transformer network model;

[0010] Step 3: In the pre-training phase, the image sample data in step 1 is input into the regional visual feature extraction model, and the output of the regional visual feature extraction model is used as the input of the visual mapping text model; the image description data in step 1 is input into the BERT word vector extraction model; the output of the BERT word vector extraction model and the output of the visual mapping text model are input into the multimodal Transformer network model after feature alignment for image-text text matching learning and text mask learning;

[0011] Step 4: During the training phase, the parameters of the regional visual feature extraction model and the visual mapping text model learned in the pre-training phase are transferred to the visual feature extraction module. Images from two domain datasets are input, where source domain images participate in target detection training and target domain images only participate in global domain adaptation training. During the training process, the classifier weights of the detection head are replaced with fixed word vectors of known categories for target detection training.

[0012] During the test, different fixed text features were used to replace the classification head of the detection model.

[0013] Technical principle: In the pre-training stage, an "image description" dataset is used, and the image information and text description are respectively input into the visual feature extraction model and the text feature extraction model to further obtain regional visual features and single text features. After the visual features, a visual mapping text model is designed to ensure that the output visual features and text features have the same dimensions. Finally, the distance between the two modal features is calculated, and the similarity of multimodal features is further improved by reducing the specific loss function value. In the pre-training process, the model mainly learns the visual mapping text model and the visual feature extraction model, and realizes the construction of the dictionary by combining the two models.

[0014] During the training phase, FasterR-CNN was introduced as the target detection model. Due to the influence of environmental factors, most of the existing target detection datasets are under normal weather conditions. Therefore, the domain adaptation method was introduced to learn the domain-invariant features of objects in normal weather and bad weather during the training process. At the same time, in the detection phase, we introduced the zero-sample target recognition method by replacing the classification head of the detection model with fixed text features, introducing the visual mapping text model trained in the pre-training phase, and finally training the target detection model.

[0015] In the training phase, the parameters of the regional visual feature extraction model and the visual mapping text model learned in the pre-training phase are transferred to the target detection model. In the training phase, images from two domain datasets are input, where the source domain dataset carries target annotation information, and the target domain dataset only carries domain label information. Only source domain images participate in target detection training, and target domain images only participate in global domain adaptation training. The main purpose of global domain adaptation training is to extract domain-invariant features of the source and target domains.

[0016] During the testing phase, the classification head of the detection model is replaced by different fixed text features to achieve the purpose of identifying different categories of targets in the open world.

[0017] Furthermore, during the pre-training process, the relationship between the BERT word vector and the visual features output by the visual mapping text model is measured by the dot product distance, and its formula is defined as follows:

[0018] ;

[0019] in, The visual features of the regional visual feature extraction model and the visual mapping text model are: It is the word vector feature of a single word extracted by BERT word vector. is the number of image features, is the number of features, for Features and Distance measure between features.

[0020] Furthermore, in the pre-training process, feature alignment mainly includes two parts: text-image alignment and image-text alignment. The specific loss function is as follows:

[0021] ;

[0022] in, is the image input, For text input, is the global image-text matching metric, is a text sequence, is the image sequence, is the matching degree between the image sequence and the corresponding text sequence.

[0023] Furthermore, during the pre-training process, the formulas for image text matching and text mask learning that allow the model to run in a self-supervised manner are defined as follows:

[0024] ;

[0025] in, is the text mask block, is the mean, is the conditional probability function under the model parameters, Generate functions for image classification scores.

[0026] Furthermore, in the global domain adaptation training in the training phase, the distance between the two domains is shortened to extract domain-invariant features in different domains. The formula is defined as follows:

[0027] ;

[0028] in, =0 means the feature comes from the source domain, =1 means the feature comes from the target domain, It is the prediction result of domain features, and the entire loss function is a binary cross entropy loss function.

[0029] Furthermore, during the training phase, the category to which the word vector with the smallest distance belongs is selected as the classification of the feature, and its formula is defined as follows:

[0030] ;

[0031] in, Characterized by is the BERT word vector, represents all-zero background embedding, is the image feature With text features The distance metric, is the image feature Background characteristics The distance metric, is the image feature With different text features The distance metric of .

[0032] Furthermore, different categories of detection tasks are completed by replacing the classification head with BERT word vectors of different categories.

[0033] An open-world object detection training device based on dictionary creation and domain adaptation, comprising a pre-training module, a training module, and a detection module;

[0034] The pre-training module is used to introduce the image description dataset and the multimodal feature extraction network, align the text modality and visual modality features output by the two, and introduce the multimodal Transformer network to perform text matching learning and text mask learning of image text, so that the entire pre-training model can run in a self-supervised manner;

[0035] The training module is used to transfer the parameters of the regional visual feature extraction model and the visual mapping text model learned in the pre-training stage to the target detection model during the training stage. The images from the two domain datasets are input, where the source domain images participate in the target detection training, and the target domain features only participate in the global domain adaptation training. During the training process, the classifier weights of the detection head are replaced with fixed word vectors of known categories to perform target detection training.

[0036] A testing module used to replace the detection model classification head with different fixed text features.

[0037] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0038] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the steps of the above method.

[0039] The beneficial effects of the present invention are as follows:

[0040] (1) The open-world target detection method based on dictionary creation and domain adaptation invented in this patent can identify new types of models in bad weather. It effectively combines domain adaptation with zero-shot recognition, which not only greatly improves the accuracy of zero-shot recognition on bad weather datasets, but also surpasses most of the current methods based on domain adaptation in terms of known categories.

[0041] (2) The open-world target detection method based on dictionary creation and domain adaptation of the present invention constructs an open-world target detection model based on domain adaptation, which has the ability to detect known classes and unknown classes in bad weather. It not only solves the problem of poor generalization ability of single zero-sample detection in the face of weather differences, but also solves the problem that the domain adaptation method cannot detect new classes in the face of knowledge differences. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a pre-trained model diagram of open world object detection in an embodiment;

[0043] Figure 2 is an open world target detection training model diagram in an embodiment. DETAILED DESCRIPTION

[0044] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0045] Example 1

[0046] Reference Figure 1~2 This embodiment provides an open-world object detection training method based on dictionary creation and domain adaptation, which solves the problem that the current open-world object detection is limited to a single scene. At the same time, it has the ability to detect known classes and unknown classes in bad weather, which not only solves the problem of poor generalization ability of single zero-sample detection in the face of weather differences, but also solves the problem that the domain adaptation method cannot detect new classes in the face of knowledge differences.

[0047] (1) Pre-training stage

[0048] In order to establish the explicit relationship between category text and visual features, we introduced the "image description" dataset and the multimodal feature extraction network, and worked to align the text modality and visual modality features output by the two. At the same time, we introduced the multimodal Transformer network for text matching learning and text mask learning of image text. The latter will allow the entire pre-training model to run in a self-supervised manner. The entire pre-training stage mainly learns the visual feature extraction network and the visual mapping text model, such as Figure 1 shown.

[0049] Throughout the pre-training process, we use the dot product distance The formula for measuring the relationship between the BERT word vector and the visual features output by the visual mapping text model is defined as follows:

[0050] ;

[0051] in, The visual features of the regional visual feature extraction model and the visual mapping text model are: It is the word vector feature of a single word extracted by BERT word vector. is the number of image features, is the number of features, for Features and The distance measure between features. The specific loss function is as follows:

[0052] ;

[0053] in, is the image input, For text input, is the global image-text matching metric, is a text sequence, is the image sequence, is the matching degree between the image sequence and the corresponding text sequence.

[0054] The main purpose of image text matching and text mask learning is to allow the model to run in a self-supervised manner. The formula is defined as follows:

[0055] ;

[0056] in, is the text mask block, is the mean, is the conditional probability function under the model parameters, Generate functions for image classification scores.

[0057] (2) Training phase

[0058] In the training phase, we transfer the parameters of the regional visual feature extraction model and the visual mapping text model learned in the pre-training phase to the target detection model. In the training phase, images from two domain datasets are input, where the source domain dataset has target annotation information, and the target domain dataset only has domain label information. In addition, only source domain images participate in target detection training, and target domain features only participate in global domain adaptation training. During the training process, the classifier weights of the detection head are replaced with fixed word vectors of known categories for target detection training. The entire network model is as follows: Figure 2 shown.

[0059] The global domain adaptive method mainly extracts domain-invariant features in different domains by narrowing the distance between the two domains. Its formula is defined as follows:

[0060] ;

[0061] in, =0 means the feature comes from the source domain, =1 means the feature comes from the target domain, It is the prediction result of domain features, and the entire loss function is a binary cross entropy loss function.

[0062] In the final classification process, we compare the distance between the output feature and the BERT word vectors of different words, and select the category to which the word vector with the smallest distance belongs as the classification of the feature. The formula is defined as follows:

[0063]

[0064] in, Characterized by is the BERT word vector, represents all-zero background embedding, is the image feature With text features The distance metric, is the image feature Background characteristics The distance metric, is the image feature With different text features The distance metric of .

[0065] (3) Testing phase

[0066] Different categories of detection tasks can be completed by replacing the classification head with different categories of BERT word vectors. The testing process is the same.

[0067] Experimental tests and results

[0068] The mAP@50 indicator is used to evaluate the model effect. mAP@50 refers to the proportion of the correctly predicted target coordinate position and the coordinate position of the annotation information that overlap by more than 50%. It is also the current mainstream target detection evaluation method. In the pre-training stage, we used the "Picture Description" dataset (COCOCaption dataset) to establish an explicit mapping relationship between features and text features. In the training stage, we introduced the autonomous driving Cityscapes dataset and FoggyCityscapes dataset, with specific categories: Car, Person, Rider, Motor, Train, Truck, Bus, and Bike. Three new classes were divided, namely: Rider and Motor, Train and Truck, Bus and Bike, and the corresponding remaining categories were old classes. We tested our multiple experimental settings on the validation sets of the Cityscapes dataset and the FoggyCityscapes dataset. By analyzing the experimental results, we can conclude that our method is superior to the current mainstream methods in detecting new classes or old classes in bad weather. In Table 1, on the FoggyCityscapes dataset, when the new categories are Rider and Motor, our method has mAP@50 values ​​of 21.97 and 0.51 higher than the state-of-the-art methods for testing new and old categories separately. Other unknown category settings have different degrees of improvement over the state-of-the-art methods. In Table 2, under different new category settings, our mAP@50 values ​​for detecting old categories are 3.05, 1.62, and 0.66 higher than the state-of-the-art methods, respectively, proving the effectiveness of our method.

[0069] At the same time, the inventor compared the technical solution of the present application with the technical solution disclosed in the current relevant literature, and the specific experimental results are shown in Table 1 and Table 2.

[0070] The relevant literature is as follows:

[0071] [1] Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, andAjay Divakaran. 2018. Zero-shot object detection. In Proceedings of theEuropean Conference on Computer Vision (ECCV). 384–400.

[0072] [2] Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc VanGool. 2018. Domain adaptive faster r-cnn for object detection in the wild. InProceedings of the IEEE conference on Computer Vision and PatternRecognition. 3339–3348.

[0073] [3] Jinhong Deng, Wen Li, Yuhua Chen, and Lixin Duan. 2021. Unbiasedmean teacher for cross-domain object detection. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 4091–4101.

[0074] [4] Zhenwei He and Lei Zhang. 2019. Multi-adversarial faster-rcnn forunrestricted object detection. In Proceedings of the IEEE / CVF InternationalConference on Computer Vision. 6668–6677.

[0075] [5] Congcong Li, Dawei Du, Libo Zhang, Longyin Wen, Tiejian Luo,Yanjun Wu, and Pengfei Zhu. 2020. Spatial attention pyramid network forunsupervised domain adaptation. In European Conference on Computer Vision.Springer, 481–497.

[0076] [6] Shafin Rahman, Salman Khan, and Nick Barnes. 2020. Improvedvisual-semantic alignment for zero-shot object detection. In Proceedings ofthe AAAI Conference on Artificial Intelligence, Vol. 34. 11932–11939.

[0077] [7] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko.2019. Strong-weak distribution alignment for adaptive object detection. InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition. 6956–6965.

[0078] [8] Zhiqiang Shen, Harsh Maheshwari, Weichen Yao, and MariosSavvides. 2019. Scl: Towards accurate domain adaptive object detection via gradientdetach based stacked complementary losses. arXiv preprint arXiv:1911.02559(2019).

[0079] [9] Vibashan VS, Vikram Gupta, PoojanOza, Vishwanath A Sindagi, andVishal M Patel. 2021. Mega-cda: Memory guided attention for category-awareunsupervised domain adaptive object detection. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 4516–4526.

[0080]

[10] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-FuChang. 2021. Open-vocabulary object detection using captions. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 14393–14402.

[0081] Table 1. Results of new and old class detection on the FoggyCityscapes dataset under different experimental settings

[0082] Table 2 Results of old class detection on the FoggyCityscapes dataset under different experimental settings

[0083] Example 2

[0084] Reference Figure 1~2 This embodiment provides an open-world target detection device based on dictionary creation and domain adaptation, which solves the problem that the current open-world target detection is limited to a single scene. At the same time, it has the ability to detect known classes and unknown classes in bad weather, which not only solves the problem of poor generalization ability of single zero-sample detection in the face of weather differences, but also solves the problem that the domain adaptation method cannot detect new classes in the face of knowledge differences. The detection device specifically includes: a pre-training module, a training module, and a detection module;

[0085] Pre-training modules

[0086] In order to establish the explicit relationship between category text and visual features, we introduced the "image description" dataset and the multimodal feature extraction network, and worked to align the text modality and visual modality features output by the two. At the same time, we introduced the multimodal Transformer network for text matching learning and text mask learning of image text. The latter will allow the entire pre-training model to run in a self-supervised manner. The entire pre-training stage mainly learns the visual feature extraction network and the visual mapping text model, such as Figure 1 shown.

[0087] Throughout the pre-training process, we use the dot product distance The formula for measuring the relationship between the BERT word vector and the visual features output by the visual mapping text model is defined as follows:

[0088] ;

[0089] in, The visual features of the regional visual feature extraction model and the visual mapping text model are: It is the word vector feature of a single word extracted by BERT word vector. is the number of image features, is the number of features, for Features and The distance measurement between features, the specific loss function is as follows:

[0090] ;

[0091] in, is the image input, For text input, is the global image-text matching metric, is a text sequence, is the image sequence, is the matching degree between the image sequence and the corresponding text sequence.

[0092] The main purpose of image text matching and text mask learning is to allow the model to run in a self-supervised manner. The formula is defined as follows:

[0093] ;

[0094] in, is the text mask block, is the mean, is the conditional probability function under the model parameters, Generate functions for image classification scores.

[0095] Training Module

[0096] In the training phase, we transfer the parameters of the regional visual feature extraction model and the visual mapping text model learned in the pre-training phase to the target detection model. In the training phase, images from two domain datasets are input, where the source domain dataset has target annotation information, and the target domain dataset only has domain label information. In addition, only source domain images participate in target detection training, and target domain features only participate in global domain adaptation training. During the training process, the classifier weights of the detection head are replaced with fixed word vectors of known categories for target detection training. The entire network model is as follows: Figure 2 shown.

[0097] The global domain adaptive method mainly extracts domain-invariant features in different domains by narrowing the distance between the two domains. Its formula is defined as follows:

[0098] ;

[0099] in, =0 means the feature comes from the source domain, =1 means the feature comes from the target domain, It is the prediction result of domain features, and the entire loss function is a binary cross entropy loss function.

[0100] In the final classification process, we compare the distance between the output feature and the BERT word vectors of different words, and select the category to which the word vector with the smallest distance belongs as the classification of the feature. The formula is defined as follows:

[0101]

[0102] in, Characterized by is the BERT word vector, represents all-zero background embedding, is the image feature With text features The distance metric, is the image feature Background characteristics The distance metric, is the image feature With different text features The distance metric of .

[0103] Test Module

[0104] Different categories of detection tasks can be completed by replacing the classification head with different categories of BERT word vectors. The testing process is the same.

[0105] Example 3

[0106] This embodiment also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned open-world target detection method based on dictionary creation and domain adaptation.

[0107] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote control, a touch pad, or a voice control device.

[0108] The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is often used to store the operating system and various application software installed on the computer device, such as the program code for running the abdominal lymph node detection method based on semi-supervised learning, etc. In addition, the memory can also be used to temporarily store various types of data that have been output or are to be output.

[0109] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the abdominal lymph node detection method based on semi-supervised learning.

[0110] Example 4

[0111] This embodiment also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the above-mentioned open-world target detection method based on dictionary creation and domain adaptation.

[0112] The computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor so that the at least one processor performs the steps of the abdominal lymph node detection method based on semi-supervised learning.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the methods described in each embodiment of the present application.

Claims

1. An open-world object detection training method based on dictionary creation and domain adaptation, characterized in that: The following steps are involved: Step 1, obtaining image sample data and picture description data corresponding to the image sample data; Step 2: Build a regional visual feature extraction model, a visual mapping text model, a BERT word vector extraction model, and a multimodal Transformer network model; Step 3: In the pre-training phase, the image sample data in step 1 is input into the regional visual feature extraction model, and the output of the regional visual feature extraction model is used as the input of the visual mapping text model; the image description data in step 1 is input into the BERT word vector extraction model; the output of the BERT word vector extraction model and the output of the visual mapping text model are input into the multimodal Transformer network model after feature alignment for image-text text matching learning and text mask learning; Step 4: In the training phase, the parameters of the regional visual feature extraction model and the visual mapping text model learned in the pre-training phase are transferred to the visual feature extraction module. Images from two domain datasets are input, where source domain images participate in target detection training, and target domain images only participate in global domain adaptation training. During the training process, the classifier weights of the detection head are replaced with fixed word vectors of known categories for target detection training. During pre-training, the formulas for image text matching and text mask learning that allow the model to run in a self-supervised manner are defined as follows: ； in, is the text mask block, is the mean, is the conditional probability function under the model parameters, Generate functions for image classification scores; In the global domain adaptation training of the training phase, the distance between the two domains is shortened to extract domain-invariant features in different domains. The formula is defined as follows: ； in, =0 means the feature comes from the source domain, =1 means the feature comes from the target domain, It is the prediction result of domain features, and the entire loss function is a binary cross entropy loss function.

2. The open-world object detection training method based on dictionary creation and domain adaptation according to claim 1, characterized in that: During the pre-training process, the relationship between the BERT word vector and the visual features output by the visual mapping text model is measured by the dot product distance, and its formula is defined as follows: ； in The visual features of the regional visual feature extraction model and the visual mapping text model are: It is the word vector feature of a single word extracted by BERT word vector. is the number of image features, is the number of features, for Features and Distance measure between features.

3. The open-world object detection training method based on dictionary creation and domain adaptation according to claim 1, characterized in that: During the pre-training process, feature alignment mainly includes two parts: text-image alignment and image-text alignment. The specific loss function is as follows: ； in, is the image input, For text input, is the global image-text matching metric, is a text sequence, is the image sequence, is the matching degree between the image sequence and the corresponding text sequence.

4. The open-world object detection training method based on dictionary creation and domain adaptation according to claim 1, characterized in that: During the training phase, the category to which the word vector with the smallest distance belongs is selected as the classification of the feature. The formula is defined as follows: ； in, Characterized by is the BERT word vector, represents all-zero background embedding, is the image feature With text features The distance metric, is the image feature Background characteristics The distance metric, is the image feature With different text features The distance metric of .

5. The open-world object detection training method based on dictionary creation and domain adaptation according to claim 1, characterized in that: Different categories of detection tasks are completed by replacing the classification head with BERT word vectors of different categories.

6. An open-world object detection training device based on dictionary creation and domain adaptation, characterized in that: Including pre-training module and training module; The pre-training module is used to introduce the image description dataset and the multimodal feature extraction network, align the text modality and visual modality features output by the two, and introduce the multimodal Transformer network to perform text matching learning and text mask learning of image text, so that the entire pre-training model can run in a self-supervised manner; The training module is used to transfer the parameters of the regional visual feature extraction model and the visual mapping text model learned in the pre-training stage to the target detection model during the training stage. The images from the two domain datasets are input, where the source domain images participate in the target detection training, and the target domain features only participate in the global domain adaptation training. During the training process, the classifier weights of the detection head are replaced with fixed word vectors of known categories to perform target detection training. During pre-training, the formulas for image text matching and text mask learning that allow the model to run in a self-supervised manner are defined as follows: ； During pre-training, the formulas for image text matching and text mask learning that allow the model to run in a self-supervised manner are defined as follows: ； in, is the text mask block, is the mean, is the conditional probability function under the model parameters, Generate functions for image classification scores; In the global domain adaptation training of the training phase, the distance between the two domains is shortened to extract domain-invariant features in different domains. The formula is defined as follows: ； in, =0 means the feature comes from the source domain, =1 means the feature comes from the target domain, It is the prediction result of domain features, and the entire loss function is a binary cross entropy loss function.

7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 5.

Images