Open vocabulary object detection method and apparatus for aerospace

Abstract

The embodiment of the present disclosure provides a kind of aerospace field-oriented open vocabulary target detection method and device, it is related to target detection technical field.The method comprises: the image to be detected and the description text corresponding to the image to be detected are input into the detection model pre-trained, obtain at least one mark frame and the target prediction category of the object in each mark frame of detection model output, target prediction category is the entity noun included in description text.Detection model is obtained according to pre-constructed sample dataset training, sample dataset includes self-built dataset;For each sample image in self-built dataset, the real category of sample object is used as training label, real category includes entity noun and extension word, by jointly as training label with key word and extension word, it can directly incorporate domain expertise into model training, enhance the semantic understanding ability of detection model in aerospace field, effectively improve the generalization ability and accuracy of detection model identification unseen class.

Description

Technical Field

[0001] This disclosure relates to the field of target detection technology, and more specifically, to an open vocabulary target detection method and apparatus for the aerospace field. Background Technology

[0002] Open-vocabulary object detection (OVOD) has received widespread attention in recent years as an important research direction in the field of computer vision. OVOD proposes to detect unseen object categories by leveraging the generalization capabilities of the model; that is, OVOD can identify and locate categories that do not appear in the training data.

[0003] Most methods leverage general-domain data by pre-training an open-vocabulary object detection model on large datasets in the general domain and then fine-tuning it on datasets in the vertical domain. However, data distribution in vertical domains is diverse and complex, especially in the aerospace field where entity concepts are highly specialized, complex, and cutting-edge, resulting in significant differences in data distribution between the general domain and the aerospace field.

[0004] Currently, open vocabulary models used in the aerospace field typically rely on generalization ability learned from large amounts of general data. They perform poorly in detecting unseen categories and are prone to misclassifying new categories as the most similar known categories. Summary of the Invention

[0005] This disclosure provides an open vocabulary target detection method and apparatus for the aerospace field, which can solve the above-mentioned problems of the prior art. The technical solution is as follows:

[0006] According to one aspect of the present disclosure, an open vocabulary target detection method for the aerospace field is provided, comprising:

[0007] Input the image to be detected and the corresponding descriptive text into the pre-trained detection model to obtain at least one bounding box and the target predicted category of the object in each bounding box. The target predicted category is the entity noun included in the descriptive text.

[0008] The detection model is trained based on a pre-built sample dataset, which includes a self-built dataset.

[0009] The training data in the self-built dataset is generated in the following ways:

[0010] Obtain news in the aerospace field, extract multiple sample images from the news, and obtain corresponding descriptive text for each sample image;

[0011] For each description text, entity nouns are extracted from the description text, and expanded words of entity nouns are determined, including hypernyms and abbreviations;

[0012] For each sample image, the sample image is used as a training sample. A bounding box is labeled for each sample object in each sample image. The bounding box and the true category of the sample object in the bounding box are used as training labels. The true categories include entity nouns and extended words.

[0013] According to another aspect of the present disclosure, an open vocabulary target detection device for the aerospace field is provided, the device comprising:

[0014] An open vocabulary object detection module is used to input the image to be detected and the corresponding descriptive text of the image to be detected into a pre-trained detection model, and obtain at least one bounding box and the target prediction category of the object in each bounding box. The target prediction category is the entity noun included in the descriptive text.

[0015] The detection model is trained based on a pre-built sample dataset, which includes a self-built dataset.

[0016] The training data in the self-built dataset is generated in the following ways:

[0017] Obtain news in the aerospace field, extract multiple sample images from the news, and obtain corresponding descriptive text for each sample image;

[0018] For each description text, entity nouns are extracted from the description text, and expanded words of entity nouns are determined, including hypernyms and abbreviations;

[0019] For each sample image, the sample image is used as a training sample. A bounding box is labeled for each sample object in each sample image. The bounding box and the true category of the sample object in the bounding box are used as training labels. The true categories include entity nouns and extended words.

[0020] According to another aspect of the present disclosure, an electronic device is provided, the electronic device including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the above-described method.

[0021] According to another aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described method.

[0022] According to one aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method.

[0023] The beneficial effects of the technical solutions provided in this disclosure are:

[0024] The detection model for open vocabulary detection in the aerospace field provided in this disclosure is trained based on a pre-built sample dataset. This sample dataset includes a self-built dataset, which uses news from the aerospace field as its data source. Compared to existing technologies that use large amounts of general data as training data, this approach better meets the aerospace field's requirements for specialization, complexity, and cutting-edge technology. Furthermore, for each sample image in the self-built dataset, the training labels include keywords indicating the sample object and extended words. By using both keywords and extended words as training labels, domain-specific knowledge is directly integrated into the model training, improving the model's understanding of domain-specific concepts, enhancing the detection model's semantic understanding ability in the aerospace field, and effectively improving the detection model's generalization ability and accuracy in identifying unseen categories. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below.

[0026] Figure 1 A flowchart illustrating an open vocabulary target detection method for the aerospace field provided in this embodiment of the disclosure;

[0027] Figure 2 A flowchart illustrating a method for generating sample data in a self-built dataset, as provided in an embodiment of this disclosure;

[0028] Figure 3 A schematic diagram illustrating an entity noun expansion method provided in an embodiment of this disclosure;

[0029] Figure 4 A schematic diagram illustrating a cross-modal matching method provided in an embodiment of this disclosure;

[0030] Figure 5 This is a schematic diagram of the structure of a detection model provided in an embodiment of the present disclosure;

[0031] Figure 6 This is a schematic diagram of the appearance of a deformable aircraft provided in an embodiment of the present disclosure;

[0032] Figure 7 This is a schematic diagram of the exterior of a conventional aircraft provided in an embodiment of the present disclosure;

[0033] Figure 8 A schematic diagram of an open vocabulary target detection device for the aerospace field provided in this embodiment of the disclosure;

[0034] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0035] The embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this disclosure, and do not constitute a limitation on the technical solutions of the embodiments of this disclosure.

[0036] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this disclosure mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element are connected through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.”

[0037] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0038] The following description of several exemplary embodiments illustrates the technical solutions of this disclosure and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0039] The relevant technologies involved in this application are described below:

[0040] In the field of object detection research, a large amount of work has been dedicated to building image datasets, but these datasets still have limitations in meeting the needs of large-scale category vocabulary. As one of the most representative datasets in the general domain, COCO contains 123,000 images and 80 categories, and is widely used in computer vision research.

[0041] To meet the pre-training needs of deep learning models, researchers have developed large-scale datasets such as Objects365 and Open Images. Objects365 provides labeled data for 638,000 images covering 365 categories, while Open Images contains 1.51 million images and 600 categories. Although these datasets represent significant progress in sample size, the number of categories is still insufficient to support the training needs of category-generalized detection models.

[0042] The LVIS dataset has achieved significant breakthroughs in this area, containing 120,000 images and 1203 categories, providing rich labeled data for object detection and instance segmentation tasks. In fine-grained image understanding, the ADE20K dataset, through pixel-level annotation of 25,000 images, achieves accurate segmentation of object instances, "fillers," and parts. This dataset contains approximately 3000 named objects, filled regions, and parts, providing crucial support for scene understanding research. The establishment of these datasets lays the foundation for open-vocabulary object detection research, but further expansion of category coverage and annotation granularity is still needed to meet the growing demands of model training.

[0043] In the vertical domains, the iNaturalist dataset contains nearly 900,000 images with bounding box annotations for approximately 5,000 plant and animal species. The Recipe1M+ dataset contains over 1 million cooking recipes and 13 million food images. For a more granular vertical domain, the SPARK dataset provides approximately 5,000 satellite component images, covering bounding boxes and status annotations for targets such as solar panels and antennas. Its combination of component functional text descriptions supports the detection of unknown anomalies, and it includes 2,806 aerial images collected from different sensors and platforms. The DOTA dataset contains images approximately 4000×4000 pixels each and includes objects of various scales, orientations, and shapes. DOTA images are annotated by aerial image interpretation experts using 15 common object categories, and fully annotated DOTA images contain 188,000 instances. The DIOR dataset is a large-scale benchmark dataset for object detection in optical remote sensing images, containing 23,000 images and 190,000 object instances, covering 20 object categories.

[0044] Open-vocabulary object detection (OVOD) has garnered significant attention in computer vision in recent years. Traditional object detection methods typically rely on fixed class labels, requiring predefined classes and supervised learning on the dataset during training. However, this fixed-vocabulary-based approach exhibits considerable limitations when faced with the ever-changing classes in real-world scenarios. To address this issue, OVOD addresses the need to detect unseen class objects by generalizing the model's capabilities—that is, to identify and locate classes not present in the training data. This research direction is not only challenging but also holds significant practical value.

[0045] Currently, although some progress has been made in target detection and recognition technology for open vocabulary in vertical domains, existing methods still face several key challenges in target detection and recognition in the field of aerospace science and technology news that urgently need to be addressed:

[0046] 1. Data distribution in vertical domains is diverse and complex, especially in the aerospace field. Due to the non-public nature of datasets in the aerospace field, the variability of target categories (such as fine-grained targets like meteorites), and the cutting-edge nature of the field (such as newly released spacecraft), there are significant differences in data distribution between general and vertical domains.

[0047] The conventional training method, which involves pre-training the detection model on a large dataset in the general domain and then fine-tuning it on a dataset in the vertical domain, results in a detection model with poor generalization ability, poor detection performance for unseen categories, and a tendency to misclassify new categories as the most similar known categories.

[0048] 2. The entity concepts in aerospace technology news are highly specialized, complex, and cutting-edge, with numerous entity categories and a tendency to generate new, unseen categories. Current methods of entity category labeling cannot meet the demands for fine-grained and real-time performance in entity classification.

[0049] 3. The aerospace field is characterized by high specialization, semantic complexity, and heterogeneous data. Currently used cross-modal matching methods cannot perform deep semantic representation, resulting in poor image-text alignment.

[0050] Based on this, the present disclosure provides an open vocabulary target detection method for the aerospace field, which to a certain extent solves the technical problem that the open vocabulary target detection technology in the above-mentioned related technologies has poor detection effect in the aerospace field.

[0051] The following description of several exemplary embodiments illustrates the technical solutions of this disclosure and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0052] It is understood that in the open vocabulary target detection method for the aerospace field provided in this disclosure embodiment, any method step can be executed by an electronic device and / or a server, and all steps in the method can be executed independently by an electronic device or a server, or jointly by an electronic device and a server.

[0053] The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. Electronic devices can be smartphones, tablets, laptops, desktop computers, smart voice interaction devices (such as smart speakers), wearable electronic devices (such as smartwatches), in-vehicle terminals, smart home appliances (such as smart TVs), AR / VR devices, etc., but are not limited to these.

[0054] The embodiments of this disclosure are subsequently described using electronic devices as the execution subject; however, this does not constitute a limitation on the embodiments of this disclosure. In addition to open vocabulary object detection for the aerospace field, the methods provided in the embodiments of this disclosure can be adjusted to cover different domains of news articles used to construct self-built datasets, making them applicable to open vocabulary object detection for other vertical fields.

[0055] Figure 1 This is a flowchart illustrating an open vocabulary target detection method for the aerospace field provided in an embodiment of this disclosure, as shown below. Figure 1 As shown in the embodiments of this disclosure, the open vocabulary target detection method for the aerospace field includes the following steps:

[0056] Step S101: Input the image to be detected and the corresponding descriptive text of the image to be detected into the pre-trained detection model to obtain at least one bounding box and the target predicted category of the object in each bounding box. The target predicted category is the entity noun included in the descriptive text.

[0057] The descriptive text describes the content displayed in the image to be detected, and entity nouns refer to nouns or noun phrases that represent specific things.

[0058] Specifically, the open vocabulary target detection method for the aerospace field provided in this disclosure is used to perform open vocabulary target detection on the image to be detected using a pre-trained detection model (i.e., open vocabulary detection model). The image to be detected should include at least one object belonging to the aerospace field (i.e., the object to be detected).

[0059] It is understood that, in the open vocabulary target detection in the embodiments of this disclosure, the input of the pre-trained detection model is the image to be detected and the descriptive text corresponding to the image to be detected. The descriptive text is used to describe the content displayed in the image to be detected, and the source and form of the image to be detected and the descriptive text can be determined according to actual needs.

[0060] For example, when the image to be detected is an illustration in a scientific paper in the field of aerospace, the descriptive text can be a description of the illustration in the paper. When the image to be detected is a picture in a news article in the field of aerospace, the descriptive text can be a description of the picture in the news article. The descriptive text can also be a task instruction to achieve specific detection requirements for the image to be detected.

[0061] The detection model inputs the image to be detected and the descriptive text. The detection model maps the visual information contained in the image to be detected and the semantic information contained in the descriptive text to the same embedding space (i.e., vector space) for cross-modal matching. It matches the objects identified in the image to be detected with the entity nouns in the descriptive text. For each object, it determines the entity nouns that are semantically related to the object as the predicted category. The output is a bounding box used to label the location of the object in the image, and the target predicted category of the object in each bounding box.

[0062] Understandably, unlike general object detection, which inputs the image to be detected into the model and outputs the target category, with the target category derived from the known category set labeled in the sample set during model training, in this embodiment, the detection model performs cross-modal matching of visual and semantic information to enable similarity calculation in the same embedding space, thereby determining the semantic information that best matches the object in the image and determining the predicted category of the object. This allows the predicted category to be determined from the entity nouns included in the text input to the model, without being constrained by the known categories labeled in the limited training set, thus achieving the detection of unseen categories.

[0063] Taking any image to be detected as an example, the steps of the open vocabulary target detection method for the aerospace field provided in this disclosure embodiment will be described in detail.

[0064] In step S101, any image to be detected and the corresponding descriptive text are input into a pre-trained detection model. The detection model identifies and locates the objects contained in the image to be detected, determines each object and obtains a bounding box to label the position of the object in the image, and determines the target prediction category corresponding to the object in the bounding box from the entity nouns in the descriptive text. The detection model outputs at least one bounding box and the target prediction category of the object in each bounding box.

[0065] It is understood that, in this embodiment of the disclosure, the output format of the target predicted category for each object can be determined according to actual needs. For example, only the predicted category with the highest predicted category probability score (confidence) can be output as the target predicted category, or multiple predicted categories and their category probability scores can be output as the target predicted category. Before applying the detection model, it is necessary to train the detection model using a pre-built sample dataset. The sample dataset includes sample images and the corresponding training labels for the sample images. The specific network model structure and training method can be determined according to actual needs.

[0066] In this embodiment of the disclosure, the sample dataset used for training includes not only publicly available general data for model training, but also a self-built dataset. The self-built dataset refers to a sample dataset constructed based on publicly available scientific and technological news in the aerospace field. Accordingly, Figure 2 This is a flowchart illustrating a method for generating sample data from a self-built dataset, as provided in an embodiment of this disclosure. Figure 2 As shown, the sample data in the self-built dataset in this embodiment of the disclosure is generated in the following way:

[0067] Step S201: Obtain news in the aerospace field, and extract multiple sample images and corresponding descriptive text for each sample image from the news. Each sample image includes at least one sample object.

[0068] Specifically, using news in the aerospace field as the basis for data sources, in step S201, multiple sample images and corresponding descriptive text for each sample image are obtained from the news in the aerospace field.

[0069] Understandably, news in the aerospace field typically includes text and accompanying images. Part of the text is used to describe the accompanying images. That is, any image is selected from aerospace news as a sample image, and the corresponding description of that image is obtained from the news text as the descriptive text for that sample image.

[0070] It should be noted that the images in news reports in the aerospace field can be either conventionally taken images or remote sensing images, etc.

[0071] Alternatively, a web crawler based on structural similarity can be used to obtain sample images. Based on the cover image of the news article, it can be determined whether the news is related to the aerospace field. If the news is related to the aerospace field, the web crawler can be used to obtain the internal images (i.e., internal images) of the news article, and the structural similarity between the internal images and the cover image can be calculated. Internal images with a structural similarity greater than a preset similarity threshold can be selected as sample images. This can effectively solve the problem of mismatch between the cover image and multiple image numbers in the text when commonly used web crawlers obtain images.

[0072] For example, the "news" section of the journal *NATURE* is chosen as the news data source, collecting cover images and internal images of science and technology news articles in the aerospace field. The OpenCVSharp library is used to calculate the Structural Similarity Index (SSIM) between the cover image and the internal images. SSIM is an index that measures the similarity between two images, ranging from -1 to 1, with values ​​closer to 1 indicating greater similarity.

[0073] SSIM measures the similarity between two images by considering not only the differences in pixel values ​​but also the characteristics of the Human Visual System (HVS). It can more accurately reflect the structural similarity between images by calculating the similarity in three dimensions: brightness, contrast, and structure, to obtain an overall similarity index.

[0074] The calculation formula is as follows:

[0075]

[0076] in: l(x,y) It is brightness similarity; c(x,y) It is contrast similarity; s(x,y) It is structural similarity; α , β , γ These are parameters that control the weights of the three similarity criteria; they are typically set to a value of [value missing]. α=β=γ= 1.

[0077] In the disclosed embodiment, x is the news cover image and y is the inner image. The preset similarity threshold corresponding to SSIM(x,y) is 0.65. Inner images with a similarity greater than 0.65 are considered similar to the cover image and are used as sample images.

[0078] Step S202: For each description text, extract entity nouns from the description text and determine the extended words of the entity nouns, including hypernyms and abbreviations.

[0079] Specifically, after obtaining the descriptive text corresponding to each sample, in step S202, entity names are extracted from each descriptive text. It can be understood that for extracting entity names from descriptive text corresponding to the aerospace field, the obtained entity names are generally scientific and technological terms related to the aerospace field.

[0080] Understandably, the specific methods for extracting entity names from descriptive text can be determined according to actual needs, including but not limited to methods based on matching and extraction using a pre-built database of aerospace technical terms, entity extraction methods based on semantic features, and keyword extraction methods based on models.

[0081] For example, large language models (LLMs), such as DeepSeek or Kimi, can be used to extract entity nouns from descriptive text.

[0082] Based on the characteristics of news headlines and image descriptions, a specialized prompt word template is designed to guide a large language model to accurately extract entity nouns from the descriptive text and output them in JSON format, using prompt engineering. For example, a prompt word target template could be designed as: "You are an expert in text entity recognition, and you need to extract noun entities from a given sentence." The output should be in JSON format, such as {"entity_text": " Space Launch System "}. Note: 1. Each line of output must be a valid JSON string; 2. If no entity is found, output "No entity found".

[0083] After obtaining the entity nouns in the description text, this embodiment of the disclosure further expands the entity nouns by determining the extended words of the entity nouns. The extended words include superordinate terms and abbreviations. That is, in this embodiment of the disclosure, for any entity noun, the entity noun is expanded from two perspectives: expansion with superordinate terms and expansion with abbreviations.

[0084] It should be noted that when expanding entity names, not all entity names can be expanded. Some entity names have superordinate terms and / or abbreviations. In this embodiment of the disclosure, only entity names that can be expanded are expanded. Moreover, the entity names that can be expanded and the superordinate terms and abbreviations obtained after expansion are all scientific and technical terms, which refer to the standardized terms in the field of science and technology.

[0085] Understandably, a hypernym refers to a broader category, containing multiple hyponyms. For example, the hyponyms of the hypernym "aerospace vehicle" include "aircraft" and "rocket." By identifying and expanding hypernym relationships, specific technical terms can be associated with broader categories. When obtaining text embedding vectors of words based on descriptive text, hypernym relationships can enrich the semantic expression of the text. Hypernym expansion not only helps improve the semantic richness of the text but also provides broader contextual information in multimodal analysis.

[0086] In this embodiment of the disclosure, the expansion of superordinate terms is performed based on a pre-constructed engineering and technical terminology system. The engineering and technical terminology system includes the correspondence between superordinate terms and subordinate terms. If any entity noun is determined to be a subordinate term in the engineering and technical terminology system, the superordinate term corresponding to the subordinate term is obtained.

[0087] The Engineering and Technology Terminology System refers to a systematic collection of engineering and technology terms constructed and maintained by authoritative institutions or professional organizations. It is a database or knowledge base containing words, definitions, synonyms, near-synonyms, antonyms, hyponyms, and relationships between words. Examples include the terminology collections developed by the International Organization for Standardization (ISO) and the Standardization Administration of the People's Republic of China (SAC), or the Engineering and Technology Terminology System developed by the Institute of Scientific and Technical Information of China (ISTIC).

[0088] Understandably, many technical terms in the aerospace field have abbreviations, which are very common in practical applications. For example, "Space Launch System" is usually abbreviated as "SLS". By identifying and expanding these abbreviations, we can ensure that important technical details are not overlooked during text processing, helping models better understand the relationship between abbreviations and full names, thereby preserving more semantic information in text embeddings.

[0089] In this embodiment of the disclosure, in terms of text data representation, the target description in the image is obtained from a global perspective, a knowledge organization system is introduced to expand the vocabulary, and a text expansion strategy is designed for professional terms and abbreviations, which significantly improves the coverage of professional abbreviations and establishes a reasonable and effective multimodal knowledge representation strategy.

[0090] In this embodiment of the disclosure, a large language model can be applied to determine whether an entity noun has an abbreviation and to obtain the corresponding abbreviation.

[0091] Figure 3 This is a schematic diagram of an entity noun expansion method provided in an embodiment of this disclosure, such as... Figure 3 As shown, for example, for any entity noun, the CITIC Engineering and Technology Terminology System can be used to expand the entity noun to obtain its superordinate terms, and LLM can be used to expand the entity noun to obtain its abbreviations.

[0092] Step S203: For each sample image, use the sample image as a training sample, annotate the bounding box of each sample object in each sample image, and use the bounding box and the true category of the sample object in the bounding box as training labels. The true category includes entity nouns and extended words.

[0093] Specifically, for each sample image, the sample image is used as a training sample. A bounding box indicating the location of each sample object in the sample image is annotated. The bounding boxes and the true categories of the sample objects within them are used as training labels to annotate the sample images, thus obtaining a self-built dataset. The true categories include entity nouns and extended words.

[0094] Understandably, targets in aerospace science and technology news images are usually quite conspicuous, occupying a large portion of the image, and are not small targets. Although the targets are large, the form of aerospace launch systems may be quite complex, such as different types of rockets, different launch stages (vertical position, ignition and takeoff, different angles during flight), different backgrounds (daytime, nighttime, different weather conditions), and situations where multiple targets are included in the same image.

[0095] After obtaining the first information corresponding to each sample image and its corresponding descriptive text, manual annotation is performed using the "labelimg" software. The sample image may include multiple sample objects, each sample object corresponds to a bounding box, and each bounding box can be labeled with one or more training labels (entity nouns and extended words).

[0096] The detection model for open vocabulary detection in the aerospace field provided in this disclosure is trained based on a pre-built sample dataset. This sample dataset includes a self-built dataset, which uses news from the aerospace field as its data source. Compared to existing technologies that use large amounts of general data as training data, this approach better meets the aerospace field's requirements for specialization, complexity, and cutting-edge technology. Furthermore, for each sample image in the self-built dataset, the training labels include keywords indicating the sample object and extended words. By using both keywords and extended words as training labels, domain-specific knowledge is directly integrated into the model training, improving the model's understanding of domain-specific concepts, enhancing the detection model's semantic understanding ability in the aerospace field, and effectively improving the detection model's generalization ability and accuracy in identifying unseen categories.

[0097] In one possible implementation, the sample dataset also includes a public dataset;

[0098] The detection model is trained in the following way:

[0099] The detection model was trained at least once using both a self-built dataset and a publicly available dataset.

[0100] For each training round, the first sample set for this training round is determined. The first sample set includes at least one self-built sample and at least one public sample. The ratio of self-built samples to public samples is positively correlated with the training round. Self-built samples are from self-built datasets, and public samples are from public datasets.

[0101] Specifically, in this embodiment, the sample dataset used during the training of the detection model also includes a public dataset, which refers to publicly available, general-purpose data used for model training. The pre-trained detection model is trained at least once using both the self-built dataset and the public dataset, employing a progressive hybrid training approach for few-shot adaptive learning.

[0102] During the training of the detection model, for each round of training, the first sample set to be used in this round of training is determined, and the ratio of self-built samples and public samples in the first sample set is adjusted according to the current training round. Self-built samples come from self-built datasets, and public samples come from public datasets.

[0103] It is understandable that as the number of training rounds increases, the proportion of self-built samples in the first sample set gradually increases. That is, the first sample set includes at least one self-built sample and at least one public sample, and the ratio of self-built samples to public samples is positively correlated with the number of training rounds.

[0104] For example, the public datasets include the FAIR1M and DIOR remote sensing datasets. In training epochs 1-50, the proportion of publicly available samples in the first dataset was 0.8, and the proportion of self-built samples was 0.2; in training epochs 51-100, the proportion of publicly available samples in the first dataset was 0.5, and the proportion of self-built samples was 0.5; in training epochs 101-150, the proportion of publicly available samples in the first dataset was 0.2, and the proportion of self-built samples was 0.8.

[0105] It is understood that the ratio of public samples to self-built samples in each round above is only a specific example. In actual application of the technical solution provided in this disclosure embodiment, the total number of training rounds, the number of samples in the first sample set in each round, and the ratio of self-built samples to public samples can be determined according to actual needs.

[0106] In the technical solution provided in this disclosure, during the training of the detection model, the proportion of self-built samples is gradually increased as the training rounds increase. This utilizes both the breadth of publicly available datasets and the professionalism of self-built datasets. While focusing on the aerospace field, it avoids the bias or insufficient information problems that may be caused by a single data source, effectively improving the generalization ability of the detection model and the accuracy of the detection model when applied in the aerospace field.

[0107] In one possible implementation, each training round includes the following steps:

[0108] For each sample image in the first sample set, identify and locate the sample objects in the sample image, determine the predicted bounding box for each sample object, and obtain the image embedding vector of the candidate region corresponding to each predicted bounding box in the sample image.

[0109] Encode the descriptive text corresponding to each sample image to obtain the text embedding vector of the category label corresponding to each real category in the descriptive text;

[0110] All candidate regions and all category labels are matched one by one to obtain multiple matching pairs. The similarity between the image embedding vector and the text embedding vector corresponding to each matching pair is calculated to obtain a similarity matrix. Each element in the similarity matrix is ​​the similarity of the corresponding matching pair.

[0111] Using the Hungarian algorithm, predicted matching pairs are determined from the similarity matrix, and the similarity matrix is ​​mapped to the probability range to obtain the predicted confidence matrix. The predicted matching pairs include candidate regions and predicted labels, and the predicted labels are the category labels that best match the candidate regions.

[0112] For each sample image, based on the prediction confidence matrix, the difference between the predicted probability distribution of each candidate region in the sample image and the probability distribution of the corresponding true label is calculated, and the cross-entropy loss is obtained.

[0113] For each sample image, the difference between the predicted category and the true category of each candidate region in the sample image is calculated based on the similarity corresponding to each predicted match, and the classification loss is obtained.

[0114] The sum of the classification loss and cross-entropy loss for each sample image in the first sample set is taken as the total loss;

[0115] The parameters of the detection model are adjusted with the goal of minimizing the total loss.

[0116] Specifically, in this embodiment of the disclosure, during each round of training, the detection model first determines the first sample set to be used in this round of training and inputs the first sample set into the detection model. To better describe the training method of the detection model provided in this embodiment, the detailed steps of the detection model's data processing are explained with the specific structure of the detection model as an aid. The detection model is divided into multiple modules according to the functions implemented by each module, including: a region localization module, an embedding representation module, a cross-modal matching module, and a classification head. The embedding representation module includes: a text encoder and an image encoder.

[0117] During each round of model training, each sample image in the first sample set is input into the region localization module. The region localization module identifies and locates the sample objects in the sample images, determines the predicted bounding box for each sample object, and obtains the image embedding vector of the candidate region corresponding to each predicted bounding box in the sample image through the image encoder.

[0118] Construct a candidate matrix R using all the image embedding vectors corresponding to the sample image. The candidate matrix R includes the image embedding vector r1-rm for each candidate region (m is an integer representing the number of candidate regions).

[0119] The descriptive text and the true categories in the training labels are input into the text encoder. The text encoder encodes the descriptive text corresponding to each sample image to obtain the text embedding vector of the category label corresponding to each true category in the descriptive text.

[0120] Construct a text embedding matrix W using all the text embedding vectors corresponding to the sample image. The text embedding matrix W includes the text embedding vector w1-wn for each category label (n is an integer representing the number of training labels).

[0121] It should be noted that the core of the detection model is to achieve cross-modal understanding and matching. Therefore, the obtained text embedding vector and image embedding vector need to be in the same embedding space so that the model can directly compare the semantic similarity of text and image with a unified metric, thereby achieving efficient matching between text and image.

[0122] Based on the above requirements, in this embodiment of the disclosure, by comparing and learning pre-trained matching image encoders and text encoders, the image embedding vectors generated by the image encoder and the text embedding vectors generated by the text encoder can be aligned to a shared vector space. The type of encoder (model) and training method of the specific application can be determined according to actual needs.

[0123] For example, the Contrastive Language-Image Pre-training (CLIP) model can be applied for embedding representation. The descriptive text is taken as input to the text encoder of the CLIP model to obtain the text embedding vector of the descriptive text. Since the CLIP text encoder generates an embedding representation for each token in the sentence, the embedding representation of the token corresponding to the entity noun is obtained as the text embedding vector of the entity noun. The CLIP model is a powerful multimodal model that not only preserves the semantic information of the text but also achieves seamless integration of text and images in multimodal analysis.

[0124] The above methods can significantly improve the accuracy and robustness of models when processing complex text and image data in the aerospace technology field, effectively expand and enrich text data in the aerospace technology field, and provide stronger semantic support for subsequent multimodal analysis and target detection tasks.

[0125] It is understandable that the method for obtaining the text embedding vector of extended words is the same as the method for obtaining the text embedding vector of entity nouns above. The difference is that the entity nouns in the descriptive text are replaced with the corresponding extended words. The method for obtaining them is described above and will not be repeated here.

[0126] The text embedding vector and image embedding vector are input into the cross-modal matching module. Through the cross-modal matching layer, all candidate regions and all category labels are matched one by one to obtain multiple matching pairs. A matching matrix is ​​constructed, and each element in the matching matrix indicates a matching pair composed of image embedding vector and text embedding vector.

[0127] In other words, in cross-modal matching, unlike conventional methods that perform one-to-one or one-to-many matching between candidate regions and category labels, the embodiments of this disclosure use a virtual node replication method to transform the original one-to-many matching problem into a one-to-one matching problem.

[0128] For each image, the candidate matrix R and the text embedding matrix W are obtained by exhaustive one-to-one matching to obtain the one-to-one correspondence between all candidate regions and all category labels, resulting in several matching pairs. Any matching pair is denoted as ri-wj (i is at most m and j is at most n).

[0129] For example, Figure 4 This is a schematic diagram of a cross-modal matching method provided in an embodiment of the present disclosure, as shown below. Figure 4 As shown, if m is 4 and n is 5 (i.e., i is at most 4 and j is at most 5), all matching pairs can form a 4×5 matching matrix, which includes 20 elements to indicate different matching pairs.

[0130] The similarity between the image embedding vector and the text embedding vector corresponding to each matching pair is calculated by the cross-modal matching module. The similarity of each matching pair in the matching matrix is ​​obtained, and the similarity matrix is ​​obtained. Each element in the similarity matrix is ​​the similarity of the corresponding matching pair.

[0131] That is, for any matching pair ri-wj in the matching matrix, calculate the similarity between ri and wj, denote the similarity value as S[ri][wj], and construct a similarity matrix using the similarity values ​​of all matching pairs.

[0132] The similarity matrix is ​​input into the classification head, and the Hungarian algorithm is used to determine the predicted matching pairs from the similarity matrix. The predicted matching pairs include candidate regions and predicted labels, thus realizing image-text alignment based on the binary matching method.

[0133] The Hungarian algorithm can find an assignment method that minimizes the total cost (or maximizes the total benefit). In this embodiment, the similarity matrix is ​​input into the Hungarian algorithm, which will find an optimal matching method and specify which candidate regions should be matched with which category labels. That is, for each candidate region, the best matching category label is determined as the predicted label, and the candidate region and the corresponding predicted label are used as a predicted matching pair. The predicted label is the category label that best matches the candidate region.

[0134] The similarity matrix is ​​mapped to the probability range using the classification head to obtain the prediction confidence matrix.

[0135] In this embodiment, a hierarchical design is used to address the model optimization problem. Cross-modal matching loss is applied to the feature space alignment stage to solve the problem of "whether the image and text match"; classification loss is applied to the final decision layer to solve the problem of "what category the target belongs to". The two play their roles at different levels of the neural network, much like different processes on a factory assembly line have their own standards.

[0136] For each sample image, based on the prediction confidence matrix, the difference between the predicted probability distribution of each candidate region in the sample image and the probability distribution of the corresponding true label is calculated, and the cross-entropy loss is obtained.

[0137] Cross-entropy loss, also known as cross-modal matching loss (Lcm), operates on the feature extraction layer and the cross-modal matching layer. Its core function is to align the semantic space of image regions with the semantic space of text embeddings, and its optimization goal is to solve the problem of image-text correlation.

[0138] When calculating the cross-entropy loss, for each sample image in the first sample set, a fully connected layer is connected based on the similarity matrix corresponding to the sample image. When optimizing the loss, the image embedding vector of each candidate region is compared with all possible text embedding vectors by using the binary cross-entropy loss function, ensuring that the model can learn multiple categories that each candidate region may correspond to.

[0139] It also introduces the idea of ​​adversarial learning, which separates positive and negative samples and uses information from the positive sample set P(i) and the negative sample set N(i) to guide the model to learn more complex contrast relationships.

[0140] Specifically, positive samples are correctly matched candidate regions and their corresponding training labels, while negative samples are other mismatched training labels. By calculating the similarity between the image embedding vector and the text embedding vector of each candidate region, and using the Sigmoid function to map the similarity to a probability range, the predicted matching degree between each candidate region and each training label (i.e., the predicted confidence of the corresponding matching pair in the prediction confidence matrix) is obtained.

[0141] The binary cross-entropy loss function measures the difference between the probability distribution corresponding to the predicted matching degree between each candidate region and each training label and the actual real label after the model makes its prediction. This optimizes the model parameters, makes the prediction results more in line with the real labels, enables the model to learn more complex contrast relationships, and can better handle many-to-one matching scenarios, thereby improving the detection capability of multi-class targets.

[0142] The formula for calculating the binary cross-entropy loss function is as follows:

[0143]

[0144] In the formula, P(i) is the positive sample set of candidate region ri, including training labels that match correctly; N(i) is the negative sample set of candidate region ri, including training labels that do not match candidate region ri, and sij and sik are the corresponding predicted matching degrees.

[0145] For each sample image, based on the similarity corresponding to each predicted match, the difference between the predicted category and the true category of each candidate region in the sample image is calculated, and the classification loss is obtained.

[0146] The classification loss operates on the classification head, its core function being to improve the classification accuracy of small sample classes, with the optimization goal of resolving the sample imbalance problem. Understandably, predicted matching pairs and similarity can quantify the difference between candidate regions and true labels, and the specific loss function calculation method can be selected or designed according to actual needs.

[0147] The sum of the classification loss and cross-entropy loss for each image in the first sample set is used as the total loss.

[0148] In this embodiment of the disclosure, the preset loss function calculation formula is as follows:

[0149] L all =L cls + L region-word

[0150] In the formula, L region-word Let cross-entropy be the loss function. L cls This is the classification loss function.

[0151] To minimize the total loss, the parameters of the detection model are adjusted to ensure that the direction of model parameter adjustment is consistent with reducing prediction error.

[0152] The technical solution provided in this disclosure adopts a 1v1 matching method in cross-modal matching, unifying 1-to-1 and 1-to-many problems into a 1v1 problem. This eliminates the need for the detection model to switch strategies between different types of matching tasks (1-to-1 or 1-to-many). The loss function is designed with classification loss and cross-entropy loss to make gradient updates more consistent during training, which helps the model converge faster. This simplifies the cross-modal matching process and facilitates loss calculation.

[0153] Current academic approaches to open vocabulary detection are mainly categorized into five types: knowledge distillation, region text pre-training, model training using balanced datasets, cue modeling, and region text alignment. Compared to region text pre-training, region text alignment aims to better match region visual and textual features during base category training without introducing additional data.

[0154] Existing methods for region-text alignment include: OV-DETR introduces a transformer-based detector for open-vocabulary object detection, replacing the bipartite graph matching method with a conditional binary matching mechanism. DetCLIPv2 uses ATSS as the object detector and is trained on three datasets: a standard detection dataset, a localization dataset, and an image-text pair dataset for word-text alignment. BARON proposes aligning embeddings of different regions, not just individual regions. It first groups context-related regions into a bag, treating each region in the bag as a word in a sentence. The region bags are then fed into a text encoder to obtain region bag embeddings, which are then aligned with cropped region embeddings from the image encoder of VLMs. CoDet reformulates region-word alignment as a co-occurrence object discovery problem and aligns co-occurrence objects with shared concepts. F-VLM discovers that the original CLIP features already have a grouping effect. It is a two-branch method similar to ViLD-text, using the CLIP visual encoder as a backbone and applying a VLM feature pooler to the region features of the backbone to obtain VLM predictions. The final result of F-VLM combines detection scores and VLM predictions.

[0155] Each of the above-mentioned region-text alignment methods has its own advantages and disadvantages. Region-text alignment remains a core research topic with room for further exploration. The region-text alignment method provided in this disclosure, which adopts a 1v1 matching approach and unifies 1v1 and 1v0 problems into a 1v1 problem, can effectively reduce computational complexity, help the model converge faster, and simplify the cross-modal matching process.

[0156] This disclosed technical solution constructs an efficient text-visual representation and alignment framework, designs an adaptive deep semantic representation method, and realizes the mapping of multimodal features in a unified semantic space. It captures fine-grained semantics of visual targets through region feature learning and establishes semantic associations between text and vision by combining an optimized matching model, thus solving the problem of heterogeneous data alignment. This framework supports cross-modal representation and understanding of knowledge in the aerospace field, providing technical support for news analysis and knowledge mining.

[0157] In one possible implementation, the classification loss for each sample image is calculated as follows:

[0158] For each real category in the first sample set, obtain the number of sample images labeled with the real category, and determine the label weight of the category label corresponding to the real category. The label weight is negatively correlated with the number of samples.

[0159] For each predicted matching pair in the sample image, the product of the similarity corresponding to the predicted matching pair and the label weight corresponding to the predicted label is used as the classification loss.

[0160] Specifically, when calculating the classification loss, a label weight ωc is introduced. For each sample image in the first sample set, the number of sample images labeled with the true category is obtained, and the label weight of the category label corresponding to each true category is set to be negatively correlated with the number of sample images corresponding to each true category.

[0161] For each predicted matching pair in each sample image, the product of the alignment weight of the predicted matching pair and the corresponding label weight is calculated as the classification loss of the predicted matching pair. For each sample image, the classification loss of all predicted matching pairs is summed to obtain the classification loss.

[0162] The formula for calculating the classification loss function is as follows:

[0163]

[0164] In the formula, i refers to the candidate region number. ω ci ri refers to the label weight corresponding to the candidate region, and c refers to the number of samples corresponding to the category label. BCE(i) The alignment weight of the predicted matching pair corresponding to the candidate region ri (i.e., the similarity of the predicted matching pair to which the candidate region is located).

[0165] For example, a quantity threshold can be set. When the number of sample images corresponding to any category label is less than the quantity threshold, that category label is treated as a small sample category and assigned a corresponding small sample category label weight.

[0166] For example, if the threshold for the number of samples is set to 10, the classification loss weight is increased to three times for categories with fewer than 10 samples, and the label weight... ω c The value of can be represented as:

[0167]

[0168] It is understood that the above specific method of assigning label weights is only used as an example to illustrate this technical solution. In addition, other methods can be used, such as setting a function to set the weights of sample images and labels, or setting a range of sample images and assigning different label weights to each range. The specific method can be determined according to actual needs.

[0169] The technical solution provided in this disclosure introduces label weights, and the label weights are negatively correlated with the corresponding number of samples. This makes full use of existing labeling resources, assigns higher weights to categories with fewer samples, prevents the model from overfitting to categories with more samples, and helps improve the detection performance of the detection model for small sample categories.

[0170] In one possible implementation, the detection model includes a region localization module;

[0171] The detection model was trained at least once using both a self-built dataset and a public dataset, and prior to this, it also included:

[0172] Based on the training labels of each training sample in the public dataset, the training samples in the public dataset are sampled with a pre-set first sampling weight and second sampling weight to obtain a sampled sample set. The first sampling weight is greater than the second sampling weight. The first sampling weight is the sampling weight of the training labels related to the aerospace field, and the second sampling weight is the sampling weight of the training labels unrelated to the aerospace field.

[0173] The sampled set is used as the input to the region localization module. The optimization objective is to minimize the difference between the bounding boxes of the training labels corresponding to the sample images and the bounding boxes of the region localization module. The parameters of the region localization module are adjusted to obtain the pre-trained detection model.

[0174] Specifically, in order to maximize the use of self-built datasets and public datasets, a two-stage strategy is adopted to train the detection model in this embodiment. In the first stage, the public dataset is used for pre-training, and in the second stage, the self-built dataset and the public dataset are used for mixed training.

[0175] That is, in this embodiment of the disclosure, before training the detection model with a self-built dataset and a public dataset, the region localization module included in the detection model is also pre-trained. During the pre-training process, dynamic category sampling weights are introduced to increase the sampling weights of aerospace-related categories during training and enhance the sensitivity of the detection model to core concepts in the aerospace field.

[0176] Based on the training labels of each training sample in the public dataset, a first sampling weight and a second sampling weight are pre-set. The first sampling weight is greater than the second sampling weight. The first sampling weight is the sampling weight of the training labels related to the aerospace field, and the second sampling weight is the sampling weight of the training labels unrelated to the aerospace field.

[0177] In other words, to determine whether a sample image contains a sample object belonging to the aerospace field, the region localization module in the detection model uses training samples from a public dataset with different sampling weights during pre-training to obtain a sample set.

[0178] Understandably, the way to determine whether a sample image is related to the aerospace field is to determine whether the category of the label of the sample image is aerospace related category. For example, if the category of the label in the sample image is "airplane", then it is considered to be a sample image related to the aerospace field.

[0179] The region localization module is pre-trained using pre-set sampling weights for both relevant datasets (including sample images related to the aerospace field) and irrelevant datasets (including sample images unrelated to the aerospace field). The sampling weights for the relevant datasets (first sampling weights) are greater than the sampling weights for the irrelevant datasets (second sampling weights).

[0180] For example, public datasets include the FAIR1M and DIOR remote sensing datasets. The first sampling weight can be set as a multiple of the second sampling weight. For instance, the sampling probability of training labels related to the aerospace field can be increased by a factor of 2. The sampling weight is shown in the following formula:

[0181]

[0182] In the formula, Cacro For the relevant dataset, pbase This represents the original sampling probability.

[0183] The sampled set is used as the input to the region localization module. The optimization objective is to minimize the difference between the bounding boxes of the training labels corresponding to the sample images and the bounding boxes of the region localization module. The goal is to make the bounding boxes predicted by the region localization module cover the real bounding boxes as accurately as possible. The parameters of the region localization module are adjusted to obtain the pre-trained detection model.

[0184] It is understandable that the specific structure of the regional positioning module, the specific type of loss function, and the optimization method can be determined according to actual needs.

[0185] For example, in the embodiments of this disclosure, the model type applied to the region localization module can be selected during the pre-training stage. Based on the target detection requirements of the region localization module, various target detection models can be selected for experimentation, such as region proposal-based models: Faster Region-based Convolutional Neural Network (Faster R-CNN), Mask Region-based Convolutional Neural Network (Mask R-CNN), Region-based Fully Convolutional Network (R-FCN), etc. Single-stage detection models: YOLO (You Only Look Once) (such as YOLOv5, YOLOv8), Single Shot MultiBox Detector (SSD), Focal Loss for Dense Object Detection (RetinaNet), etc. Lightweight models: MobileNet-YOLO, Tiny-YOLO, etc.

[0186] The public dataset was divided into training, validation, and test sets, and various object detection models were trained on each set. Training parameters included learning rate, batch size, number of training epochs, and data augmentation (using methods such as random pruning, flipping, rotation, and color dithering to enhance data diversity). The training set (approximately 70%) was used to train the object detection models. The validation set (approximately 15%) was used to adjust hyperparameters and monitor model performance during training. The test set (approximately 15%) was used for final model performance evaluation.

[0187] Average precision (mAP), recall, precision, and inference time are used as performance evaluation metrics to evaluate the performance of different target detection models, and the best model is selected as the area localization module in this embodiment.

[0188] Mean accuracy (mAP) measures the average accuracy of the model across all classes. The formula for calculating mean accuracy is:

[0189]

[0190] In the formula, N It is the total number of categories. AP iIt is the first i Average precision for each category.

[0191] Recall measures the proportion of correctly detected targets out of all actual targets. The formula for calculating recall is:

[0192]

[0193] Precision measures the proportion of targets correctly detected by a model out of all detected targets. The formula for calculating precision is:

[0194]

[0195] In the formula, TP represents the number of correctly detected targets (True Positives), FN represents the number of undetected targets (False Negatives), and FP represents the number of falsely detected targets (False Positives).

[0196] Inference time measures the time it takes for a model to process an image. The formula for calculating inference time is:

[0197]

[0198] In the formula, Total Time It represents the total time the model takes to process all images. Number of Images It represents the number of images.

[0199] It is understood that in the embodiments of this disclosure, the detection model is applied to open vocabulary target detection. After determining the target detection model corresponding to the region localization module, the structure of the target detection model needs to be adaptively adjusted. For example, the traditional Faster R-CNN model is trained for predefined categories in the category classification head, but it has limitations when applied to open vocabulary target detection tasks. If the Faster R-CNN model is used, its category classification head needs to be improved to a category-independent classification head so that it can flexibly handle unseen categories, improve the detection capability of unknown targets, and adapt to the constantly changing technologies and target types in the field.

[0200] The technical solution provided in this disclosure adopts a two-stage strategy to train the detection model. The region localization module is pre-trained with a public dataset, which can quickly learn general image features. The pre-trained region localization module already has a certain image understanding ability. By combining the self-built dataset and the public dataset to jointly train the detection model, general knowledge can be transferred to specific fields more effectively, and the adaptability of the detection model to the aerospace field can be improved.

[0201] In addition, the technical solution provided in this embodiment divides the training samples in the public dataset into categories based on whether they are related to the aerospace field during the pre-training stage of the regional positioning module. It gives higher sampling weights to the training samples related to the aerospace field, so that the first-stage training process focuses more on learning features related to the aerospace field, thereby improving the detection accuracy and efficiency of the final detection model in the aerospace field.

[0202] In one possible implementation, the corresponding descriptive text for each sample image includes: an original descriptive text and a preset number of reference descriptive texts;

[0203] Obtain news in the aerospace field, extract multiple sample images from the news articles, and retrieve corresponding descriptive text for each sample image, including:

[0204] Obtain at least one sample image from the news. For each sample image, obtain the corresponding record from the news and use the record as the original descriptive text of the sample image.

[0205] Repeat the steps of filtering reference description text until the number of reference description texts obtained reaches the preset number;

[0206] The steps for filtering reference description text include:

[0207] Each sample image and description generation instruction are input into the large language model to obtain the initial description text output by the large language model. The description generation instruction instructs the large language model to output the content displayed by the sample image in text form.

[0208] Calculate the similarity between the initial description text and the original description text;

[0209] If the similarity is determined to be greater than the similarity threshold, the initial description text will be used as the reference description text.

[0210] Specifically, to enrich the textual and image description data used in the self-built dataset application, this embodiment of the disclosure expands the description text by generating a preset number of precise description texts using a large language model, as follows:

[0211] Obtain at least one sample image from science and technology news in the aerospace field. For each sample image, obtain the corresponding record in the science and technology news and use the record used to describe the sample image as the original descriptive text of the sample image.

[0212] Each sample image and description generation instruction are input into the large language model to obtain the initial description text generated by the large language model to describe the sample image. The step of filtering reference description text is repeated until the number of reference description texts obtained reaches a preset number.

[0213] The following example illustrates the steps for filtering reference description text in this embodiment of the disclosure.

[0214] For example, obtain sample images from technology news articles, along with the original descriptive text of those images.

[0215] The sample image and description generation instructions are input into the large language model. The description generation instructions instruct the large language model to generate descriptive text to describe the sample image, and the descriptive text output by the large language model is used as the initial description text. It is understandable that the number of initial description texts generated by the large language model as instructed by the description generation instructions can be determined according to actual needs.

[0216] After obtaining the initial description text generated by the large language model, the similarity between the initial description text and the original description text is calculated.

[0217] If the semantic similarity between the initial description text and the original description text is greater than the similarity threshold, the initial description text is used as the reference description text. If the similarity between the initial description text and the original description text is less than the similarity threshold, the initial description text is considered to be inaccurate in describing the sample image, and the large language model is instructed to regenerate a new initial description text until the number of reference description texts reaches three, so that all the obtained reference description texts can accurately describe the sample image.

[0218] For example, the original description text reads, "To fix its Space Launch System rocket, NASA might need to roll it back into a building at the Kennedy Space Center in Florida."

[0219] The reference description text generated by the corresponding large language model can be obtained, including:

[0220] 1. The Space Launch System (SLS) stands tall and ready for launch within the Vehicle Assembly Building, showing its impressive size and the complexity of the surrounding scaffolding and support equipment.

[0221] 2. NASA's Space Launch System (SLS) is captured in a moment of anticipation, poised vertically on the launchpad, highlighting the rocket's central position and the intricate metal framework that envelops it.

[0222] 3. The SLS rocket, a symbol of advanced launch vehicle technology for human exploration beyond Earth's orbit, is illuminated by bright lights as itawaits its historic mission.

[0223] That is, for each sample image, the original description text and a preset number of reference description texts of the sample image are obtained, and the original description text and the reference description texts are used together as the description text of the sample image. Each sample image corresponds to multiple description texts.

[0224] This patent focuses on open-vocabulary object detection in cutting-edge technology fields, unlike existing patents or papers. The project innovates in dataset construction, focusing on aerospace technology news, deeply exploring its characteristics, identifying reliable data collection sources, and expanding the data to improve its relevance and quality, laying the foundation for model training and validation.

[0225] It is understood that, in this embodiment of the disclosure, in order to increase the semantic similarity between the initial description text generated by the large language model and the original description text, the large language model can be fine-tuned using a dataset (containing images and corresponding original description text) so that the large language model has better performance when performing the task of generating initial description text for describing images.

[0226] The dataset is divided into training, validation, and test sets. The training set (approximately 70%) is used to train the object detection model. The validation set (approximately 15%) is used to tune hyperparameters and monitor model performance during training. The test set (approximately 15%) is used for final model performance evaluation.

[0227] To verify the semantic understanding capabilities of large language models, three evaluation metrics can be applied: BLEU (Bilingual Evaluation Understudy), the Metric for Evaluation of Translation with Explicit Ordering (METEOR), and the Consensus-Based Image Description Evaluation (CIDEr). These metrics can be used to jointly assess the similarity between the descriptive text generated by the large language model and the original descriptive text.

[0228] The BLEU metric assesses the matching degree by calculating the co-occurrence frequency of n-grams in the initial description text generated by the large language model and the original description text.

[0229] The METEOR metric assesses the degree of matching by aligning the initial description text generated from the construction of a large language model with the original description text.

[0230] The CIDEr metric calculates the matching degree of an n-gram based on the Term Frequency-Inverse Document Frequency (TF-IDF) weight.

[0231] The technical solution provided in this disclosure uses a large language model to obtain reference description text and expand the description text, and ensures that the reference description text can accurately describe the sample image. The use of high-quality expanded description text helps the model learn multiple ways to describe the image object, reduces the dependence on a single description, and improves the robustness of the detection model to changes in text description.

[0232] In one possible implementation, the image to be detected and its corresponding descriptive text are input into a pre-trained detection model to obtain at least one bounding box and the target predicted category of the object in each bounding box, including:

[0233] The image to be detected and the corresponding descriptive text are input into a pre-trained detection model. The detection model identifies and locates the objects to be detected in the image, determines the bounding box of each object, and obtains the image embedding vector of each bounding box in the target region of the image to be detected.

[0234] Extract entity nouns from the description text as candidate categories, encode the description text, and obtain the text embedding vector for each candidate category;

[0235] All target regions and all candidate categories are matched one by one to obtain multiple matching pairs. The similarity between the image embedding vector and the text embedding vector in each matching pair is calculated to obtain a similarity matrix. Each element in the similarity matrix is ​​the similarity of the corresponding matching pair.

[0236] The similarity matrix is ​​mapped to the probability range to obtain the prediction confidence matrix. For each bounding box, based on the prediction confidence between the target region corresponding to the bounding box and each candidate category, all bounding boxes are determined and output, as well as the target prediction category corresponding to each bounding box.

[0237] Specifically, in the application phase of the trained detection model, the image to be detected and its corresponding descriptive text are input into the pre-trained detection model. Figure 5 This is a schematic diagram of the structure of a detection model provided in an embodiment of the present disclosure, such as... Figure 5 As shown, the detection model is divided into multiple modules according to the functions implemented by each module in the detection model, including: category extraction module, region localization module, embedding representation module, cross-modal matching module and classification head. The embedding representation module includes: text encoder and image encoder.

[0238] The image to be detected is input into the region localization module. The region localization module identifies and locates the objects to be detected in the image, determines the bounding box of each object to be detected, and obtains the image embedding vector of each bounding box in the target region of the image to be detected through the image encoder.

[0239] The descriptive text corresponding to the image to be detected is input into the category extraction module. The category extraction module extracts entity nouns from the descriptive text as candidate categories, and the descriptive text is encoded by the text encoder to obtain the text embedding vector of each candidate category.

[0240] All text embeddings and all image embeddings are input into the cross-modal matching module. Through the cross-modal matching layer, all target regions and all candidate categories are matched one by one to obtain multiple matching pairs. The similarity between the image embedding vector and the text embedding vector in each matching pair is calculated to obtain a similarity matrix. Each element in the similarity matrix is ​​the similarity of the corresponding matching pair.

[0241] The similarity matrix is ​​input into the classification head, and the similarity matrix is ​​mapped to the probability range through the classification head to obtain the prediction confidence matrix. For each bounding box, based on the prediction confidence between the target region corresponding to the bounding box and each candidate category, all bounding boxes are determined and output, as well as the target prediction category corresponding to each bounding box.

[0242] Science and technology news covers various aspects such as science and technology policies, achievements, and perceptions, combining scientific accuracy, knowledge, and accessibility, making it an important channel for the public to understand science and technology. With the rapid development of science and technology, visual information in science and technology news has become key to the dissemination of cutting-edge information; however, current intelligence analysis relies heavily on textual interpretation, neglecting in-depth analysis of images.

[0243] Images, as a crucial component of science and technology news, provide direct visual evidence and details that text cannot convey. Traditional intelligence analysis, focusing primarily on text, has limitations and cannot fully comprehend the entirety of an event. Open-vocabulary object detection, combining textual and image information, can identify more unknown categories and provide direct visual evidence. Especially in the demonstration of emerging technologies and the identification of misinformation, it can efficiently and accurately grasp the core of an event, providing reliable support for the verification of science and technology intelligence.

[0244] The following example illustrates the technical solution of this disclosure by applying the open vocabulary target detection method for the aerospace field provided in the embodiments of this disclosure in an intelligence analysis scenario:

[0245] The goal of using open-vocabulary object detection methods for intelligence analysis is to identify more granular aircraft categories. Based on this, the FAIR1M and DIOR remote sensing datasets were selected to jointly construct a public dataset, with the FAIR1M dataset including images of various aircraft models.

[0246] The detection images are trained using a public dataset and a self-built dataset constructed using the method provided in this embodiment, and the specific training method is not described in detail here.

[0247] By training with a combination of remote sensing datasets and self-built datasets, we can not only increase the model's generalization ability by category, but also learn the characteristics of optical modes in remote sensing images, thereby increasing the model's generalization ability by image modes.

[0248] By applying the trained detection model to intelligence gathering, the target image and descriptive text obtained from cutting-edge news or papers can be input into the detection model to identify the objects in the target image and predict the target category of the objects.

[0249] For example, extracting attached images and their corresponding descriptive text from news reports. Figure 6 This is a schematic diagram of the appearance of a deformable aircraft provided in an embodiment of this disclosure, as shown in the attached figure. Figure 6 As shown, the description text states that "the research on intelligent deformable aircraft involves the forefront and hot topics of multiple disciplines such as unsteady aerodynamics, time-varying structural mechanics, dynamic modeling and control, aerodynamic servo elasticity, intelligent materials and structural mechanics, nonlinear system dynamics, sensor and actuator design, and perception and control science."

[0250] By inputting the obtained attached images and descriptive text into the detection model, we can obtain... Figure 6 The detection category for the object is "Deformable Aircraft".

[0251] Traditional object detection algorithms often fail to recognize this new category. Figure 7 This is a schematic diagram of the appearance of a conventional aircraft provided in an embodiment of this disclosure. Conventional target detection algorithms can only detect... Figure 6-7 Both the modified and conventional aircraft shown are identified as "aircraft".

[0252] The detection model provided in this disclosure is guided by text information and can effectively detect unknown scientific and technological entities in aerospace science and technology news images. Compared with the zero-shot methods commonly used in the current target detection field, it has strong applicability and professionalism in specific scientific and technological fields and meets the needs of cross-modal analysis of scientific and technological intelligence.

[0253] The open vocabulary target detection method for the aerospace field provided in this disclosure has the following advantages:

[0254] In the field of aerospace science and technology intelligence, open-vocabulary object detection technology can accurately identify key technologies and R&D trends in news images, such as satellite configurations and rocket engine nozzles. This unstructured visual information is difficult to extract effectively using traditional text analysis methods. However, open-vocabulary object detection technology, combined with the powerful representation capabilities of deep learning models and cross-modal alignment techniques, overcomes the limitations of predefined categories and can accurately identify undisclosed technical details, such as new aerospace components and special material processes.

[0255] Specifically, pre-trained visual language models can learn joint representations of image and text features, aligning image features with text descriptions to identify known and unknown categories. This technology not only improves the flexibility and adaptability of object detection but also plays a crucial role in identifying misinformation. By constructing a multimodal evidence chain verification system, open-vocabulary object detection technology can quickly identify key targets in images and compare them with text content for verification, effectively addressing the impact of deepfake technology on the science and technology intelligence ecosystem.

[0256] For example, in science and technology news involving the demonstration of emerging technologies, the intuitive visual evidence provided by images can help to grasp the core content of the event more efficiently and accurately. At the same time, by comparing images with text descriptions, discrepancies between images and text and false information can be discovered, providing more reliable support for the identification of science and technology intelligence.

[0257] The development of deep learning has provided strong technical support for open vocabulary target detection, enabling it to achieve a leap "from seeing to knowing" in the field of aerospace science and technology intelligence, significantly improving the accuracy and reliability of intelligence analysis, and opening up new possibilities for the automated and intelligent processing of aerospace science and technology intelligence.

[0258] Furthermore, the aerospace field, as a cutting-edge technology area that the country is vigorously developing, often generates many emerging technologies of unknown categories. Traditional target detection algorithms may be unable to identify these new categories, thus failing to achieve accurate intelligence analysis.

[0259] It is understood that the above detailed description of the technical effects of the present invention is only intended to facilitate the understanding of the present invention and to distinguish it from the open vocabulary target detection method for the aerospace field provided in the embodiments of this disclosure, and does not serve to limit it in any way.

[0260] Figure 8 This is a schematic diagram of the structure of an open vocabulary target detection device for the aerospace field provided in an embodiment of this disclosure, as shown below. Figure 8 As shown, the open vocabulary target detection device 80 for the aerospace field includes:

[0261] The open vocabulary object detection module 801 is used to input the image to be detected and the corresponding descriptive text of the image to be detected into a pre-trained detection model, and obtain at least one bounding box and the target prediction category of the object in each bounding box. The target prediction category is the entity noun included in the descriptive text.

[0262] The detection model is trained based on a pre-built sample dataset, which includes a self-built dataset.

[0263] The training data in the self-built dataset is generated in the following ways:

[0264] Obtain news in the aerospace field, extract multiple sample images from the news, and obtain corresponding descriptive text for each sample image;

[0265] For each description text, entity nouns are extracted from the description text, and expanded words of entity nouns are determined, including hypernyms and abbreviations;

[0266] For each sample image, the sample image is used as a training sample. A bounding box is labeled for each sample object in each sample image. The bounding box and the true category of the sample object in the bounding box are used as training labels. The true categories include entity nouns and extended words.

[0267] In one possible implementation, the sample dataset also includes a public dataset;

[0268] The detection model is trained in the following way:

[0269] The detection model was trained at least once using both a self-built dataset and a publicly available dataset.

[0270] For each training round, the first sample set for this training round is determined. The first sample set includes at least one self-built sample and at least one public sample. The ratio of self-built samples to public samples is positively correlated with the training round. Self-built samples are from self-built datasets, and public samples are from public datasets.

[0271] In one possible implementation, each training round includes the following steps:

[0272] For each sample image in the first sample set, identify and locate the sample objects in the sample image, determine the predicted bounding box for each sample object, and obtain the image embedding vector of the candidate region corresponding to each predicted bounding box in the sample image.

[0273] Encode the descriptive text corresponding to each sample image to obtain the text embedding vector of the category label corresponding to each real category in the descriptive text;

[0274] All candidate regions and all category labels are matched one by one to obtain multiple matching pairs. The similarity between the image embedding vector and the text embedding vector corresponding to each matching pair is calculated to obtain a similarity matrix. Each element in the similarity matrix is ​​the similarity of the corresponding matching pair.

[0275] Using the Hungarian algorithm, predicted matching pairs are determined from the similarity matrix, and the similarity matrix is ​​mapped to the probability range to obtain the predicted confidence matrix. The predicted matching pairs include candidate regions and predicted labels, and the predicted labels are the category labels that best match the candidate regions.

[0276] For each sample image, based on the prediction confidence matrix, the difference between the predicted probability distribution of each candidate region in the sample image and the probability distribution of the corresponding true label is calculated, and the cross-entropy loss is obtained.

[0277] For each sample image, the difference between the predicted category and the true category of each candidate region in the sample image is calculated based on the similarity corresponding to each predicted match, and the classification loss is obtained.

[0278] The sum of the classification loss and cross-entropy loss for each sample image in the first sample set is taken as the total loss;

[0279] The parameters of the detection model are adjusted with the goal of minimizing the total loss.

[0280] In one possible implementation, the classification loss for each sample image is calculated as follows:

[0281] For each real category in the first sample set, obtain the number of sample images labeled with the real category, and determine the label weight of the category label corresponding to the real category. The label weight is negatively correlated with the number of samples.

[0282] For each predicted matching pair in the sample image, the product of the similarity corresponding to the predicted matching pair and the label weight corresponding to the predicted label is used as the classification loss.

[0283] In one possible implementation, the detection model includes a region localization module;

[0284] The detection model was trained at least once using both a self-built dataset and a public dataset, and prior to this, it also included:

[0285] Based on the training labels of each training sample in the public dataset, the training samples in the public dataset are sampled with a pre-set first sampling weight and second sampling weight to obtain a sampled sample set. The first sampling weight is greater than the second sampling weight. The first sampling weight is the sampling weight of the training labels related to the aerospace field, and the second sampling weight is the sampling weight of the training labels unrelated to the aerospace field.

[0286] The sampled set is used as the input to the region localization module. The optimization objective is to minimize the difference between the bounding boxes of the training labels corresponding to the sample images and the bounding boxes of the region localization module. The parameters of the region localization module are adjusted to obtain the pre-trained detection model.

[0287] In one possible implementation, the corresponding descriptive text for each sample image includes: an original descriptive text and a preset number of reference descriptive texts;

[0288] Obtain news in the aerospace field, extract multiple sample images from the news articles, and retrieve corresponding descriptive text for each sample image, including:

[0289] Obtain at least one sample image from the news. For each sample image, obtain the corresponding record from the news and use the record as the original descriptive text of the sample image.

[0290] Repeat the steps of filtering reference description text until the number of reference description texts obtained reaches the preset number;

[0291] The steps for filtering reference description text include:

[0292] Each sample image and description generation instruction are input into the large language model to obtain the initial description text output by the large language model. The description generation instruction instructs the large language model to output the content displayed by the sample image in text form.

[0293] Calculate the similarity between the initial description text and the original description text;

[0294] If the similarity is determined to be greater than the similarity threshold, the initial description text will be used as the reference description text.

[0295] In one possible implementation, the image to be detected and its corresponding descriptive text are input into a pre-trained detection model to obtain at least one bounding box and the target predicted category of the object in each bounding box, including:

[0296] The image to be detected and the corresponding descriptive text are input into a pre-trained detection model. The detection model identifies and locates the objects to be detected in the image, determines the bounding box of each object, and obtains the image embedding vector of each bounding box in the target region of the image to be detected.

[0297] Extract entity nouns from the description text as candidate categories, encode the description text, and obtain the text embedding vector for each candidate category;

[0298] All target regions and all candidate categories are matched one by one to obtain multiple matching pairs. The similarity between the image embedding vector and the text embedding vector in each matching pair is calculated to obtain a similarity matrix. Each element in the similarity matrix is ​​the similarity of the corresponding matching pair.

[0299] The similarity matrix is ​​mapped to the probability range to obtain the prediction confidence matrix. For each bounding box, based on the prediction confidence between the target region corresponding to the bounding box and each candidate category, all bounding boxes are determined and output, as well as the target prediction category corresponding to each bounding box.

[0300] The apparatus of this disclosure embodiment can execute the method provided in this disclosure embodiment, and the implementation principle is similar. The actions performed by each module in the apparatus of each disclosure embodiment correspond to the steps in the method of each disclosure embodiment. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0301] Furthermore, in this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Moreover, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0302] This disclosure provides an electronic device (computer device / equipment / system) including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the method provided in any optional embodiment of this disclosure and achieve the corresponding technical effects.

[0303] In one alternative embodiment, an electronic device is provided. Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as... Figure 9 As shown, the electronic device 90 includes a processor 901 and a memory 903. The processor 901 and the memory 903 are connected, for example, via a bus 902. Optionally, the electronic device 900 may further include a transceiver 904, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 904 is not limited to one type, and the structure of the electronic device 900 does not constitute a limitation on the embodiments of this disclosure.

[0304] Processor 901 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. Processor 901 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0305] Bus 902 may include a pathway for transmitting information between the aforementioned components. Bus 902 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 902 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0306] The memory 903 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0307] The memory 903 stores computer programs that execute embodiments of the present disclosure and is controlled by the processor 901. The processor 901 executes the computer programs stored in the memory 903 to implement the steps shown in the foregoing method embodiments.

[0308] The electronic devices in this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable devices, and fixed terminals such as digital TVs and desktop computers.

[0309] This disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0310] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0311] It should be noted that the computer-readable storage medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0312] In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0313] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0314] The terms “first,” “second,” “third,” “fourth,” “1,” “2,” etc. (if present) in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in a sequence other than that shown in the figures or text.

[0315] It should be understood that although arrows indicate various operation steps in the flowcharts of the embodiments of this disclosure, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of the embodiments of this disclosure, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured as required, and the embodiments of this disclosure do not limit this.

[0316] The above description is only an optional implementation method for some implementation scenarios of this disclosure. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this disclosure without departing from the technical concept of this disclosure also fall within the protection scope of the embodiments of this disclosure.

Claims

1. An open vocabulary object detection method for aerospace domain, characterized in that, The method comprises the following steps: inputting a to-be-detected image and a description text corresponding to the to-be-detected image into a pre-trained detection model to obtain at least one annotation box and a target prediction category of an object in each annotation box output by the detection model, wherein the target prediction category is an entity noun included in the description text; wherein the detection model is trained according to a pre-constructed sample data set, and the sample data set comprises a self-built data set; the training data in the self-built data set is generated by the following method: obtaining news in the field of aerospace, and obtaining a plurality of sample images and corresponding description texts of each sample image from the news; for each description text, extracting an entity noun from the description text and determining an extension word of the entity noun, wherein the extension word comprises a hypernym and an abbreviation; for each sample image, taking the sample image as a training sample, labeling an annotation box of each sample object in the sample image, and taking the annotation box and the real category of the sample object in the annotation box as a training label, wherein the real category comprises an entity noun and an extension word; the detection model is trained by the following method: training the detection model at least one round with the sample data set; each round of training comprises the following steps: from the sample data set, determine a first sample set applied in this round of training, for each sample image in the first sample set, identify and locate a sample object in the sample image, determine a prediction annotation box of each sample object, and obtain an image embedding vector of a candidate region corresponding to each prediction annotation box in the sample image; encode the description text corresponding to each sample image to obtain a text embedding vector of each real category corresponding to the class label in the description text; match all candidate regions and all class labels to obtain a plurality of matching pairs, and calculate the similarity between the image embedding vector and the text embedding vector corresponding to each matching pair to obtain a similarity matrix, wherein each element in the similarity matrix is the similarity of the corresponding matching pair; determine a prediction matching pair from the similarity matrix using the Hungarian algorithm, and map the similarity matrix to a probability range to obtain a prediction confidence matrix, wherein the prediction matching pair comprises a candidate region and a prediction label, and the prediction label is the class label best matched with the candidate region; for each sample image, calculate the difference between the prediction probability distribution of each candidate region in the sample image and the probability distribution of the corresponding real label according to the prediction confidence matrix to obtain a cross-entropy loss; for each sample image, calculate the difference between the prediction category and the real category of each candidate region in the sample image according to the similarity corresponding to each prediction matching pair to obtain a classification loss; sum the classification loss and the cross-entropy loss corresponding to each sample image in the first sample set as a total loss; adjust the parameters of the detection model with the optimization goal of minimizing the total loss.

2. The method of claim 1, wherein, the sample data set further comprises a public data set; the detection model is trained by the following method: train the detection model at least one round with the self-built data set and the public data set; Wherein, for each training round, the first sample set comprises at least one self-built sample and at least one public sample, and the proportion of the self-built sample to the public sample is positively correlated with the training round, the self-built sample is derived from the self-built dataset, and the public sample is derived from the public dataset.

3. The method of claim 1, wherein, The classification loss of each sample image is calculated by the following method: For each real category in the first sample set, the number of sample images labeled with the real category is obtained, and the label weight of the category label corresponding to the real category is determined, the label weight being negatively correlated with the number of sample images; For each predicted matching pair in the sample image, the product of the similarity corresponding to the predicted matching pair and the label weight corresponding to the predicted label is taken as the classification loss.

4. The method of claim 2, wherein, The detection model comprises a region positioning module; Before the at least one round of training of the detection model with the self-built dataset and the public dataset, the method further comprises: According to the training labels of each training sample in the public dataset, sampling the training samples in the public dataset with a pre-set first sampling weight and a second sampling weight to obtain a sampling sample set, the first sampling weight being greater than the second sampling weight, the first sampling weight being the sampling weight of a training label related to the aerospace field, and the second sampling weight being the sampling weight of a training label unrelated to the aerospace field; Taking the sampling sample set as the input of the region positioning module, taking the minimization of the difference between the annotation frame corresponding to the training label of the sample image and the annotation frame output by the region positioning module as the optimization target, and adjusting the parameters of the region positioning module to obtain a pre-trained detection model.

5. The method according to any one of claims 1 to 4, characterized in that, The description text corresponding to each sample image comprises an original description text and a preset number of reference description texts. The method of obtaining the news in the aerospace field and obtaining a plurality of sample images and the description text corresponding to each sample image from the news comprises: For each sample image, obtaining the corresponding record from the news, and taking the record as the original description text of the sample image; Repeating the step of screening the reference description text until the number of obtained reference description texts reaches the preset number; Wherein, the step of screening the reference description text comprises: inputting each sample image and a description generation instruction into a large language model to obtain an initial description text output by the large language model, the description generation instruction instructing the large language model to output the content displayed by the sample image in the form of text; calculating the similarity between the initial description text and the original description text; if it is determined that the similarity is greater than a similarity threshold, taking the initial description text as a reference description text.

6. The method according to any one of claims 1 to 4, characterized in that, The method of inputting the to-be-detected image and the description text corresponding to the to-be-detected image into the pre-trained detection model to obtain at least one annotation frame and the target prediction category of the object in each annotation frame output by the detection model comprises: inputting the to-be-detected image and the description text corresponding to the to-be-detected image into a pre-trained detection model, identifying and locating a to-be-detected object in the to-be-detected image through the detection model, determining a labeling box of each to-be-detected object, and obtaining an image embedding vector of each labeling box in a target region corresponding to the to-be-detected image; extracting entity nouns from the description text as candidate categories, and encoding the description text to obtain a text embedding vector of each candidate category; matching all target regions and all candidate categories to obtain a plurality of matching pairs, and calculating the similarity between the image embedding vector and the text embedding vector in each matching pair to obtain a similarity matrix, each element in the similarity matrix being the similarity of the corresponding matching pair; mapping the similarity matrix to a probability range to obtain a predicted confidence matrix, and for each labeling box, determining and outputting all labeling boxes and a target predicted category corresponding to each labeling box according to the predicted confidence between the target region corresponding to the labeling box and each candidate category.

7. An aerospace-oriented open vocabulary object detection apparatus, characterized by, It comprises: an open vocabulary object detection module for inputting a to-be-detected image and a description text corresponding to the to-be-detected image into a pre-trained detection model, obtaining at least one labeling box and a target predicted category of an object in each labeling box output by the detection model, and the target predicted category being an entity noun included in the description text; wherein the detection model is trained according to a pre-constructed sample data set, and the sample data set includes a self-built data set; the training data in the self-built data set is generated by the following method: obtaining news in the field of aerospace, and obtaining a plurality of sample images and description texts corresponding to each sample image from the news; for each description text, extracting entity nouns from the description text and determining the extension words of the entity nouns, the extension words including hypernyms and abbreviations; for each sample image, taking the sample image as a training sample, labeling the labeling box of each sample object in each sample image, and taking the labeling box in the sample image and the real category of the sample object in the labeling box as a training label, the real category including entity nouns and extension words; the detection model is trained by the following method: training the detection model at least one round with the sample data set; the steps of each round of training include: from the sample data set, determining a first sample set applied in the current round of training, for each sample image in the first sample set, identifying and locating a sample object in the sample image, determining a predicted labeling box of each sample object, and obtaining an image embedding vector of each predicted labeling box in a candidate region corresponding to the sample image; encoding the description text corresponding to each sample image to obtain a text embedding vector of each real category corresponding to the class label in the description text; matching all candidate regions and all class labels to obtain a plurality of matching pairs, and calculating the similarity between the image embedding vector and the text embedding vector corresponding to each matching pair to obtain a similarity matrix, each element in the similarity matrix being the similarity of the corresponding matching pair; determining, from the similarity matrix, a predicted matching pair comprising a candidate region and a predicted label that is a class label best matching the candidate region, using a Hungarian algorithm, and mapping the similarity matrix to a probability range to obtain a predicted confidence matrix; for each sample image, calculating, according to the predicted confidence matrix, a difference between a predicted probability distribution of each candidate region in the sample image and a probability distribution of a corresponding ground-truth label to obtain a cross-entropy loss; for each sample image, calculating, according to a similarity corresponding to each predicted matching pair, a difference between a predicted class of each candidate region in the sample image and a ground-truth class to obtain a classification loss; summing the classification loss and the cross-entropy loss corresponding to each sample image in the first sample set as a total loss; adjusting parameters of the detection model with an optimization objective of minimizing the total loss.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, is arranged to perform the method of any one of claims 1 to 7. The processor executes the computer program to implement the method of any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1-6.

Images