Method and device for generating multi-level subtitles for remote sensing images based on hybrid intelligence

Abstract

The present invention discloses a method and device for generating multi-level subtitles for remote sensing images based on hybrid intelligence, which relates to the fields of remote sensing image processing and computer vision technology, and can solve the technical problems of coarse annotation granularity, simple subtitles, and single scenes in the current remote sensing image subtitle generation algorithm in the remote sensing field. The invention comprises: determining the ground object elements and target scene categories corresponding to the remote sensing image; automatically correcting the ground object elements according to the target scene category and the ground object element correction rules to obtain the automatically corrected ground object elements; generating subtitles for the remote sensing image using the subtitle generation algorithm and spatial relationship prior knowledge to obtain spatial relationship subtitles; obtaining the first multi-level subtitles according to the automatically corrected ground object elements, spatial relationship subtitles, and target scene categories; in response to an instruction to manually correct the first multi-level subtitles, receiving the manual correction of the first multi-level subtitles based on the remote sensing image subtitle prior knowledge to obtain the target multi-level subtitles.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing and computer vision technology, and in particular to a method and device for generating multi-level captions of remote sensing images based on hybrid intelligence. Background Art

[0002] With the continuous development of remote sensing technology, remote sensing image data presents new features such as massive, high-resolution, multi-temporal, and multi-source heterogeneity. The massive remote sensing image data poses huge challenges to manual interpretation and analysis. There is an urgent need for intelligent information extraction methods to convert images into semantic expressions that can be understood by humans.

[0003] Remote sensing image subtitles refer to the use of natural language to describe the content of remote sensing images, converting the visual information in remote sensing images into text subtitles, so that people can more easily understand the content of the image. Remote sensing image subtitles can provide high-level semantic information of image content, which helps to alleviate the semantic gap problem in image understanding. Remote sensing image subtitles not only need to identify the objects in the image, but also need to analyze the spatial relationship between the objects, understand the overall semantics of the scene, and convert it into fluent and accurate language expression, thereby forming a picture-text pair data set. However, limited by the professionalism and complexity of remote sensing images, the current remote sensing image subtitle generation algorithm in the remote sensing field has shortcomings such as coarse annotation granularity, simple subtitles, and single scenes, which makes it difficult to support image interpretation and analysis of complex scenes. Summary of the invention

[0004] In view of this, the present invention provides a method and device for generating multi-level subtitles for remote sensing images based on hybrid intelligence, which can solve the technical problems of the current remote sensing image subtitle generation algorithm in the remote sensing field, such as coarse annotation granularity, simple subtitles, and single scenes.

[0005] According to a first aspect of the present invention, a method for generating multi-level captions for remote sensing images based on hybrid intelligence is provided, the method comprising:

[0006] Acquire a remote sensing image data set, and for each remote sensing image in the remote sensing image data set, determine a ground object element and a target scene category corresponding to the remote sensing image;

[0007] Automatically correcting the ground object elements according to the target scene category and the ground object element correction rule to obtain the automatically corrected ground object elements corresponding to the remote sensing image, generating subtitles for the remote sensing image using a subtitle generation algorithm and spatial relationship prior knowledge to obtain spatial relationship subtitles corresponding to the remote sensing image, and obtaining a first multi-level subtitle according to the automatically corrected ground object elements, the spatial relationship subtitles and the target scene category;

[0008] In response to an instruction to manually correct the first multi-layer subtitles, prior knowledge of remote sensing image subtitles is displayed, and manual correction of the first multi-layer subtitles based on the prior knowledge of remote sensing image subtitles is received to obtain target multi-layer subtitles.

[0009] According to a second aspect of the present invention, a hybrid intelligent remote sensing image multi-level caption generation device is provided, the device comprising:

[0010] An acquisition module is used to acquire a remote sensing image data set, and for each remote sensing image in the remote sensing image data set, determine the ground object element and target scene category corresponding to the remote sensing image;

[0011] A generation module, configured to automatically correct the ground object elements according to the target scene category and the ground object element correction rule to obtain the automatically corrected ground object elements corresponding to the remote sensing image, generate subtitles for the remote sensing image using a subtitle generation algorithm and spatial relationship prior knowledge to obtain spatial relationship subtitles corresponding to the remote sensing image, and obtain a first multi-level subtitle according to the automatically corrected ground object elements, the spatial relationship subtitles and the target scene category;

[0012] The display module is used to respond to the instruction of manually correcting the first multi-level subtitles, display the remote sensing image subtitle prior knowledge, receive the manual correction of the first multi-level subtitles based on the remote sensing image subtitle prior knowledge, and obtain the target multi-level subtitles.

[0013] According to the third aspect of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for generating multi-level subtitles for remote sensing images based on hybrid intelligence is implemented.

[0014] According to the fourth aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein when the processor executes the program, the method for generating multi-level subtitles for remote sensing images based on hybrid intelligence is implemented.

[0015] By means of the above technical scheme, the present invention provides a method and device for generating multi-level subtitles for remote sensing images based on hybrid intelligence. First, a remote sensing image data set is obtained. For each remote sensing image in the remote sensing image data set, the ground object elements and target scene categories corresponding to the remote sensing image are determined; then, the ground object elements are automatically corrected according to the target scene category and the ground object element correction rule to obtain the automatically corrected ground object elements corresponding to the remote sensing image; subtitles are generated for the remote sensing image using a subtitle generation algorithm and spatial relationship prior knowledge to obtain spatial relationship subtitles corresponding to the remote sensing image; a first multi-level subtitle is obtained according to the automatically corrected ground object elements, the spatial relationship subtitles and the target scene category; finally, in response to an instruction to manually correct the first multi-level subtitles, the remote sensing image subtitle prior knowledge is displayed, and the manual correction of the first multi-level subtitles based on the remote sensing image subtitle prior knowledge is received to obtain the target multi-level subtitles. Through the technical scheme of the present invention, the ground object elements are automatically deleted and supplemented based on the ground object element correction rule, the spatial relationship subtitles obtained based on the spatial relationship prior knowledge have more detailed information, the semantic accuracy of the subtitles is improved, the ground object elements, spatial relationship subtitles and target scene categories after automatic correction are independently composed of a first multi-level subtitle in multiple layers instead of integrating the three together, the richness and flexibility of the subtitle layer are improved, the first multi-level subtitles are manually corrected based on the remote sensing image subtitle prior knowledge, so that the obtained target multi-level subtitles can better express the characteristics of the remote sensing image, reduce repetition and supplement more detailed information, and improve the semantic quality of the subtitles.

[0016] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation on the local application. In the drawings:

[0018] Figure 1 A schematic diagram of a flow chart of a method for generating multi-level captions for remote sensing images based on hybrid intelligence provided by an embodiment of the present invention is shown;

[0019] Figure 2 A schematic diagram of a process of another method for generating multi-level captions of remote sensing images based on hybrid intelligence provided by an embodiment of the present invention is shown;

[0020] Figure 3A schematic diagram of the structure of a hybrid intelligent remote sensing image multi-level caption generation device provided by an embodiment of the present invention is shown;

[0021] Figure 4 A schematic diagram of the structure of another hybrid intelligent remote sensing image multi-level caption generation device provided by an embodiment of the present invention is shown;

[0022] Figure 5 An example diagram of an existing remote sensing image caption provided by an embodiment of the present invention is shown;

[0023] Figure 6 An example diagram of a multi-level subtitle provided by an embodiment of the present invention is shown;

[0024] Figure 7 An example diagram of a multi-level subtitle structure provided by an embodiment of the present invention is shown;

[0025] Figure 8 An example diagram showing a spatial relationship of unified subtitles in eight fields provided by an embodiment of the present invention is shown;

[0026] Fig. 9 An example diagram showing a comparison between a target multi-level subtitle provided by an embodiment of the present invention and an existing remote sensing image subtitle is shown. DETAILED DESCRIPTION

[0027] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0028] This embodiment provides a method for generating multi-level captions for remote sensing images based on hybrid intelligence. Figure 1 As shown, the method includes:

[0029] 101. Obtain a remote sensing image data set, and for each remote sensing image in the remote sensing image data set, determine a ground object element and a target scene category corresponding to the remote sensing image.

[0030] 102. Automatically correct the ground object elements according to the target scene category and the ground object element correction rule to obtain the automatically corrected ground object elements corresponding to the remote sensing image, generate subtitles for the remote sensing image using a subtitle generation algorithm and spatial relationship prior knowledge to obtain spatial relationship subtitles corresponding to the remote sensing image, and obtain a first multi-level subtitle based on the automatically corrected ground object elements, the spatial relationship subtitles and the target scene category.

[0031] 103. In response to an instruction to manually correct the first multi-layer subtitles, display prior knowledge of remote sensing image subtitles, receive manual correction of the first multi-layer subtitles based on the prior knowledge of remote sensing image subtitles, and obtain target multi-layer subtitles.

[0032] It should be noted that remote sensing image subtitles can provide high-level semantic information of image content, such as object categories, attributes, scene context, etc., and provide training samples for tasks such as multimodal understanding and cross-modal retrieval, which helps to alleviate the semantic gap problem in image understanding and is of great significance to promoting intelligent analysis of remote sensing images. In addition, the use of subtitles composed of natural language can interact more closely with the model in a way that humans can understand, thereby guiding the model to generate more interpretable results and acquire knowledge from the world outside the image itself. Remote sensing image subtitle generation not only needs to identify the objects in the image, but also needs to analyze the spatial relationship between the objects, understand the overall semantics of the scene, and convert it into fluent and accurate language expression, thereby forming a picture-text pair dataset, such as Figure 5 As shown in FIG, the existing remote sensing image subtitle example diagram is generated based on the current remote sensing image subtitle algorithm. However, the current remote sensing image subtitle algorithm in the remote sensing field has shortcomings such as coarse annotation granularity, simple subtitles, and single scenes, which makes it difficult to support image interpretation and analysis of complex scenes.

[0033] For step 101 of the embodiment, for determining the ground object elements corresponding to the remote sensing image, as an implementation mode, a convolutional neural network is used to perform semantic segmentation on the remote sensing image to obtain the ground object elements corresponding to the remote sensing image composed of words and / or phrases. As another implementation mode, a convolutional neural network is used to perform multi-label classification on the remote sensing image to obtain the ground object elements corresponding to the remote sensing image composed of words and / or phrases.

[0034] As an implementation method for determining the target scene category corresponding to the remote sensing image, a convolutional neural network is used to perform scene classification on the remote sensing image to obtain the target scene category corresponding to the remote sensing image composed of words and / or phrases.

[0035] Since the ground object elements directly generated by the convolutional neural network in step 101 of the embodiment are not accurate, it is necessary to automatically correct the ground object elements. For step 102 of the embodiment, the ground object elements are automatically corrected according to the target scene category and the ground object element correction rule to obtain the automatically corrected ground object elements corresponding to the remote sensing image. This step is automatic correction. For step 103 of the embodiment, the first multi-level subtitles are manually corrected based on the prior knowledge of remote sensing image subtitles to obtain the target multi-level subtitles. This step is manual correction.

[0036] The present invention provides a method and device for generating multi-level subtitles for remote sensing images based on hybrid intelligence. First, a remote sensing image data set is obtained. For each remote sensing image in the remote sensing image data set, a ground object element and a target scene category corresponding to the remote sensing image are determined; then, the ground object element is automatically corrected according to the target scene category and the ground object element correction rule to obtain the automatically corrected ground object element corresponding to the remote sensing image; a subtitle generation algorithm and spatial relationship prior knowledge are used to generate subtitles for the remote sensing image to obtain spatial relationship subtitles corresponding to the remote sensing image; a first multi-level subtitle is obtained according to the automatically corrected ground object element, the spatial relationship subtitle and the target scene category; finally, in response to an instruction to manually correct the first multi-level subtitle, the remote sensing image subtitle prior knowledge is displayed, and the manual correction of the first multi-level subtitle based on the remote sensing image subtitle prior knowledge is received to obtain the target multi-level subtitle. Through the technical scheme of the present invention, the ground object elements are automatically deleted and supplemented based on the ground object element correction rule, the spatial relationship subtitles obtained based on the spatial relationship prior knowledge have more detailed information, the semantic accuracy of the subtitles is improved, the ground object elements, spatial relationship subtitles and target scene categories after automatic correction are independently composed of a first multi-level subtitle in multiple layers instead of integrating the three together, the richness and flexibility of the subtitle layer are improved, the first multi-level subtitles are manually corrected based on the remote sensing image subtitle prior knowledge, so that the obtained target multi-level subtitles can better express the characteristics of the remote sensing image, reduce repetition and supplement more detailed information, and improve the semantic quality of the subtitles.

[0037] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, another method for generating multi-level captions for remote sensing images based on hybrid intelligence is provided, such as Figure 2 As shown, the method includes:

[0038] 201. Obtain an initial remote sensing image data set and obtain prior knowledge of remote sensing image preprocessing.

[0039] For this embodiment, initial remote sensing images of any size with invalid images removed are collected and constructed into an initial remote sensing image dataset. Invalid images include those with wrong format, incomplete information, unable to be opened, and low definition.

[0040] Based on deep learning knowledge and geoscientific knowledge, the remote sensing image preprocessing prior knowledge is extracted in advance. Among them, the geoscientific knowledge includes the definition and relationship of ground features. Its introduction can ensure the correctness and interpretability of the generated results at the semantic level. It should be noted that the remote sensing image preprocessing prior knowledge includes multiple aspects, such as size. Here, the remote sensing image preprocessing prior knowledge on size is taken as an example for explanation, as shown in Table 1:

[0041] Table 1:

[0042]

[0043]

[0044]

[0045] 202. Preprocess the initial remote sensing image data set using preprocessing rules corresponding to the remote sensing image preprocessing prior knowledge to obtain a remote sensing image data set, wherein the preprocessing rules include preset large-size image slicing, preset medium-size image cropping, and preset small-size image filling.

[0046] For this embodiment, the preprocessing rules include radiation correction, geometric correction, band selection, etc. (not listed in Table 1 of step 201 of the embodiment), and the preprocessing rules also include size preprocessing rules, namely, preset large-size image slicing, preset medium-size image cropping, and preset small-size image filling.

[0047] The initial remote sensing image data set is preprocessed using the preprocessing rules corresponding to the remote sensing image preprocessing prior knowledge to obtain a remote sensing image data set. As an implementation method, the size of the initial remote sensing image in the initial remote sensing image data set is determined. If the size of the initial remote sensing image is larger than a preset large size threshold, it is confirmed as a preset large size image, and the preprocessing performed on it is slicing. If the size of the initial remote sensing image is larger than a preset medium size threshold and less than or equal to a preset large size threshold, it is confirmed as a preset medium size image, and the preprocessing performed on it is cropping. If the size of the initial remote sensing image is smaller than a preset small size threshold, it is confirmed as a preset small size image, and the preprocessing performed on it is filling. After preprocessing each initial remote sensing image in the initial remote sensing image data set, the corresponding remote sensing image is obtained, and these remote sensing images constitute the remote sensing image data set.

[0048] 203. For each remote sensing image in the remote sensing image data set, semantic segmentation is performed on the remote sensing image to obtain ground object elements corresponding to the remote sensing image, and scene classification is performed on the remote sensing image to obtain a target scene category corresponding to the remote sensing image.

[0049] For this embodiment, as an implementation method, a convolutional neural network is used to perform semantic segmentation on the remote sensing image to obtain the ground object elements corresponding to the remote sensing image. As another implementation method, a convolutional neural network can also be used to perform multi-label classification on the remote sensing image to obtain the ground object elements corresponding to the remote sensing image, wherein the convolutional neural network for multi-label classification can be any one of Deep Danbooru and Waifu Diffusion Tagger.

[0050] For this embodiment, as an implementation method, a convolutional neural network is used to classify the remote sensing image to obtain the target scene category corresponding to the remote sensing image. The convolutional neural network can be an EfficientNet model. As another implementation method, a Transformer deep learning algorithm is used to classify the remote sensing image to obtain the target scene category corresponding to the remote sensing image.

[0051] Among them, the EfficientNet model is:

[0052]

[0053]

[0054]

[0055]

[0056] Where d, w, r are coefficients for scaling the model width, depth, and resolution, respectively; ⊙ represents continuous operation; Indicates a specific operation; X represents the input tensor; They represent the height, width, and number of channels of X respectively; target_memory is the limited video memory size; target_flops is the limited computational effort.

[0057] 204. Obtain a set of corrected ground object elements corresponding to the ground object element correction rule, wherein the set of corrected ground object elements includes: a first non-ground object element that will not appear in the remote sensing image, a second non-ground object element that will not appear in each scene category, and a preset ground object element that will appear in each scene category with a probability greater than a preset probability.

[0058] 205. Delete the first non-ground feature element from the ground feature elements, select the target second non-ground feature element that will not appear in the target scene category from the second non-ground feature elements that will not appear in each of the scene categories, delete the target second non-ground feature element from the ground feature elements, select the target preset ground feature element under the target scene category from the preset ground feature elements that will appear with a probability greater than the preset probability in each of the scene categories, and supplement the target preset ground feature element to the ground feature elements to obtain the automatically corrected ground feature element corresponding to the remote sensing image.

[0059] Steps 204 and 205 of the embodiment belong to the process of automatic correction. Specifically, after deleting the first non-ground feature element from the ground feature element corresponding to the remote sensing image, deleting the target second non-ground feature element, and supplementing the target preset ground feature element, the automatically corrected ground feature element corresponding to the remote sensing image is obtained.

[0060] Among them, the first non-ground feature elements that will not appear in remote sensing images include: sky, clouds, fishing rods, rings, earrings, etc.; the second non-ground feature elements that will not appear in each scene category, for example, if the scene category is commercial area, then the second non-ground feature elements that will not appear in the commercial area include: tanks, submarines, stairs, waterfalls, chimneys, etc.; the preset ground feature elements that will appear with a probability greater than a preset probability in each scene category, for example, if the scene category is commercial area, then the preset ground feature elements that will appear with a probability greater than a preset probability in the commercial area include: clock towers, skyscrapers, roofs, streets, roads, houses, etc.

[0061] 206. Generate subtitles for the remote sensing image using a subtitle generation algorithm and spatial relationship prior knowledge to obtain spatial relationship subtitles corresponding to the remote sensing image.

[0062] For this embodiment, the subtitle generation algorithm and spatial relationship prior knowledge are used to generate subtitles for the remote sensing image to obtain spatial relationship subtitles corresponding to the remote sensing image, including: using the subtitle generation algorithm to generate subtitles for the remote sensing image, and after decoding, obtaining a subtitle generation result corresponding to the remote sensing image; automatically determining whether the subtitle generation result includes spatial relationship prior knowledge, and if so, determining the subtitle generation result including the spatial relationship prior knowledge as the spatial relationship subtitle corresponding to the remote sensing image; if not, using the subtitle generation algorithm to regenerate subtitles for the remote sensing image until the spatial relationship subtitles corresponding to the remote sensing image are obtained.

[0063] For this embodiment, spatial relation knowledge collection (SRKC) is extracted in advance, as shown in Table 2:

[0064] Table 2:

[0065]

[0066]

[0067] Among them, the subtitle generation algorithm may include BLIP (Bootstraping Language Image Pre-training), BLIP2 (Bootstrapping Language-Image Pre-training 2), GIT (Generative Image-to-text Transformer), InstructBLIP (Instruct Bootstrapping Language-Image Pre-training), ChatGPT (Chat Generative Pretrained Model), RSGPT (Remote Sensing Generative Pretrained Model), etc. After the subtitle generation, the output result of the subtitle generation algorithm corresponding to the remote sensing image is obtained, and the output result of the subtitle generation algorithm corresponding to the remote sensing image needs to be decoded to obtain the subtitle generation result corresponding to the remote sensing image, wherein the output result of the subtitle generation algorithm corresponding to the remote sensing image is decoded using a deterministic decoding algorithm and a random decoding algorithm, wherein the deterministic decoding algorithm may include greedy search and beam search, and the random decoding algorithm may include Top-k Sampling, Top-p Sampling, and Typical Sampling.

[0068] The subtitle generation results corresponding to the remote sensing image are multiple sentences. Since the sentences must contain prior knowledge of spatial relationships, the subtitle generation results are screened using prior knowledge of spatial relationships. Only the subtitle generation results that include prior knowledge of spatial relationships can be retained. The spatial relationship subtitles corresponding to the remote sensing image are finally obtained and used to describe the spatial relationship of the automatically corrected ground elements in the remote sensing image.

[0069] 207. Obtain a first multi-level subtitle according to the automatically corrected ground feature elements, the spatial relationship subtitles, and the target scene category.

[0070] For this embodiment, Figure 6 As shown, the order of element-relationship-scene corresponds to low-level semantics-middle-level semantics-high-level semantics. Accordingly, the first multi-level subtitles are obtained by combining the automatically corrected ground feature elements, the spatial relationship subtitles, and the target scene categories in sequence. Figure 7 As shown, the specific structure of the first multi-level subtitle is shown.

[0071] 208. In response to an instruction to manually correct the first multi-layer subtitles, display remote sensing image subtitle prior knowledge, receive manual correction of the first multi-layer subtitles based on the remote sensing image subtitle prior knowledge, and obtain target multi-layer subtitles.

[0072] For this embodiment, the receiving of manual correction of the first multi-level subtitles based on the prior knowledge of the remote sensing image subtitles to obtain the target multi-level subtitles includes: receiving a selection of the spatial relationship subtitles in the first multi-level subtitles based on the prior knowledge of the remote sensing image subtitles to obtain selected spatial relationship subtitles, and obtaining a second multi-level subtitle composed of the target scene category, the automatically corrected ground object elements and the selected spatial relationship subtitles; receiving inaccurate subtitle correction of the second multi-level subtitles based on the prior knowledge of the remote sensing image subtitles, the ground object attribute knowledge and the spatial relationship of the unified subtitles in eight fields to obtain a third multi-level subtitle; receiving erroneous subtitle deletion and duplicate subtitle deletion of the third multi-level subtitles based on the prior knowledge of the remote sensing image subtitles to obtain a fourth multi-level subtitle; receiving missing ground object subtitle supplement of the fourth multi-level subtitles based on the prior knowledge of the remote sensing image subtitles to obtain the target multi-level subtitles.

[0073] For this embodiment, the remote sensing image caption prior knowledge is extracted in advance based on the image knowledge and the geological knowledge, and is used to manually correct the first multi-level caption. The geological knowledge includes the definition and relationship of the ground features, and the introduction of the knowledge can ensure the correctness and interpretability of the generated results at the semantic level, as shown in Table 3:

[0074] Table 3:

[0075]

[0076]

[0077]

[0078] For this embodiment, the spatial relationship subtitles include multiple sentences, but not all sentences can fully express the characteristics of remote sensing images. On the basis of the first multi-level subtitles of spatial relationship subtitles, the subtitles that can best express the characteristics of remote sensing images are selected according to the prior knowledge of remote sensing image subtitles to obtain the selected spatial relationship subtitles, and the second multi-level subtitles composed of the target scene category, the automatically corrected ground object elements and the selected spatial relationship subtitles are obtained.

[0079] For this embodiment, on the basis of the second and multi-level subtitles, the inaccurate subtitles of the second and multi-level subtitles are corrected according to the prior knowledge of remote sensing image subtitles, the knowledge of land feature attributes and the spatial relationship of eight-field unified subtitles to obtain third-level subtitles. Among them, the land feature attribute knowledge GEAKC (geographic element attribute knowledge collection) includes: unified words under the attribute categories of land features such as color, shape, size, quantity, age, density, and productivity. The second and multi-level subtitles all use unified words under the corresponding attribute categories. If they are not used, they will be corrected. Figure 8 As shown, the setting of the eight-domain unified subtitle spatial relationship is to ensure the unity of the spatial relationship and the specificity of the spatial relationship. The second and multi-level subtitles all use the image-centered eight-domain spatial relationship subtitles of the absolute position of the ground objects in the remote sensing image. If it is not adopted, it will be corrected.

[0080] For this embodiment, based on the third multi-level subtitles, the erroneous subtitles and the duplicated subtitles of the third multi-level subtitles are deleted according to the prior knowledge of remote sensing image subtitles to obtain the fourth multi-level subtitles.

[0081] In this embodiment, on the basis of the fourth multi-level subtitles, the missing feature subtitles of the fourth multi-level subtitles are supplemented according to the prior knowledge of remote sensing image subtitles to obtain the target multi-level subtitles.

[0082] In summary, the manual correction process is based on a human-in-the-loop approach, in which the first multi-level subtitles automatically generated are corrected and optimized in the order of selection-correction-deletion-supplementation, so that the target multi-level subtitles obtained are more accurate, complete and interpretable. Fig. 9 AAAAD in the formula indicates that Fig. 9SYDNEY, UCM, RSICD, RSITMD, and NWPU in the figure are all existing remote sensing image subtitles. By comparison, it can be seen that the target multi-level subtitles obtained in the embodiment of the present application first describe the overall scene and perspective of the remote sensing image, and then specifically describe the specific object, and are organized in the order of element-relation-scene (ERS). Based on hybrid intelligence (hybrid intelligence is an intelligent computing paradigm that combines artificial intelligence with expert knowledge. By integrating expert knowledge, hybrid intelligence can improve the rationality and interpretability of generated results), the remote sensing image is described comprehensively, accurately and meticulously at multiple levels, which can effectively improve the semantic quality, details and diversity of remote sensing image subtitles, realize the systematic and multi-angle characterization of complex remote sensing images, and open up a new research direction for the intelligent interpretation and understanding of complex remote sensing images. It can not only correctly reflect the connection between ground objects, but also ensure the integrity of ground objects, and realize the system's accurate identification of the functional effects of remote sensing images, and finally realize multi-level accurate cognition of the entire scene, which is of great significance to promoting the development of intelligent analysis of remote sensing images.

[0083] And because the three levels of elements-relationships-scenes in this application are independent of each other, users can respond to different application scenarios by selecting any level of elements-relationships-scenes, or sorting elements-relationships-scenes. However, existing remote sensing image subtitles integrate elements, relationships and scenes into one sentence, which cannot be directly selected or sorted automatically by a computer. This method improves the flexibility of target remote sensing image subtitles.

[0084] Existing remote sensing image captions only describe the most significant and main parts of the remote sensing images, while the present application includes secondary information based on the automatic supplementation of step 205 of the embodiment and the supplementation of missing land feature captions of step 208 of the embodiment.

[0085] Existing remote sensing image subtitles only roughly describe the spatial relationship. For example, the airplane is next to the white building, but there is no specific description of the specific location around the building. That is, the granularity is coarse and the subtitles are simple. The present application expresses information based on multiple attribute categories of the feature attribute knowledge in step 208 of the embodiment. The unified spatial relationship of the subtitles in eight fields ensures the unity of the spatial relationship and the specificity of the spatial relationship, thereby improving the complexity and meticulousness of the subtitles and improving the semantic quality.

[0086] Existing remote sensing image subtitles repeatedly represent information with the same meaning and contain errors. However, the present application automatically deletes information based on step 205 of the embodiment and selects subtitles in the spatial relationship subtitles that best express the characteristics of the remote sensing image based on step 208 of the embodiment, thereby deleting erroneous subtitles and duplicate subtitles, thereby improving the semantic quality.

[0087] The present invention provides a method and device for generating multi-level subtitles for remote sensing images based on hybrid intelligence. Through the technical solution of the present invention, a remote sensing image data set is obtained based on the prior knowledge of remote sensing image preprocessing, and the ground object elements are automatically deleted and supplemented based on the ground object element correction rule. The deletion reduces the erroneous information, and the supplement enriches the information. The spatial relationship subtitles obtained based on the prior knowledge of spatial relationship have more detailed information, which improves the semantic accuracy of the subtitles. The ground object elements, spatial relationship subtitles and target scene categories after automatic correction are independently composed of the first multi-level subtitles in multiple layers, rather than merging the three together, which improves the subtitles. The richness and flexibility of the levels, the first multi-level subtitles are manually corrected based on the prior knowledge of remote sensing image subtitles, so that the target multi-level subtitles can better express the characteristics of the remote sensing image, reduce repetition and supplement more detailed information, and improve the semantic quality of the subtitles. In summary, based on the combination of artificial intelligence and expert knowledge (expert knowledge includes: prior knowledge of remote sensing image preprocessing, prior knowledge of spatial relationships and prior knowledge of remote sensing image subtitles), a comprehensive, accurate and detailed description of remote sensing images at multiple levels can effectively improve the semantic quality, details and diversity of remote sensing image subtitles, and realize a systematic and multi-angle characterization of complex remote sensing images.

[0088] Further, as Figure 1 and Figure 2 The specific implementation of the method shown in the embodiment of the present invention provides a remote sensing image multi-level subtitle generation device based on hybrid intelligence, such as Figure 3 As shown, the device includes: an acquisition module 31, a generation module 32, and a display module 33;

[0089] The acquisition module 31 may be used to acquire a remote sensing image data set, and for each remote sensing image in the remote sensing image data set, determine the ground object element and the target scene category corresponding to the remote sensing image;

[0090] A generating module 32 is configured to automatically correct the ground object elements according to the target scene category and the ground object element correction rule to obtain the automatically corrected ground object elements corresponding to the remote sensing image, generate subtitles for the remote sensing image using a subtitle generation algorithm and spatial relationship prior knowledge to obtain spatial relationship subtitles corresponding to the remote sensing image, and obtain a first multi-level subtitle according to the automatically corrected ground object elements, the spatial relationship subtitles and the target scene category;

[0091] The display module 33 can be used to respond to the instruction of manually correcting the first multi-level subtitles, display the remote sensing image subtitle prior knowledge, receive the manual correction of the first multi-level subtitles based on the remote sensing image subtitle prior knowledge, and obtain the target multi-level subtitles.

[0092] Correspondingly, in order to obtain a remote sensing image data set, the acquisition module 31 can be specifically used to obtain an initial remote sensing image data set and obtain remote sensing image preprocessing prior knowledge; the initial remote sensing image data set is preprocessed using preprocessing rules corresponding to the remote sensing image preprocessing prior knowledge to obtain a remote sensing image data set, wherein the preprocessing rules include preset large-size image slicing, preset medium-size image cropping, and preset small-size image filling.

[0093] Accordingly, in order to determine the ground object elements and target scene categories corresponding to the remote sensing image, the acquisition module 31 can be specifically used to perform semantic segmentation on the remote sensing image to obtain the ground object elements corresponding to the remote sensing image; and perform scene classification on the remote sensing image to obtain the target scene category corresponding to the remote sensing image.

[0094] Accordingly, in order to automatically correct the ground object elements according to the target scene category and the ground object element correction rule to obtain the automatically corrected ground object elements corresponding to the remote sensing image, the generation module 32 may specifically include: an acquisition unit 321 and a correction unit 322;

[0095] The acquisition unit 321 may be used to acquire a set of corrected ground object elements corresponding to the ground object element correction rule, wherein the set of corrected ground object elements includes: a first non-ground object element that will not appear in the remote sensing image, a second non-ground object element that will not appear in each scene category, and a preset ground object element that will appear with a probability greater than a preset probability in each scene category;

[0096] The correction unit 322 can be used to delete the first non-ground object element in the ground object elements, filter out the target second non-ground object elements that will not appear in the target scene category from the second non-ground object elements that will not appear in each of the scene categories, delete the target second non-ground object elements in the ground object elements, filter out the target preset ground object elements in the target scene category from the preset ground object elements that appear with a probability greater than the preset probability in each of the scene categories, and supplement the target preset ground object elements in the ground object elements to obtain the automatically corrected ground object elements corresponding to the remote sensing image.

[0097] Accordingly, in order to generate subtitles for the remote sensing image using a subtitle generation algorithm and spatial relationship prior knowledge and obtain spatial relationship subtitles corresponding to the remote sensing image, a generation module 32 can be specifically used to generate subtitles for the remote sensing image using a subtitle generation algorithm, and after decoding, obtain a subtitle generation result corresponding to the remote sensing image; automatically determine whether the subtitle generation result includes spatial relationship prior knowledge, and if so, determine the subtitle generation result including the spatial relationship prior knowledge as the spatial relationship subtitle corresponding to the remote sensing image; if not, regenerate subtitles for the remote sensing image using the subtitle generation algorithm until the spatial relationship subtitles corresponding to the remote sensing image are obtained.

[0098] Correspondingly, in order to receive the manual correction of the first multi-level subtitles based on the prior knowledge of the remote sensing image subtitles and obtain the target multi-level subtitles, the display module 33 can be specifically used to receive the selection of the spatial relationship subtitles in the first multi-level subtitles based on the prior knowledge of the remote sensing image subtitles to obtain the selected spatial relationship subtitles, and obtain the second multi-level subtitles composed of the target scene category, the automatically corrected ground object elements and the selected spatial relationship subtitles; receive the inaccurate subtitle correction of the second multi-level subtitles based on the prior knowledge of the remote sensing image subtitles, the ground object attribute knowledge and the unified spatial relationship of subtitles in eight fields to obtain the third multi-level subtitles; receive the deletion of erroneous subtitles and duplicate subtitles of the third multi-level subtitles based on the prior knowledge of the remote sensing image subtitles to obtain the fourth multi-level subtitles; receive the supplement of missing ground object subtitles of the fourth multi-level subtitles based on the prior knowledge of the remote sensing image subtitles to obtain the target multi-level subtitles.

[0099] Correspondingly, in order to obtain the first multi-level subtitles according to the automatically corrected ground feature elements, the spatial relationship subtitles and the target scene categories, the display module 33 can be specifically used to obtain the first multi-level subtitles by combining the automatically corrected ground feature elements, the spatial relationship subtitles and the target scene categories in sequence.

[0100] It should be noted that for other corresponding descriptions of the functional units involved in the hybrid intelligent remote sensing image multi-level caption generation device provided in this embodiment, please refer to Figure 1 to Figure 2 The corresponding description will not be repeated here.

[0101] Based on the above Figure 1 to Figure 2 The method shown in the embodiment, accordingly, also provides a storage medium, which can be volatile or non-volatile, and stores a computer program, which is executed by a processor to implement the above-mentioned Figure 1 to Figure 2 The multi-level subtitle generation method for remote sensing images based on hybrid intelligence is shown.

[0102] Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various implementation scenarios of the present invention.

[0103] Based on the above Figure 1 to Figure 2 The method shown and Figure 3 , Figure 4 In order to achieve the above-mentioned purpose, the present embodiment further provides a computer device, which includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 to Figure 2 The multi-level subtitle generation method for remote sensing images based on hybrid intelligence is shown.

[0104] Optionally, the computer device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0105] Those skilled in the art will appreciate that the computer device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0106] The storage medium may also include an operating network communication module. The operating is a program that manages the hardware and software resources of the above-mentioned computer device, supporting the operation of the information processing program and other software and / or programs. The network communication module is used to communicate between the components inside the storage medium, and to communicate with other hardware and software in the information processing entity device.

[0107] Through the description of the above implementation modes, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform, or by hardware.

[0108] The present invention provides a method and device for generating multi-level subtitles for remote sensing images based on hybrid intelligence. Through the technical solution of the present invention, ground object elements are automatically deleted and supplemented based on ground object element correction rules. The spatial relationship subtitles obtained based on spatial relationship prior knowledge have more detailed information, which improves the semantic accuracy of the subtitles. The ground object elements, spatial relationship subtitles and target scene categories after automatic correction are independently composed of a first multi-level subtitle in multiple levels instead of being integrated together, which improves the richness and flexibility of the subtitle levels. The first multi-level subtitles are manually corrected based on remote sensing image subtitle prior knowledge, so that the obtained target multi-level subtitles can better express the characteristics of the remote sensing image, reduce repetition and supplement more detailed information, thereby improving the semantic quality of the subtitles.

[0109] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present invention. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0110] The above serial numbers of the present invention are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of the present invention, but the present invention is not limited thereto, and any changes that can be thought of by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A method for generating multi-level captions for remote sensing images based on hybrid intelligence, characterized in that: The method comprises: Acquire a remote sensing image data set, and for each remote sensing image in the remote sensing image data set, determine a ground object element and a target scene category corresponding to the remote sensing image; Automatically correcting the ground object elements according to the target scene category and the ground object element correction rule to obtain the automatically corrected ground object elements corresponding to the remote sensing image, including: obtaining a corrected ground object element set corresponding to the ground object element correction rule, the corrected ground object element set including: a first non-ground object element that will not appear in the remote sensing image, a second non-ground object element that will not appear in each scene category, and a preset ground object element that will appear with a probability greater than a preset probability in each scene category; deleting the first non-ground object element in the ground object elements, screening the target second non-ground object element that will not appear in the target scene category from the second non-ground object element that will not appear in each scene category, deleting the target second non-ground object element in the ground object elements, screening the target preset ground object element under the target scene category from the preset ground object elements that will appear with a probability greater than the preset probability in each scene category, and supplementing the target preset ground object element in the ground object elements to obtain the automatically corrected ground object elements corresponding to the remote sensing image; Generate subtitles for the remote sensing image using a subtitle generation algorithm and spatial relationship prior knowledge to obtain spatial relationship subtitles corresponding to the remote sensing image, and obtain a first multi-level subtitle according to the automatically corrected ground feature elements, the spatial relationship subtitles and the target scene category; In response to an instruction to manually correct the first multi-layer subtitles, prior knowledge of remote sensing image subtitles is displayed, and manual correction of the first multi-layer subtitles based on the prior knowledge of remote sensing image subtitles is received to obtain target multi-layer subtitles.

2. The method according to claim 1, characterized in that The step of acquiring a remote sensing image dataset comprises: Obtain the initial remote sensing image data set and obtain the prior knowledge of remote sensing image preprocessing; The initial remote sensing image data set is preprocessed using preprocessing rules corresponding to the remote sensing image preprocessing prior knowledge to obtain a remote sensing image data set, wherein the preprocessing rules include preset large-size image slicing, preset medium-size image cropping, and preset small-size image filling.

3. The method according to claim 1, characterized in that The determining of the ground object elements and the target scene category corresponding to the remote sensing image includes: Performing semantic segmentation on the remote sensing image to obtain ground object elements corresponding to the remote sensing image; The remote sensing image is subjected to scene classification to obtain a target scene category corresponding to the remote sensing image.

4. The method according to claim 1, characterized in that: The step of generating subtitles for the remote sensing image using a subtitle generation algorithm and spatial relationship prior knowledge to obtain spatial relationship subtitles corresponding to the remote sensing image includes: Generating subtitles for the remote sensing image using a subtitle generation algorithm, and obtaining a subtitle generation result corresponding to the remote sensing image after decoding; Automatically determine whether the subtitle generation result includes spatial relationship prior knowledge, and if so, determine the subtitle generation result including the spatial relationship prior knowledge as the spatial relationship subtitle corresponding to the remote sensing image; If not, the subtitle generation algorithm is used to regenerate subtitles for the remote sensing image until spatial relationship subtitles corresponding to the remote sensing image are obtained.

5. The method according to claim 1, characterized in that The receiving of the manual correction of the first multi-level subtitles based on the prior knowledge of the remote sensing image subtitles to obtain the target multi-level subtitles includes: receiving a selection of the spatial relationship subtitles in the first multi-level subtitles based on the prior knowledge of the remote sensing image subtitles, obtaining selected spatial relationship subtitles, and obtaining a second multi-level subtitle consisting of the target scene category, the automatically corrected ground feature elements, and the selected spatial relationship subtitles; receiving inaccurate subtitle corrections to the second multi-level subtitles based on the prior knowledge of remote sensing image subtitles, knowledge of ground object attributes, and spatial relationships of unified subtitles in eight fields, to obtain third multi-level subtitles; receiving, based on the prior knowledge of remote sensing image subtitles, deletion of erroneous subtitles and duplicated subtitles of the third multi-level subtitles to obtain a fourth multi-level subtitle; The missing feature subtitles of the fourth multi-level subtitles are supplemented based on the prior knowledge of the remote sensing image subtitles to obtain target multi-level subtitles.

6. The method according to claim 1, characterized in that The step of obtaining a first multi-level subtitle according to the automatically corrected ground feature elements, the spatial relationship subtitles, and the target scene category includes: The first multi-level subtitles are obtained by combining the automatically corrected ground feature elements, the spatial relationship subtitles and the target scene categories in sequence.

7. A remote sensing image multi-level subtitle generation device based on hybrid intelligence, characterized in that: The device comprises: An acquisition module is used to acquire a remote sensing image data set, and for each remote sensing image in the remote sensing image data set, determine the ground object element and target scene category corresponding to the remote sensing image; A generation module, configured to automatically correct the ground object elements according to the target scene category and the ground object element correction rule to obtain the automatically corrected ground object elements corresponding to the remote sensing image, generate subtitles for the remote sensing image using a subtitle generation algorithm and spatial relationship prior knowledge to obtain spatial relationship subtitles corresponding to the remote sensing image, and obtain a first multi-level subtitle according to the automatically corrected ground object elements, the spatial relationship subtitles and the target scene category; a display module, configured to respond to an instruction to manually correct the first multi-layered subtitles, display the remote sensing image subtitle prior knowledge, receive the manual correction of the first multi-layered subtitles based on the remote sensing image subtitle prior knowledge, and obtain the target multi-layered subtitles; The generation module is further used to obtain a set of corrected ground object elements corresponding to the ground object element correction rule, wherein the set of corrected ground object elements includes: a first non-ground object element that will not appear in the remote sensing image, a second non-ground object element that will not appear in each scene category, and a preset ground object element that will appear with a probability greater than a preset probability in each of the scene categories; delete the first non-ground object element from the ground object elements, filter out target second non-ground object elements that will not appear in the target scene category from the second non-ground object elements that will not appear in each of the scene categories, delete the target second non-ground object element from the ground object elements, filter out target preset ground object elements in the target scene category from the preset ground object elements that will appear with a probability greater than the preset probability in each of the scene categories, and supplement the target preset ground object elements in the ground object elements to obtain the automatically corrected ground object elements corresponding to the remote sensing image.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating multi-level subtitles for remote sensing images based on hybrid intelligence as described in any one of claims 1 to 6 is implemented.

9. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method for generating multi-level subtitles for remote sensing images based on hybrid intelligence as described in any one of claims 1 to 6 is implemented.

Images