A remote sensing image multi-task annotation method, device, storage medium and equipment

Abstract

In a remote sensing image multi-task annotation method, device, storage medium and equipment provided in the present specification, a remote sensing image is acquired, and each image slice is obtained by slicing. For each image slice, a classification label is determined by an image classification model and a segmentation mask is determined by an image segmentation model. According to an interpretation prompt text containing a graphic interpretation requirement preset in the image slice, a graphic interpretation text of the image slice is obtained through a graphic interpretation model. The graphic interpretation requirement at least includes a positional relationship between each scene area in the image slice and the position, image-level overview, target-level overview and target area shape of each scene area in the image slice. According to the classification label, segmentation mask and graphic interpretation text of the image slice, result annotation information is determined. Through the image classification model, image segmentation model and graphic interpretation model, multi-task annotation of remote sensing images is realized, and annotation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of image annotation, and in particular to a remote sensing image multi-task annotation method, device, storage medium and equipment. Background Art

[0002] With the promotion of major national tasks in the field of remote sensing, such as the National High-Resolution Earth Observation Major Science and Technology Project, the acquisition of geospatial data has become more and more abundant. This rich data provides strong support for the development of remote sensing multimodal basic large models, among which, remote sensing multimodal basic large models are a pre-trained general model that can process remote sensing images and perform a variety of tasks. However, this remote sensing multimodal basic large model usually has billions of parameters, and in order to achieve its expected performance, at least millions of annotated images are required for training.

[0003] At present, the traditional manual labeling method requires labelers to label each image one by one, but this manual labeling method is costly and has low labeling efficiency.

[0004] Based on this, this specification provides a remote sensing image multi-task annotation method, device, storage medium and equipment. Summary of the invention

[0005] This specification provides a remote sensing image multi-task annotation method and device to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a remote sensing image multi-task annotation method, the method comprising:

[0008] Acquire a remote sensing image, and slice the remote sensing image to obtain image slices to be labeled;

[0009] For each image slice, determine a classification label of the image slice by using an image classification model, and determine a segmentation mask of the image slice by using an image segmentation model;

[0010] According to the image slice and the preset interpretation prompt text containing the image-text interpretation requirements, a graphic-text interpretation text of the image slice is obtained through a graphic-text interpretation model, wherein the graphic-text interpretation requirements at least include the positional relationship between the objects in the image slice and the position of the objects in the image slice, the image-level overview, the object-level overview and the shape of the object area;

[0011] The result annotation information of the image slice is determined according to the classification label, segmentation mask and graphic interpretation text of the image slice.

[0012] Optionally, slicing the remote sensing image to obtain image slices to be labeled specifically includes:

[0013] Performing radiation correction and orthorectification on the remote sensing image;

[0014] According to a preset image size, the corrected remote sensing image is sliced ​​to obtain image slices to be labeled.

[0015] Optionally, slicing the remote sensing image to obtain image slices to be labeled specifically includes:

[0016] Acquire a multispectral image and a panchromatic image corresponding to the remote sensing image;

[0017] The multispectral image and the panchromatic image are fused to obtain a fused remote sensing image;

[0018] The fused remote sensing image is sliced ​​to obtain image slices to be labeled.

[0019] Optionally, the classification label of the image slice is determined by an image classification model, specifically including:

[0020] The image slice and the preset classification prompt text are input into the image classification model to obtain the classification label of the image slice output by the image classification model.

[0021] Optionally, determining a segmentation mask of the image slice by using an image segmentation model specifically includes:

[0022] The image slice is input into the target detection model to obtain a candidate annotation box for the image slice;

[0023] The image slice marked with the labeled candidate box is input into an image segmentation model, and the segmentation mask in each labeled candidate box is determined by the image segmentation model to obtain the segmentation mask of the image slice output by the image classification model.

[0024] Optionally, the result annotation information of the image slice is determined according to the classification label, segmentation mask and graphic interpretation text of the image slice, specifically including:

[0025] The classification label, segmentation mask and graphic interpretation text of the image slice are used as the initial annotation information of the image slice;

[0026] Determine adjustment notification information including the image slice and the initial annotation information of the image slice, and send the information to the first user, wherein the adjustment notification information is used to notify the first user to adjust the initial annotation information based on the image slice;

[0027] The information returned by the first user is obtained as the result annotation information of the image slice.

[0028] Optionally, the method further includes:

[0029] Determine review notification information including the result annotation information of the image slice, and send the information to the second user, wherein the review notification information is used to notify the second user to review the result annotation information of the image slice;

[0030] responding to the review result of the result annotation information returned by the second user;

[0031] When the audit result is not passed, the result annotation information of the image slice and the image slice are resent to the first user; when the audit result is passed, the result annotation information is stored in a preset storage format.

[0032] This specification provides a remote sensing image multi-task annotation device, including:

[0033] A slicing module, used to acquire a remote sensing image and slice the remote sensing image to obtain image slices to be labeled;

[0034] A first labeling module is used to determine, for each image slice, a classification label of the image slice through an image classification model, and to determine a segmentation mask of the image slice through an image segmentation model;

[0035] A second annotation module is used to obtain a graphic-text interpretation text of the image slice through a graphic-text interpretation model according to the image slice and a preset interpretation prompt text containing graphic-text interpretation requirements, wherein the graphic-text interpretation requirements at least include a positional relationship between various objects in the image slice and the position of each object in the image slice, an image-level overview, an object-level overview, and a shape of a target area;

[0036] The determination module is used to determine the result annotation information of the image slice according to the classification label, segmentation mask and graphic interpretation text of the image slice.

[0037] The present specification provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, a remote sensing image multi-task annotation method is implemented.

[0038] The present specification provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, a remote sensing image multi-task annotation method is implemented.

[0039] At least one of the above-mentioned technical solutions adopted in the present specification can achieve the following beneficial effects: in a remote sensing image multi-task annotation method provided in the present specification, a remote sensing image is acquired and sliced ​​to obtain image slices to be annotated. For each image slice, the classification label of the image slice is determined by an image classification model and the segmentation mask of the image slice is determined by an image segmentation model. According to the image slice and a preset interpretation prompt text containing a graphic interpretation requirement, a graphic interpretation text of the image slice is obtained by a graphic interpretation model. The graphic interpretation requirement includes at least a positional relationship between scene areas in the image slice and the position, image-level overview, target-level overview and target area shape of each scene area in the image slice. According to the classification label, segmentation mask and graphic interpretation text of the image slice, the result annotation information of the image slice is determined.

[0040] It can be seen from the above method that by slicing the remote sensing image to obtain the image slices to be labeled, and through the image classification model, image segmentation model and graphic interpretation model, multi-task automatic labeling of remote sensing images is realized, which reduces labor costs and improves labeling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0042] Figure 1 A flowchart of a multi-task annotation method for remote sensing images provided in this specification;

[0043] Figure 2 A schematic diagram of the structure of a remote sensing image multi-task annotation method provided in this specification;

[0044] Figure 3 A schematic diagram of the manual verification and correction process provided in this manual;

[0045] Figure 4 A schematic diagram of a remote sensing image multi-task annotation device provided in this specification;

[0046] Figure 5 A schematic diagram of the electronic device structure corresponding to a remote sensing image multi-task annotation method provided in this specification. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0048] The process of executing a remote sensing image multi-task annotation method in this specification involves the processing of image data. Therefore, in one or more embodiments of this specification, it is not limited by which device the process of a remote sensing image multi-task annotation method is implemented, such as a personal computer, a mobile terminal, and a server. However, since the subsequent steps involve operations such as data processing, and such operations with high requirements for computing resources are generally performed by a server, a remote sensing image multi-task annotation method is described as an example. Among them, the server can be a single device, or composed of multiple devices, such as a distributed server, and this specification does not limit this. For the convenience of description, the following description is based on the server as the execution subject.

[0049] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0050] Figure 1 The flowchart of a multi-task annotation method for remote sensing images provided in this specification includes the following steps:

[0051] S100: Acquire a remote sensing image, and slice the remote sensing image to obtain image slices to be labeled.

[0052] In one or more embodiments of the present specification, in order to obtain the result annotation information of the remote sensing image under multiple tasks through image classification, image segmentation and image-text interpretation in the subsequent steps. In this step, the server needs to obtain the remote sensing image and slice the remote sensing image to obtain each image slice to be labeled.

[0053] Specifically, the server may acquire a remote sensing image, and slice the remote sensing image to obtain image slices to be labeled.

[0054] It should be noted that the type and source of remote sensing images are not limited in this specification. For example, the source may be drone aerial photography, high-resolution satellite data or open source remote sensing data sets, and the type may be visible light images, infrared images, multispectral images, panchromatic images, etc. At the same time, the specific method of slicing remote sensing images is not limited in this specification, and can be set according to actual needs, such as slicing remote sensing images with a fixed image size, or slicing remote sensing images with a preset step size. At the same time, before slicing the remote sensing image, the remote sensing image may be preprocessed to improve the quality of the remote sensing image. In one or more embodiments of this specification, the server may correct the remote sensing image through a preprocessing operation, and slice the corrected remote sensing image according to a preset image size to obtain each image slice to be labeled. The specific content of the preprocessing operation is not limited in this specification. For example, the preprocessing operation may include radiation correction, orthorectification, geometric correction, etc. Radiation correction is used to correct the influence of atmospheric scattering and absorption on the image radiation value, and orthorectification is used to eliminate geometric deformation caused by factors such as sensor attitude or terrain undulation. Of course, in another embodiment, the server may also obtain a multispectral image and a panchromatic image corresponding to the remote sensing image before slicing the remote sensing image, fuse the multispectral image and the panchromatic image to obtain a fused remote sensing image, and slice the fused remote sensing image according to a preset image size to obtain image slices to be annotated. Image fusion is used to combine a high-resolution panchromatic image and a multi-band multispectral image to generate a high-resolution multispectral image. The preset image size is not limited in this specification and can be set according to actual needs.

[0055] S102: For each image slice, determine the classification label of the image slice through an image classification model, and determine the segmentation mask of the image slice through an image segmentation model.

[0056] In one or more embodiments of the present specification, in order to determine the result annotation information of the image slice in the subsequent steps. In this step, the server needs to determine the classification label of each image slice through the image classification model, and perform image segmentation on the image slice through the image segmentation model to obtain the segmentation mask of the image slice.

[0057] Specifically, the server may determine the classification label of each image slice through the image classification model, and determine the segmentation mask of the image slice through the image segmentation model. In this specification, the number of classification labels determined is not limited, and the maximum number of classification labels may be limited to focus on the main scenes in the image slice.

[0058] It should be noted that in this specification, the classification label may include a parent label and a sub-category label, and each parent label includes one or more sub-category labels. For example, the classification label may include parent labels such as agriculture, commerce, industry, residence, transportation, water area, and undeveloped. Agriculture may include sub-category labels such as forest land, grassland, and cultivated land; commerce may include sub-category labels such as commercial area; industry may include sub-category labels such as industrial area, mining area, and power station; residence may include sub-category labels such as residential area; transportation may include sub-category labels such as airport, road, port area, bridge, railway station, and viaduct; water area may include sub-category labels such as open water area and small water body. Of course, the format of determining the classification label of the image slice is not limited in the specification, such as the format may be "parent label / sub-category label". The multi-level label format of "parent label / sub-category label" makes the classification label of the image slice clearer and reflects the hierarchical structure of the scene contained in the image slice. The types of image classification models and image segmentation models are not limited in this specification, and can be set according to actual needs.

[0059] S104: According to the image slice and a preset interpretation prompt text containing graphic interpretation requirements, a graphic interpretation text of the image slice is obtained through a graphic interpretation model, wherein the graphic interpretation requirements at least include the positional relationship between each target in the image slice and the position of each target in the image slice, an image-level overview, a target-level overview, and a target area shape.

[0060] In one or more embodiments of the present specification, in order to determine the result annotation information of the image slice in the subsequent steps, in this step, the server needs to determine the image-text interpretation text of the image slice through the image-text interpretation model according to the image slice and the preset image-text interpretation requirements.

[0061] Specifically, the server can input the image slice and the preset explanatory prompt text containing the graphic interpretation requirements into the graphic interpretation model, thereby obtaining the graphic interpretation text of the image slice, wherein the graphic interpretation requirements include at least the positional relationship between the targets in the image slice and the position of each target in the image slice, the image-level overview, the target-level overview and the target area shape.

[0062] It should be noted that the type of image-text interpretation model is not limited in this specification and can be set according to actual needs. For example, the image-text interpretation model can use the existing large model of visual language to save costs. The "positional relationship between each target in the image slice and the position of each target in the image slice" in the image-text interpretation requirements is used to constrain the relative positional relationship between each target in the image slice output by the image-text interpretation model and the absolute position of each target in the image slice; the "image-level overview" in the image-text interpretation requirements is used to constrain the overall content description of the image slice output by the image-text interpretation model; the "target-level overview" in the image-text interpretation requirements is used to constrain the detailed description of each target in the image slice output by the image-text interpretation model; the "target area shape" in the image-text interpretation requirements is used to constrain the contour shape of each target in the image slice output by the image-text interpretation model. Of course, the image-text interpretation requirements can also include basic attributes such as image source, image type, image resolution, and regional attributes such as target area color and number of target areas. In order to facilitate the storage and subsequent management of the graphic interpretation text, the graphic interpretation requirement may also include a text format requirement, which is used to constrain the graphic interpretation model to output the graphic interpretation text of the text format requirement. Of course, the specific content of the graphic interpretation requirement is not limited in this specification and can be set according to actual needs. At the same time, in order to facilitate the graphic interpretation model to output the graphic interpretation text that conforms to the expected framework, the interpretation instruction may also include an example of the graphic interpretation text.

[0063] The preset interpretation prompt text containing graphic interpretation requirements is as follows:

[0064] "You are a remote sensing expert. You have received an image slice of a high-resolution multispectral satellite remote sensing image. Please provide an image-level overview of the image slice and a target-level overview of each target in the image slice. Please describe in detail the image source, image type, image resolution, target area shape, target area color, target area number, the positional relationship between the scene areas in the image slice, and the position of each scene area in the image slice. Note that all visible land use types must be described separately. The priority land use types include: forest land, cultivated land, grassland, rivers, lakes, ponds, bare land, industrial plants, commercial areas, urban and rural residential areas, country roads, urban residential areas, urban roads, highways, railways, deserts, ships, airplanes, airport terminals, runways, container terminals, viaducts, wetlands, parks, snowfields, and bridges. At the same time, the description of the number of target areas cannot be described as "the specific number cannot be determined, but it looks like there are a lot of them". Try to give an estimated specific value. The text format of the output graphic interpretation text is required to be in XML format. Here, the following is an example for reference:

[0065] <root>

[0066] <summary> This image slice from a high-resolution multispectral satellite remote sensing image from a satellite perspective shows a series of rectangular ponds that may be used for aquaculture or salt pans. The ponds appear in different shades of green and brown, indicating differences in water depth or vegetation / algae types. There are paths separating the ponds, and a larger irregularly shaped body of water is visible on the right. The upper left corner of this image slice shows a natural waterway winding through the area. This image may be interesting because it shows human use of the land for aquaculture or salt production, which can be identified by the geometric shapes that contrast with the natural environment.< / summary>

[0067] <landuse>

[0068] <scene>

[0069] <class> pond< / class>

[0070] The rectangular ponds in the image are used for aquaculture or salt pans. They are numerous and distributed in an orderly manner.

[0071] <object>

[0072] <number> There are about 30 ponds< / number>

[0073] <color> Ponds appear in varying shades of green and brown, larger bodies of water are dark blue, and natural waterways are lighter blue.< / color>

[0074] <shape> The ponds are rectangular and uniform in size.< / shape>

[0075] <space relationship> The ponds are evenly distributed across the image, and the paths are located between the ponds.

[0076] < / object>

[0077] < / scene>

[0078] <scene>

[0079] <class> the way< / class>

[0080] There are paths separating the ponds.

[0081] <object>

[0082] <number> There are approximately 36 trails< / number>

[0083] <color> The path is off-white.< / color>

[0084] <shape> Most of the road paths are straight and vary in width.< / shape>

[0085] <spacerelationship> The path is located between the ponds. There are about 5 ponds on the left side of the main road, and more ponds on the right side of the road. The natural waterway is located below the main road.< / spacerelationship>

[0086] < / object>

[0087] < / scene>

[0088] < / landuse>

[0089] < / root> ”

[0090] It should be noted that "You are a remote sensing expert. You have received an image slice of a high-resolution multispectral satellite remote sensing image. Please provide an image-level overview of the image slice and a target-level overview of each target in the image slice" is used to inform the graphic interpretation model of the role and tasks it needs to play, so that the graphic interpretation model can understand the task background and task objectives. In the above-mentioned interpretation prompt text containing graphic interpretation requirements, the example framework is to first provide an image-level overview of the image slice, and then provide a target-level overview of each target in the image slice, and finally determine the descriptive text corresponding to the target under each preset graphic interpretation requirement. The specific content of the examples given in the above-mentioned interpretation prompt text containing graphic interpretation requirements is not limited in this specification and can be set according to actual needs.

[0091] S106: Determine result annotation information of the image slice according to the classification label, segmentation mask and graphic interpretation text of the image slice.

[0092] In one or more embodiments of the present specification, in the above steps, multi-task annotation of image slices is achieved through image classification, image segmentation and image-text interpretation, and the multi-task refers to the image classification task, image segmentation task and image-text interpretation task in the above steps S102 to S104. In this step, the server needs to determine the result annotation information of the image slice according to the classification label, segmentation mask and image-text interpretation text of the image slice.

[0093] Specifically, the server uses the classification label, segmentation mask, and graphic interpretation text of the image slice as the result annotation information of the image slice.

[0094] Optionally, the server obtains result annotation information of the image slice through manual verification and correction based on the classification label, segmentation mask and graphic interpretation text of the image slice.

[0095] In the above method, the server obtains a remote sensing image and slices the remote sensing image to obtain each image slice to be annotated. For each image slice, the server determines the classification label of the image slice through the image classification model and determines the segmentation mask of the image slice through the image segmentation model. According to the image slice, the preset interpretation prompt text and the preset image-text interpretation requirements, the interpretation instruction is determined, and the interpretation instruction is input into the image-text interpretation model, so as to determine the image-text interpretation text of the image slice, that is, the multi-task annotation of the remote sensing image is realized through image classification, image segmentation and image-text interpretation, and in the process of automating the multi-task annotation of the remote sensing image, the labor cost is reduced and the annotation efficiency is improved. At the same time, since the image-text interpretation requirements clarify the requirements of the image-text interpretation task for the output image-text interpretation text, and the structured framework of the image-text interpretation task is provided in the interpretation instruction, the image-text interpretation text output by the image-text interpretation model has high quality. At the same time, the classification label, segmentation mask and image-text interpretation text of the image slice are manually verified and corrected, which ensures the quality of the multi-task annotation and improves the reliability of the result annotation information.

[0096] In addition, this specification provides a process of performing cloud screening and similarity screening on each image slice determined in step S100, which is as follows:

[0097] In one or more embodiments of the present specification, in order to improve the quality of each image slice obtained after slicing the remote sensing image, the server needs to perform cloud screening and similarity screening on each image slice to be labeled.

[0098] Specifically, the server may perform cloud and fog screening on each image slice to be annotated to improve the quality of the image slices blocked by clouds and fog, and may perform similarity screening on each image slice to be annotated to eliminate redundant image slices.

[0099] It should be noted that the specific methods of cloud and fog screening and similarity screening are not limited in this specification and can be set according to actual needs. Cloud and fog screening refers to identifying and excluding clouds and fog in images, improving the quality of image slices, and improving the accuracy of subsequent annotation tasks such as image classification. Similarity screening refers to comparing the similarities between different image slices, identifying and proposing repeated or highly similar image slices, so as to reduce redundant data and improve the efficiency of subsequent annotation tasks such as image classification.

[0100] In addition, this specification provides a process for obtaining the classification label of the image slice based on the image slice and the preset classification prompt text through the image classification model, which is as follows:

[0101] In one or more embodiments of the present specification, the server may input the image slice and the preset classification prompt text into the image classification model for each image slice, and obtain the classification label of the image slice output by the image classification model. Among them, the image-text classification model may use the existing large visual language model to save costs. The classification prompt text at least represents the format requirements for the classification label of the image slice. For example, the classification prompt text may be "You are a remote sensing expert. You have received an image slice of a high-resolution multispectral satellite remote sensing image. Please summarize the image content and explain the classification label corresponding to the image slice from the preset scene category list (agriculture, commerce, industry, residential, transportation, water, undeveloped, etc.), and output the classification label in the form of parent category label / subcategory label". The specific content of the scene category list is not limited in the present specification. Please refer to the relevant content of the classification label in step S102, which will not be repeated here.

[0102] In addition, the present specification provides a method for determining a candidate annotation box of an image slice based on a target detection model, and then obtaining a segmentation mask of the image slice based on the candidate annotation box of the image slice through an image segmentation model.

[0103] In one or more embodiments of the present specification, the server may input each image slice into a target detection model to obtain a labeled candidate box for the image slice, and input the image slice labeled with the labeled candidate box into an image segmentation model, and use the image segmentation model to respectively determine the segmentation mask within each labeled candidate box to obtain the segmentation mask of the image slice output by the image segmentation model.

[0104] It should be noted that the types of target detection models and image segmentation models are not limited in this specification. For example, the target detection model may be a YOLO (You Only Look Once) model or a RP (Region Proposal) network, and the image segmentation model may be a SAM model (Segment Anything Model).

[0105] In the above, a target detection model is placed in front of the image segmentation model, and the target detection model is used to quickly generate annotation candidate boxes. Based on each annotation candidate box, the segmentation mask of the image slice is obtained through the segmentation capability of the image segmentation model, thereby improving the efficiency and accuracy of image segmentation.

[0106] like Figure 2 As shown, it is a schematic diagram of the structure for image classification, image segmentation and image-text interpretation of image slices provided in this specification.

[0107] In addition, this specification provides a process for manually verifying and correcting the classification labels, segmentation masks, and graphic interpretation text of image slices, as follows:

[0108] In one or more embodiments of the present specification, the server may, for each image slice, use the classification label, segmentation mask, and graphic interpretation text of the image slice as the initial annotation information of the image slice; determine adjustment notification information including the image slice and the initial annotation information of the image slice, and send the adjustment notification information to the first user, wherein the adjustment notification information is used to prompt the first user to adjust the initial annotation information based on the image slice; and obtain the information returned by the first user as the result annotation information of the image slice.

[0109] Optionally, when there are multiple first users, the server may use the classification label, segmentation mask, and graphic interpretation text of each image slice as the initial annotation information of the image slice; divide each image slice and the initial annotation information of each image slice into multiple initial annotation sets. Determine and return the adjustment notification information containing the initial annotation set to each first user; and obtain the information returned by each first user. The specific method of dividing each image slice and the initial annotation information of each image slice is not limited in this specification, such as evenly distributing according to the number of first users.

[0110] It should be noted that after verification and correction by the first user, it can be determined that the review notification information containing the result annotation information of the image slice is sent to the second user, and the review notification information is used to prompt the second user to review the result annotation information of the image slice to ensure the accuracy and reliability of the multi-task annotation of the image slice. That is, in one or more embodiments of the present specification, the server can determine the review notification information containing the result annotation information of the image slice, send it to the second user, and respond to the review result of the result annotation information returned by the second user. When the review result is not passed, the result annotation information of the image slice and the image slice are resent to the first user so that the first user can modify the result annotation information of the image slice again. When the review result is passed, the result annotation information of the image slice is stored in a preset storage format.

[0111] In the above, the accuracy of the annotation of the image slices is ensured through the manual verification and modification by the first user, and the reliability of the annotation of the image slices is improved through the review by the second user.

[0112] like Figure 3 The figure shows a flowchart of manual verification and correction provided in this manual.

[0113] The above is a remote sensing image multi-task annotation method provided in one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding remote sensing image multi-task annotation device, such as Figure 4 shown.

[0114] The slicing module 500 acquires a remote sensing image and slices the remote sensing image to obtain image slices to be labeled;

[0115] The first labeling module 501 determines, for each image slice, a classification label of the image slice through an image classification model, and determines a segmentation mask of the image slice through an image segmentation model;

[0116] The second annotation module 502 obtains the image-text interpretation text of the image slice through the image-text interpretation model according to the image slice and the preset interpretation prompt text containing the image-text interpretation requirements, wherein the image-text interpretation requirements at least include the positional relationship between the objects in the image slice and the position of the objects in the image slice, the image-level overview, the object-level overview and the shape of the object area;

[0117] The determination module 503 determines the result annotation information of the image slice according to the classification label, segmentation mask and graphic interpretation text of the image slice.

[0118] Optionally, the slicing module 500 is specifically used to perform radiation correction and orthorectification on the remote sensing image; and slice the corrected remote sensing image according to a preset image size to obtain image slices to be labeled.

[0119] Optionally, the slicing module 500 can also be used to obtain a multispectral image and a panchromatic image corresponding to the remote sensing image; fuse the multispectral image and the panchromatic image to obtain a fused remote sensing image; and slice the fused remote sensing image to obtain image slices to be labeled.

[0120] Optionally, the first labeling module 501 is specifically configured to input the image slice and a preset classification prompt text into an image classification model to obtain a classification label of the image slice output by the image classification model.

[0121] Optionally, the first labeling module 501 can also be used to input the image slice into a target detection model to obtain a labeling candidate box for the image slice; input the image slice marked with the labeling candidate box into an image segmentation model, and use the image segmentation model to determine the segmentation masks within each labeling candidate box, thereby obtaining the segmentation mask of the image slice output by the image classification model.

[0122] Optionally, the determination module 503 is specifically used to use the classification label, segmentation mask and graphic interpretation text of the image slice as the initial annotation information of the image slice; determine the adjustment notification information containing the image slice and the initial annotation information of the image slice, and send it to the first user, the adjustment notification information is used to prompt the first user to adjust the initial annotation information based on the image slice; obtain the information returned by the first user as the result annotation information of the image slice.

[0123] Optionally, the determination module 503 may also be used to determine review notification information including the result annotation information of the image slice, and send it to a second user, wherein the review notification information is used to prompt the second user to review the result annotation information of the image slice; in response to the review result of the result annotation information returned by the second user; when the review result is not passed, the result annotation information of the image slice and the image slice are resent to the first user; when the review result is passed, the result annotation information is stored in a preset storage format.

[0124] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A multi-task annotation method for remote sensing images is provided.

[0125] This manual also provides Figure 5 The schematic structure diagram of the electronic device shown in FIG. Figure 5 As mentioned above, at the hardware level, the device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The multi-task annotation method for remote sensing images. Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0126] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0127] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0128] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0129] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0130] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0131] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0132] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0134] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0135] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0136] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0137] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0138] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0139] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0140] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A remote sensing image multi-task annotation method, characterized in that: include: Acquire a remote sensing image, and slice the remote sensing image to obtain image slices to be labeled; For each image slice, determine a classification label of the image slice by using an image classification model, and determine a segmentation mask of the image slice by using an image segmentation model; According to the image slice and the preset interpretation prompt text containing the image-text interpretation requirements, a graphic-text interpretation text of the image slice is obtained through a graphic-text interpretation model, wherein the graphic-text interpretation requirements at least include the positional relationship between the objects in the image slice and the position of the objects in the image slice, the image-level overview, the object-level overview and the shape of the object area; The result annotation information of the image slice is determined according to the classification label, segmentation mask and graphic interpretation text of the image slice.

2. The method according to claim 1, characterized in that: Slicing the remote sensing image to obtain image slices to be labeled specifically includes: Performing radiation correction and orthorectification on the remote sensing image; According to a preset image size, the corrected remote sensing image is sliced ​​to obtain image slices to be labeled.

3. The method according to claim 1, characterized in that: Slicing the remote sensing image to obtain image slices to be labeled specifically includes: Acquire a multispectral image and a panchromatic image corresponding to the remote sensing image; The multispectral image and the panchromatic image are fused to obtain a fused remote sensing image; The fused remote sensing image is sliced ​​to obtain image slices to be labeled.

4. The method according to claim 1, characterized in that: The classification label of the image slice is determined by the image classification model, including: The image slice and the preset classification prompt text are input into the image classification model to obtain the classification label of the image slice output by the image classification model.

5. The method according to claim 1, characterized in that: The segmentation mask of the image slice is determined by the image segmentation model, specifically including: The image slice is input into the target detection model to obtain the annotation candidate box of the image slice; The image slice marked with the labeled candidate box is input into an image segmentation model, and the segmentation mask in each labeled candidate box is determined by the image segmentation model to obtain the segmentation mask of the image slice output by the image classification model.

6. The method according to claim 1, characterized in that: According to the classification label, segmentation mask and graphic interpretation text of the image slice, the result annotation information of the image slice is determined, specifically including: The classification label, segmentation mask and graphic interpretation text of the image slice are used as the initial annotation information of the image slice; Determine adjustment notification information including the image slice and the initial annotation information of the image slice, and send the information to the first user, wherein the adjustment notification information is used to notify the first user to adjust the initial annotation information based on the image slice; The information returned by the first user is obtained as the result annotation information of the image slice.

7. The method according to claim 6, characterized in that: The method further comprises: Determine review notification information including the result annotation information of the image slice, and send the information to the second user, wherein the review notification information is used to notify the second user to review the result annotation information of the image slice; responding to the review result of the result annotation information returned by the second user; When the audit result is not passed, the result annotation information of the image slice and the image slice are resent to the first user; when the audit result is passed, the result annotation information is stored in a preset storage format.

8. A remote sensing image multi-task annotation device, characterized in that: include: A slicing module, used to acquire remote sensing images and slice the remote sensing images to obtain image slices to be labeled; A first labeling module is used to determine, for each image slice, a classification label of the image slice through an image classification model, and to determine a segmentation mask of the image slice through an image segmentation model; A second annotation module is used to obtain a graphic-text interpretation text of the image slice through a graphic-text interpretation model according to the image slice and a preset interpretation prompt text containing graphic-text interpretation requirements, wherein the graphic-text interpretation requirements at least include a positional relationship between various objects in the image slice and the position of each object in the image slice, an image-level overview, an object-level overview, and a shape of a target area; The determination module is used to determine the result annotation information of the image slice according to the classification label, segmentation mask and graphic interpretation text of the image slice.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the method according to any one of claims 1 to 7 is implemented when the processor executes the program.

Images