Aerial image detection method, device, storage medium and electronic equipment

Abstract

This specification discloses an aerial image detection method, device, storage medium and electronic device. By acquiring aerial images and corresponding first object labels as training samples and determining corresponding annotations, the image features, object features and text features corresponding to first object labels at various scales are obtained through an encoder. The decoder determines the enhanced text features and the enhanced object features, and the detection layer determines the target text features and target labels corresponding to the enhanced object features. The total loss is determined according to the difference between the output of the detection layer and the annotation, and the aerial detection model is trained with the minimum total loss as the adjustment target. By integrating scale features into text features, the trained aerial detection model can use multi-scale text features as a guide, further promote the alignment of image and text features, enhance the association between image features and text features, and break through the category limitations of traditional aerial detection.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an aerial image detection method, device, storage medium and electronic device. Background Art

[0002] At present, with the development of computer technology, aerial detection technology realized by satellites, drones and other aircraft has been widely used in various scenarios, such as road inspection, urban planning, disaster response, disaster search, etc. Aerial detection technology usually requires the use of a closed set detector to identify the collected aerial images and match the identified objects with preset classification labels. However, when identifying objects in aerial images based on a closed set detector, the classification label of the object must be known to the closed set detector during the training stage. If the classification label corresponding to the detected object is not in the classification label set known to the closed set detector during the training stage, the closed set detector will incorrectly classify the object into the classification label in the known classification label set. In other words, the closed set detector cannot match the classification label of an unknown type of object.

[0003] In the prior art, in order to break through the classification categories used in detector training and the limitations on the detector's recognition of objects, when training the detector model, a contrastive learning method is usually used to make image features and corresponding text description features close in the feature space and far away from features of other categories. A cross-modal attention mechanism is also introduced to better utilize text descriptions to enhance the representation of image features, realize open vocabulary category detection, and enable the model to recognize and process new categories or words that have not been seen in the training phase.

[0004] However, the existing open vocabulary detection is usually only applicable to the recognition and processing of natural images. Compared with natural images, aerial images have more complex backgrounds and a larger number of small objects, which leads to the fact that some important small objects are often ignored when the existing open vocabulary detection is used to process aerial images. Therefore, how to integrate scale features into open vocabulary detection has become an urgent problem to be solved. To this end, this specification provides an aerial image detection method, device, storage medium and electronic device. Summary of the invention

[0005] This specification provides an aerial image detection method, device, storage medium and electronic device to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] A method for detecting aerial images, the method being used to train an aerial detection model, the aerial detection model comprising at least an encoder, a decoder and a detection layer, including:

[0008] Acquire an aerial image and corresponding first object labels as training samples, determine features of objects on the aerial image and second object labels corresponding to the objects as corresponding annotations, input the training samples into the encoder, and obtain image features at various scales of the aerial image, features of objects in the aerial image, and text features corresponding to the first object labels;

[0009] Inputting the features output by the encoder into the decoder, and for each scale of the aerial image, respectively determining the image features matched by each text feature through the decoder, and fusing the matched image features into the text features to determine the enhanced text features of each text feature at the scale;

[0010] For each object feature, the decoder performs weighted fusion of the image features and the enhanced text features with the object feature to determine an enhanced object feature of the object feature;

[0011] Inputting each enhanced object feature and each text feature into the detection layer, determining the text feature corresponding to each enhanced object feature through the detection layer as the target text feature, and determining the target label corresponding to each enhanced object feature according to each target text feature;

[0012] The aerial photography detection model is trained according to the difference between the output of the detection layer and the annotation, and the trained aerial photography detection model is used to determine the text label corresponding to each object in the image according to the input image and the text label set.

[0013] Optionally, before inputting the features output by the encoder into the decoder, the method further includes:

[0014] For each image feature, the image feature is matched with each of the text features, and the text feature to be fused is determined according to the matching result;

[0015] The encoder is used to fuse the text features to be fused into the image features to determine enhanced image features.

[0016] Optionally, matching the image feature with each of the text features and determining the text feature to be fused according to the matching result specifically includes:

[0017] Determining, by the encoder, a first similarity between the image feature and each of the text features;

[0018] The text features to be fused are determined from the text features according to the first similarity.

[0019] Optionally, fusing the to-be-fused text features into the image features through the encoder to determine enhanced image features, specifically including:

[0020] determining a first weight according to the first similarity;

[0021] The encoder performs weighted fusion of the text feature to be fused and the image feature according to the first weight to determine an enhanced image feature.

[0022] Optionally, determining the image features matched by each of the text features by the decoder respectively includes:

[0023] For each text feature, determining a second similarity between the text feature and each of the image features;

[0024] According to the magnitude of the second similarity, an image feature matching the text feature is determined.

[0025] Optionally, for each object feature, weighted fusion of the image features and the enhanced text features with the object feature is performed by the decoder to determine an enhanced object feature of the object feature, specifically including:

[0026] Inputting each of the image features, each of the enhanced text features, and each of the object features into the decoder, so that the decoder determines, for each object feature, a third similarity between the object feature and each of the enhanced text features, determines a second weight corresponding to each of the enhanced text features according to each of the third similarities, fuses each of the enhanced text features to the object feature according to the second weight, and determines a first object feature of the object feature;

[0027] Determine fourth similarities between the first object feature and each of the image features, determine third weights corresponding to each of the image features according to each of the fourth similarities, fuse each of the image features into the first object feature according to the third weight, and determine an enhanced object feature of the object feature.

[0028] Optionally, training the aerial photography detection model according to the difference between the output of the detection layer and the annotation specifically includes:

[0029] For each enhanced object feature, determining a first loss according to a difference between a target label corresponding to the enhanced object feature and a second object label corresponding to the annotation;

[0030] Determining a second loss according to a fifth similarity between each enhanced object feature and the corresponding target text feature, wherein the fifth similarity is negatively correlated with the second loss;

[0031] The total loss is determined according to each first loss and the second loss, and the aerial photography detection model is trained with the minimum total loss as an adjustment target.

[0032] This specification provides an aerial image detection device, which is used to train an aerial detection model. The aerial detection model at least includes an encoder, a decoder, and a detection layer, including:

[0033] An extraction module is used to obtain an aerial image and corresponding first object labels as training samples, and determine the features of each object on the aerial image and the second object labels corresponding to each object as corresponding annotations, input the training samples into the encoder, and obtain the image features of each scale of the aerial image, the features of each object in the aerial image, and the text features corresponding to each first object label;

[0034] A first enhancement module is used to input the features output by the encoder into the decoder, and for each scale of the aerial image, determine the image features matched by each text feature through the decoder, and fuse the matched image features into the text features to determine the enhanced text features of each text feature at the scale;

[0035] A second enhancement module is used for, for each object feature, weightedly fusing the image features and the enhanced text features with the object feature through the decoder to determine an enhanced object feature of the object feature;

[0036] A detection module, used to input each enhanced object feature and each text feature into the detection layer, determine the text feature corresponding to each enhanced object feature through the detection layer as the target text feature, and determine the target label corresponding to each enhanced object feature according to each target text feature;

[0037] A training module is used to train the aerial photography detection model according to the difference between the output of the detection layer and the annotation. The trained aerial photography detection model is used to determine the text label corresponding to each object in the image according to the input image and text label set.

[0038] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned aerial image detection method is implemented.

[0039] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the above-mentioned aerial image detection method is implemented when the processor executes the program.

[0040] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0041] In an aerial image detection method provided in this specification, an aerial image and corresponding first object labels are obtained as training samples, and the features of each object on the aerial image and the second object labels corresponding to each object are determined as corresponding annotations, and the training samples are input into the encoder to obtain the image features at each scale of the aerial image, the object features of the aerial image, and the text features corresponding to each first object label. Then, the scale features of the image are integrated into the text features through the decoder to determine the enhanced text features, and then the enhanced text features and image features are integrated into the object features through the decoder to determine the enhanced object features. Finally, the target text features and target labels corresponding to each enhanced object feature are determined through the detection layer. Thus, the total loss is determined according to the difference between the output of the detection layer and the annotation, and the aerial detection model is trained with the minimum total loss as the adjustment target.

[0042] In the above-mentioned aerial image detection method, when fusing text labels with images, scale features are integrated into text features, so that the trained aerial detection model can use multi-scale text features as a guide, further promote the alignment of image and text features, and enhance the correlation between image features and text features, breaking through the category limitations of traditional aerial detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0044] Figure 1 A flowchart of a model training method provided in this specification;

[0045] Figure 2 A flow chart of a page detection method provided in this specification;

[0046] Figure 3 A schematic diagram of a process for determining output text of a page image provided in this specification;

[0047] Figure 4 A schematic diagram of a process for determining total loss provided for this specification;

[0048] Figure 5 A schematic diagram of a model training device provided in this specification;

[0049] Figure 6 A schematic diagram of a page navigation detection device provided in this specification;

[0050] Figure 7 A method corresponding to the Figure 1Schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] With the development of computer technology, aerial detection technology realized by satellites, drones and other aircraft has been widely used in various scenarios, such as road inspection, urban planning, disaster response, disaster search, etc. The current aerial detection technology usually pre-trains a closed set detector to achieve target detection in aerial pictures, that is, during training, the closed set detector is trained to match the recognized objects with the pre-set text labels accurately, and then the trained closed set detector is used to match the objects recognized in the picture with the text labels in the text label set used during training.

[0053] However, although the existing closed-set detector has high accuracy in matching image features with text features, it takes a long time to train, requires a large number of training samples, and cannot match image features with unknown categories. As a result, the existing aerial photography detection technology needs to retrain the closed-set detector when it is applied to detection in different scenes or different regions, which is inefficient.

[0054] In order to break through the classification categories used in the training of closed-set detectors and the limitations on the detector's recognition of objects, open vocabulary detection can also be used at present. That is, when training the detector model, the image features and the corresponding text description features are made close in the feature space through contrastive learning methods, but far away from the features of other categories. A cross-modal attention mechanism is introduced to better utilize text descriptions to enhance the representation of image features, realize open vocabulary category detection, and enable the model to recognize and process new categories or words that have not been seen in the training phase.

[0055] However, existing open vocabulary detection is usually used to recognize and process natural images. Compared with natural images, aerial images have more complex backgrounds and a larger number of small objects, which leads to the fact that existing open vocabulary detection often ignores some important small objects when detecting aerial images. In other words, the existing open vocabulary detection has low detection accuracy for small objects when detecting aerial images. However, the focus of aerial detection technology is on the detection of small objects. Therefore, how to integrate scale information into open vocabulary detection has become a problem that needs to be solved urgently.

[0056] This manual uses a comparative learning method to link image features and text features, while integrating image scale information into text features, so as to guide the aerial detection model to detect objects of different scales in aerial images through enhanced text features that incorporate scale features.

[0057] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0058] Figure 1 A flow chart of an open vocabulary category detection method provided in an embodiment of this specification includes the following steps:

[0059] S100: Acquire an aerial image and corresponding first object labels as training samples, determine features of objects on the aerial image and second object labels corresponding to the objects as corresponding annotations, input the training samples into the encoder, and obtain image features at various scales of the aerial image, object features of the aerial image, and text features corresponding to the first object labels.

[0060] In one or more embodiments of the present specification, there is no restriction on which specific device executes the aerial image detection method, for example, a mobile terminal and a server, etc. However, since subsequent steps involve model training, feature extraction, feature fusion and other steps, these steps are generally executed by a server. Therefore, this specification also uses the server executing the aerial image detection method as an example for description, wherein the server can be a single device or composed of multiple devices, for example, a distributed server. This specification does not impose any restrictions on this, and it can also refer to a server for cloud services.

[0061] In order to train the aerial photography detection model to be trained, the aerial photography detection model can extract image features of the input aerial photography pictures based on the input text label set, and determine the text labels corresponding to each image feature to realize open vocabulary detection. The server first obtains the aerial photography images and each first object label as training samples. Then the training samples are input into the aerial photography detection model to be trained, and the aerial photography detection model is trained.

[0062] Specifically, the server obtains an aerial image and each first object label as a training sample, wherein each first object label is preset for each object in the aerial image, and the aerial detection model to be trained is used to identify each object in the aerial image according to the input aerial image and the first object label, and determine the correspondence between each object and each first object label. Then, the features of each object on the aerial image and the corresponding second object labels are determined as annotations for the training of the aerial detection model. Then, the aerial image and each first object label are input into the encoder of the aerial detection model to be trained, and the features of the aerial image and each first object label in the text label set are extracted respectively, and the image features of each scale of the aerial image, the features of each object in the aerial image, and the text features corresponding to each first object label are determined.

[0063] The determined image features of each scale are the features of the aerial image at each scale after the aerial image is scale-transformed, and the object features are the features of each object in the aerial image, including the object outer frame used to mark each object, and the features of each object.

[0064] It should be noted that each first object label includes each second object label, that is, the number of each first object label is not less than the number of each object in the aerial image, each second object label belongs to each first object label, and the number of each second object label is equal to the number of each object in the aerial image.

[0065] In addition, in one or more embodiments of the present specification, there is no restriction on the specific model used to implement the aerial photography detection model. It can be implemented based on a transform detector, can be implemented through a convolutional neural network model (CNN), or can be constructed by combining a transform detector and a CNN, such as through a contrastive language-image pre-training model (CLIP), etc. It can be set based on actual needs, and this specification does not impose any restrictions on this.

[0066] S102: Input the features output by the encoder into the decoder, and for each scale of the aerial image, determine the image features that match the text features respectively through the decoder, and fuse the matched image features into the text features to determine the enhanced text features of the text features at the scale.

[0067] In order to fuse the scale feature with the text feature, the scale feature corresponding to the text feature can be used to match the image feature with a scale similar to the text feature, so as to better align the image feature with the text feature. The server can input the output of the decoder into the decoder of the aerial photography detection model to be trained, so that the decoder determines the image features that match the text features for each scale of the aerial image, fuses the matched image features to the text features, and determines the enhanced text features of each text feature at the scale, thereby fusing the scale features of the image features into the text features, and determining the enhanced text features of each scale corresponding to each text feature.

[0068] It should be noted that in one or more embodiments of this specification, the decoder is not limited to the specific method used to fuse the matched image features into the text features. For example, the weights of the image features and the text features are dynamically adjusted through the gating mechanism of the gated fusion, so as to achieve the fusion of the image features into the text features. Among them, the gating mechanism can be implemented by a gated recurrent unit or a long short-term memory network. It is also possible to associate each image feature with the text feature through a bilinear pooling method.

[0069] In addition, it can also be achieved through a multi-head cross attention module. ,in, represents the enhanced text features of the i-th level scale, Represents text features, represents a multi-head cross attention module, Represents the image features of the i-th scale. Since there are many methods that can be used, they can be set according to actual needs, so they will not be described here. In the following, we will take the multi-head cross attention module to achieve feature combination as an example to illustrate.

[0070] S104: For each object feature, the decoder performs weighted fusion of the image features and the enhanced text features with the object feature to determine an enhanced object feature of the object feature.

[0071] After the enhanced text features including the scale features are determined, in order to facilitate the subsequent matching of the scale features of each object feature with each first object label, each image feature and each enhanced text feature may be weightedly fused into each object feature through a decoder.

[0072] Specifically, for each object feature, the server fuses each enhanced text feature into each object feature through a decoder, thereby injecting the text feature and the scale feature of the target into the object feature. That is, for each object feature, the third similarity between the enhanced text feature of each scale and the object feature is calculated, and the second weight corresponding to the enhanced text feature of each scale is determined according to the magnitude of the third similarity. Then, each enhanced text feature is fused into the object feature according to each second weight to determine the first object feature of the object feature.

[0073] Of course, it is also possible to integrate each enhanced text feature into each object feature through a multi-head cross attention module, such as ,in, represents the enhanced object features at the i-th scale, represents the object features at the i-th scale, Represents the criss-cross attention module.

[0074] Then, based on the same idea, the image features of each scale are fused with the first object features to determine the enhanced object features, that is, the fourth similarities between the first object features and each image feature are determined, and the third weights corresponding to each image feature are determined according to each fourth similarity. According to each third weight, each image feature is fused to the first object feature to determine the enhanced object feature of the object feature.

[0075] It should be noted that in one or more embodiments of the present specification, there is no restriction on the specific order in which the aerial photography detection model determines the enhanced object features. That is, each image feature may be fused with each object feature first, and then each enhanced text may be fused therein to determine the enhanced object features.

[0076] S106: According to the difference between the output of the detection layer and the annotation, the aerial photography detection model is trained, and the trained aerial photography detection model is used to determine the text label corresponding to each object in the image according to the input image and the text label set.

[0077] After determining the features of each enhanced object, the aerial photography detection model can match the features of each enhanced object with the features of each text through the detection layer to determine the target label corresponding to each enhanced object feature.

[0078] Specifically, each enhanced object feature and each text feature are input into the detection layer, and the detection layer calculates the similarity score between each enhanced text feature and each text feature, and then determines the text feature corresponding to each enhanced object feature according to each similarity score as the target text feature. Then, the target label corresponding to each enhanced object feature is determined according to the target text feature corresponding to each enhanced object feature.

[0079] S108: Determine the loss according to the difference between the output of the detection layer and the annotation, adjust the aerial photography detection model according to the loss, and use the trained aerial photography detection model to determine the text label corresponding to each object in the image according to the input image and text label set.

[0080] After determining the output of the detection layer, the aerial photography detection model can determine the loss based on the difference between the output of the detection layer and the annotation, and train the aerial photography detection model with the minimum loss as the optimization goal.

[0081] Specifically, a first loss is determined based on the difference between the target label corresponding to each enhanced object feature and the corresponding second object label in the annotation. A second loss is determined based on the fifth similarity between each enhanced object feature and the text feature corresponding to the enhanced object feature. A total loss is determined based on the first loss and the second loss. The parameters of the aerial photography detection model are adjusted based on the determined total loss. Through the reverse propagation of the model, the trained aerial photography detection model can adjust the similarity between the object feature and the corresponding first object label.

[0082] The trained aerial photography detection model is used to determine the text label corresponding to each image feature in the image according to the input image and text label set.

[0083] It should be noted that in one or more embodiments of this specification, the aerial photography detection model is not limited to the specific method used to calculate the fifth similarity, which can be calculated vector similarity, cosine similarity, or other methods, which are not limited in this specification. ,in, Describe the image features, Represents text features, Indicates the fifth similarity between text features and image features, represents the Euclidean norm, which is the L2 norm.

[0084] based on Figure 1The aerial image detection method shown in the figure obtains the aerial image and the corresponding first object labels as training samples, determines the features of each object on the aerial image and the second object labels corresponding to each object as corresponding annotations, inputs the training samples into the encoder, obtains the image features at each scale of the aerial image, the object features of the aerial image and the text features corresponding to each first object label, then integrates the scale features of the image into the text features through the decoder to determine the enhanced text features, then integrates the enhanced text features and image features into the object features through the decoder to determine the enhanced object features, and finally determines the target text features and target labels corresponding to each enhanced object feature through the detection layer. Thus, the total loss is determined according to the difference between the output of the detection layer and the annotation, and the aerial detection model is trained with the minimum total loss as the adjustment target.

[0085] In the above-mentioned aerial image detection method, when fusing text labels with images, scale features are integrated into text features, so that the trained aerial detection model can use multi-scale text features as a guide, further promote the alignment of image and text features, and enhance the correlation between image features and text features, thus breaking through the category limitations of traditional aerial detection.

[0086] In step S100, when determining the multi-scale image features, this specification does not limit the specific method used by the aerial photography detection model to extract scale features. A feature pyramid method can be used, or a model with the last global pooling layer removed from ResNet-50 can be used to extract multi-scale image features. Since there are many methods that can be used, this specification will not go into details here and can be set according to actual needs.

[0087] In order to further enhance the extraction of image features, the aerial photography detection model may also fuse text features into image features before inputting each image feature and each text feature into the decoder to enhance the extraction of image features.

[0088] Specifically, the aerial photography detection model will match each image feature with each text feature, determine the target text feature based on the matching result, and then fuse the target text feature with the image feature through the encoder to determine the fused image feature.

[0089] When matching, the similarity between the image feature and each text feature can be calculated to determine the text feature with the highest similarity as the target text feature of the image feature. Of course, this specification does not limit the specific method used by the aerial photography detection model to calculate the similarity, which can be vector similarity, pre-similarity, etc., or the similarity score can be calculated through the attention mechanism, such as ,in, represents the similarity score, represents the image features of the i-th level scale, represents the text features of the jth scale, and the rest are preset parameters.

[0090] Furthermore, in order to prevent the fused text features from covering or interfering with the original features on the image features, the aerial photography detection model can also control the strength of the text features fused to the image features, that is, during the fusion process, the fusion weight is determined according to the similarity score of the two, and they are merged together based on the determined fusion weight.

[0091] It should be noted that in one or more embodiments of this specification, there is no limitation on the specific fusion method used to fuse text features into image features. In addition to the above method, it can also be implemented by Softmax function and representative Sigmoid function. Figure 2 As shown, Figure 2 The following is a schematic diagram of a text feature and image feature fusion process provided in this specification. The fused image feature can also be expressed as ,in, represents the Softmax function, Represents the Sigmoid function.

[0092] If the aerial photography detection model fuses the image features, then when determining the enhanced object features, the aerial photography detection model can use the fused image features instead of the image features extracted from the aerial image to determine the enhanced object features. Figure 3 As shown, Figure 3 A schematic diagram of a process for determining enhanced object features provided in this specification.

[0093] When determining the enhanced object features, the aerial photography detection model can also enhance the features of each object through multiple layers of decoders, so that decoders at different layers are used to optimize object features of different scales. For example, for object features of different scales, each object feature is input into a decoder of a preset number of layers, so that the decoder fuses the enhanced text features and image features into the object features of that scale, and determines the enhanced object features corresponding to the object features of that scale.

[0094] Then, when training the aerial photography detection model, the parameters of each layer of decoders are adjusted so that each layer of decoders can better enhance the features of objects of corresponding scales.

[0095] The aerial inspection model can also use other methods when determining losses, such as Figure 4 As shown, Figure 4This specification also provides a schematic diagram of the process of realizing target detection through an aerial detection model. The aerial detection model includes an encoder, a decoder, and a detection layer, wherein the detection layer includes a comparison head and a bounding box regression head, and the encoder also includes an image encoder, a text encoder, and a dual attention cross-modal fusion encoder. The multi-scale category semantic features used by the decoder guide the multi-modal fusion decoder.

[0096] The image encoder is used to extract image features from aerial images, the text encoder is used to extract text features from each text tag in the text tag set, and the dual attention cross-modal fusion encoder is used to fuse text features into image features of each scale. The multi-scale category semantic feature guided multi-modal fusion decoder is used to fuse image features of each scale into text features, determine enhanced text features, and enhance each object feature with enhanced text features and fused image features to determine enhanced object features. The contrast head is used to help the aerial detection model distinguish positive samples from negative samples through contrast learning, where the positive sample refers to the corresponding relationship between each object feature and the second object label, and the negative sample refers to the relationship between each object feature and other first object labels. The bounding box regression head is used to predict the bounding box coordinates of each enhanced object feature in order to more accurately locate each object.

[0097] based on Figure 4 The aerial photography detection model shown in the figure can also determine the total loss of the aerial photography detection model by first determining the similarity between each enhanced object feature and the corresponding target label based on the comparison head, thereby determining the comparison loss, and then determining the bounding box corresponding to each enhanced object feature and the bounding box of each object feature in the annotation based on the bounding box regression head, and using the bounding box corresponding to each enhanced object feature and the bounding box of each object feature in the annotation to determine the intersection over union loss (IOU), and determine the smoothing loss (Smooth L1 Loss) to measure the difference between the predicted offset and the actual offset. Then, the contrast loss, IOU loss and L1 loss are used as the total loss. That is, Figure 5 As shown, Figure 5 A schematic diagram of a process for determining total loss provided in this specification.

[0098] Based on the same idea as an aerial image detection method provided in one or more embodiments of this specification, this specification also provides a corresponding aerial image detection device, such as Figure 6 shown.

[0099] Figure 6 A schematic diagram of an open vocabulary category detection device provided in this specification specifically includes:

[0100] The extraction module 500 is used to obtain the aerial image and the corresponding first object labels as training samples, and determine the features of each object on the aerial image and the second object labels corresponding to each object as corresponding annotations, input the training sample into the encoder, and obtain the image features of each scale of the aerial image, the features of each object in the aerial image, and the text features corresponding to each first object label;

[0101] The first enhancement module 501 is used to input the features output by the encoder into the decoder, and for each scale of the aerial image, determine the image features matched by each text feature through the decoder, and fuse the matched image features into the text features to determine the enhanced text features of each text feature at the scale;

[0102] The second enhancement module 502 is used for weightedly fusing the image features and the enhanced text features with the object features through the decoder for each object feature to determine an enhanced object feature of the object feature;

[0103] A detection module 503 is used to input each enhanced object feature and each text feature into the detection layer, determine the text feature corresponding to each enhanced object feature through the detection layer as the target text feature, and determine the target label corresponding to each enhanced object feature according to each target text feature;

[0104] The training module 504 is used to train the aerial photography detection model according to the difference between the output of the detection layer and the annotation. The trained aerial photography detection model is used to determine the text label corresponding to each object in the image according to the input image and text label set.

[0105] Optionally, the extraction module 500 is used to match each image feature with the text features, and determine the text feature to be fused according to the matching result; and fuse the text feature to be fused into the image feature through the encoder to determine the enhanced image feature.

[0106] Optionally, the extraction module 500 is used to determine a first similarity between the image feature and the text features through the encoder; and determine a target text feature from the text features according to the first similarity.

[0107] Optionally, the extraction module 500 is used to determine a first weight according to the first similarity; and to perform weighted fusion of the target text feature and the image feature according to the first weight through the encoder to determine an enhanced image feature.

[0108] Optionally, the extraction module 500 is used to determine, for each text feature, a second similarity between the text feature and each of the image features; and determine an image feature that matches the text feature according to the magnitude of the second similarity.

[0109] Optionally, the first enhancement module 501 is used to input the image features, the enhanced text features and the object features into the decoder, so that the decoder determines, for each object feature, a third similarity between the object feature and the enhanced text features, determines a second weight corresponding to each enhanced text feature according to each third similarity, fuses the enhanced text features to the object feature according to the second weight, and determines a first object feature of the object feature; determines a fourth similarity between the first object feature and the image features, determines a third weight corresponding to each image feature according to each fourth similarity, fuses the image features to the first object feature according to the third weight, and determines an enhanced object feature of the object feature.

[0110] Optionally, the training module 504 is used to determine, for each enhanced object feature, a first loss based on a difference between a target label corresponding to the enhanced object feature and a corresponding second object label in the annotation; determine a second loss based on a fifth similarity between each enhanced object feature and the corresponding target text feature, wherein the fifth similarity is negatively correlated with the second loss; determine a total loss based on each first loss and the second loss, and train the aerial photography detection model with the minimum total loss as an adjustment target.

[0111] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 An aerial image detection method is provided.

[0112] This manual also provides Figure 7 The schematic structure diagram of the electronic device shown in FIG. Figure 7 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The aerial image detection method.

[0113] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0114] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0115] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0116] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0117] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0118] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0122] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0123] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0124] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0125] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0126] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

[0128] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0129] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for detecting aerial images, characterized in that: The method is used to train an aerial photography detection model, which includes at least an encoder, a decoder, and a detection layer, including: Acquire an aerial image and corresponding first object labels as training samples, determine features of objects on the aerial image and second object labels corresponding to the objects as corresponding annotations, input the training samples into the encoder, and obtain image features at various scales of the aerial image, features of objects in the aerial image, and text features corresponding to the first object labels; Inputting the features output by the encoder into the decoder, and for each scale of the aerial image, respectively determining the image features matched by each text feature through the decoder, and fusing the matched image features into the text features to determine the enhanced text features of each text feature at the scale; For each object feature, the decoder performs weighted fusion of the image features and the enhanced text features with the object feature to determine an enhanced object feature of the object feature; Inputting each enhanced object feature and each text feature into the detection layer, determining the text feature corresponding to each enhanced object feature through the detection layer as the target text feature, and determining the target label corresponding to each enhanced object feature according to each target text feature; The aerial photography detection model is trained according to the difference between the output of the detection layer and the annotation, and the trained aerial photography detection model is used to determine the text label corresponding to each object in the image according to the input image and the text label set.

2. The method according to claim 1, characterized in that Before inputting the features output by the encoder into the decoder, the method further comprises: For each image feature, the image feature is matched with each of the text features, and the text feature to be fused is determined according to the matching result; The encoder is used to fuse the text features to be fused into the image features to determine enhanced image features.

3. The method according to claim 2, characterized in that The image feature is matched with each of the text features, and the text feature to be fused is determined according to the matching result, specifically including: Determining, by the encoder, a first similarity between the image feature and each of the text features; The text features to be fused are determined from the text features according to the first similarity.

4. The method according to claim 3, characterized in that The encoder is used to fuse the text feature to be fused into the image feature to determine the enhanced image feature, specifically including: determining a first weight according to the first similarity; The encoder performs weighted fusion of the text feature to be fused and the image feature according to the first weight to determine an enhanced image feature.

5. The method according to claim 1, characterized in that Determining the image features matched by each of the text features by the decoder respectively specifically includes: For each text feature, determining a second similarity between the text feature and each of the image features; According to the magnitude of the second similarity, an image feature matching the text feature is determined.

6. The method according to claim 1, characterized in that For each object feature, the decoder sequentially weights and fuses the image features and the enhanced text features with the object feature to determine an enhanced object feature of the object feature, specifically including: Inputting each of the image features, each of the enhanced text features, and each of the object features into the decoder, so that the decoder determines, for each object feature, a third similarity between the object feature and each of the enhanced text features, determines a second weight corresponding to each of the enhanced text features according to each of the third similarities, fuses each of the enhanced text features to the object feature according to the second weight, and determines a first object feature of the object feature; Determine fourth similarities between the first object feature and each of the image features, determine third weights corresponding to each of the image features according to each of the fourth similarities, fuse each of the image features into the first object feature according to the third weight, and determine an enhanced object feature of the object feature.

7. The method according to claim 5, characterized in that Training the aerial photography detection model according to the difference between the output of the detection layer and the annotation, specifically including: For each enhanced object feature, determining a first loss according to a difference between a target label corresponding to the enhanced object feature and a second object label corresponding to the annotation; Determining a second loss according to a fifth similarity between each enhanced object feature and the corresponding target text feature, wherein the fifth similarity is negatively correlated with the second loss; The total loss is determined according to each first loss and the second loss, and the aerial photography detection model is trained with the minimum total loss as an adjustment target.

8. An aerial image detection device, characterized in that: The device is used to train an aerial photography detection model, and the aerial photography detection model at least includes an encoder, a decoder, and a detection layer, including: An extraction module is used to obtain an aerial image and corresponding first object labels as training samples, and determine the features of each object on the aerial image and the second object labels corresponding to each object as corresponding annotations, input the training samples into the encoder, and obtain the image features of each scale of the aerial image, the features of each object in the aerial image, and the text features corresponding to each first object label; A first enhancement module is used to input the features output by the encoder into the decoder, and for each scale of the aerial image, determine the image features matched by each text feature through the decoder, and fuse the matched image features into the text features to determine the enhanced text features of each text feature at the scale; A second enhancement module is used for, for each object feature, weightedly fusing the image features and the enhanced text features with the object feature through the decoder to determine an enhanced object feature of the object feature; A detection module, used to input each enhanced object feature and each text feature into the detection layer, determine the text feature corresponding to each enhanced object feature through the detection layer as the target text feature, and determine the target label corresponding to each enhanced object feature according to each target text feature; A training module is used to train the aerial photography detection model according to the difference between the output of the detection layer and the annotation. The trained aerial photography detection model is used to determine the text label corresponding to each object in the image according to the input image and text label set.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

Images