Construction method, device and equipment of model for remote sensing image target detection

Abstract

The invention provides a model construction method, device and equipment for remote sensing image target detection, and relates to the technical field of image processing. A construction method of a model for remote sensing image target detection comprises the following steps: respectively acquiring multi-scale visual features of an input remote sensing image and text semantic features of a preset text prompt to generate semantic enhanced multi-scale visual features corresponding to the remote sensing image; performing cross-scale attention modulation on the initial model based on the semantic enhanced multi-scale visual features to update target query parameters of the model; determining a rotation bounding box parameter corresponding to the target query parameter and the probability of the target category to obtain an output result of the model; constructing training sample data of the model; and based on the output result, training the model through the training sample data to optimize the parameters of the model. According to the embodiment of the invention, the method can effectively improve the detection precision of a small target and an inclined target in a complex scene.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method, apparatus, and device for constructing a model for target detection in remote sensing images. Background Technology

[0002] Due to the characteristics of remote sensing images, such as large field of view, complex backgrounds, and significant differences and arbitrary orientations of targets, a pre-designed model is needed to achieve highly reliable and accurate target detection in remote sensing images. GroundingDINO is an open-set target detection algorithm based on Transformer, which can improve the detection model's target perception ability in open scenes through "text-image alignment" pre-training.

[0003] However, if the model designed using the Grounding DINO algorithm is directly applied to target recognition tasks in remote sensing satellite images, the following problems may occur.

[0004] First, the discriminative features of small-scale targets mainly exist in the shallow high-resolution features of remote sensing images. Existing "visual-language" models are unable to effectively distinguish and focus on multi-scale features based on semantic information, which leads to the easy loss of detailed information of small targets during feature fusion.

[0005] Secondly, the model uses horizontal box regression, which lacks the ability to directly represent rotational geometric features and is difficult to adapt to targets that are arbitrarily arranged in remote sensing images.

[0006] Finally, the distribution of target categories in remote sensing scenarios is extremely unbalanced, with some categories having very few samples. During end-to-end training, these categories are easily misled by the gradients of other categories, resulting in low recall rates for rare categories. Summary of the Invention

[0007] This application provides a method, apparatus, and device for constructing a model for target detection in remote sensing images. By using semantically guided cross-scale attention modulation and rotating bounding box modeling, the detection accuracy of small and tilted targets in complex scenes is effectively improved.

[0008] According to one aspect of this application, a method for constructing a model for target detection in remote sensing images is provided, comprising: acquiring multi-scale visual features of an input remote sensing image and textual semantic features of a preset text prompt, respectively, to generate semantically enhanced multi-scale visual features corresponding to the remote sensing image; performing cross-scale attention modulation on an initial model based on the semantically enhanced multi-scale visual features to update the model's target query parameters; determining the rotation bounding box parameters and the probability of the target category corresponding to the target query parameters to obtain the model's output results; constructing training sample data for the model; and training the model using the training sample data based on the output results to optimize the model's parameters.

[0009] According to some embodiments, multi-scale visual features of the input remote sensing image and text semantic features of the preset text prompt are obtained respectively to generate semantically enhanced multi-scale visual features corresponding to the remote sensing image, including: obtaining multi-scale visual features based on preset feature dimensions; obtaining text semantic features based on preset target text prompts; and performing cross-modal enhancement on the multi-scale visual features and text semantic features to generate semantically enhanced multi-scale visual features.

[0010] According to some embodiments, based on semantically enhanced multi-scale visual features, cross-scale attention modulation is applied to the initial model to update the model's target query parameters, including: obtaining a first target query feature corresponding to a preset target query set; obtaining feature level weights corresponding to the semantic information of the target query set based on the first target query feature and text semantic features; performing cross-scale attention modulation on the model based on the semantically enhanced multi-scale visual features through feature level weights to obtain a third target query feature; and performing nonlinear processing on the third target query feature to obtain and update the target query parameters.

[0011] According to some embodiments, based on the first target query feature and the text semantic feature, the feature level weight corresponding to the semantic information of the target query set is obtained, including: fusing the first target query feature and the text semantic feature to obtain the second target query feature; and performing level weight prediction based on the second target query feature to obtain the feature level weight.

[0012] According to some embodiments, based on semantically enhanced multi-scale visual features, cross-scale attention modulation is performed on the model through feature hierarchy weights to obtain a third target query feature, including: based on the second target query feature, attention sampling is performed on the semantically enhanced multi-scale visual features to obtain multiple sampling points; based on the multiple sampling points, attention scores corresponding to the semantically enhanced multi-scale visual features are calculated through feature hierarchy weights; the attention scores are weighted and aggregated with the semantically enhanced multi-scale visual features to obtain aggregated features; and the third target query feature is generated based on the aggregated features and the second target query feature.

[0013] According to some embodiments, determining the rotated bounding box parameters and the probability of the target category corresponding to the target query parameters to obtain the model output includes: performing a rotated bounding box parameter regression operation on the target query parameters to obtain the rotated bounding box parameters; obtaining the category text vector through text semantic features; calculating the dot product similarity between the target query parameters and the category text vector to determine the probability of the target category; and combining the rotated bounding box parameters and the probability of the target category to obtain the model output.

[0014] According to some embodiments, the training sample data includes first sample data and second sample data; the training sample data for building the model includes: acquiring training set data, which includes image data corresponding to multiple target categories; determining the sampling probability of a preset first category detection target based on the training set data to obtain first sample data; and performing data augmentation processing on the first category detection target in the first sample data.

[0015] According to some embodiments, the training sample data for building the model further includes: stitching together multiple images in the training set data to generate a stitched image; and rotating the stitched image to obtain second sample data.

[0016] According to one aspect of this application, an apparatus for constructing a model for target detection in remote sensing images is provided, comprising: a feature extraction unit, configured to acquire multi-scale visual features of an input remote sensing image and textual semantic features of a preset text prompt, to generate semantically enhanced multi-scale visual features corresponding to the remote sensing image; an attention modulation unit, configured to perform cross-scale attention modulation on an initial model based on the semantically enhanced multi-scale visual features, to update the target query parameters of the model; a detection output unit, configured to determine the rotated bounding box parameters and the probability of the target category corresponding to the target query parameters, to obtain the output result of the model; a sample construction unit, configured to construct training sample data for the model; and a model training unit, configured to train the model based on the output result and the training sample data, to optimize the parameters of the model.

[0017] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to perform the method as described above.

[0018] According to one aspect of this application, a computer-readable storage medium is provided that stores a computer program or instructions thereon, which, when executed by a processor, implement the method as described above.

[0019] According to embodiments of this application, a semantically guided cross-scale attention modulation mechanism effectively alleviates the systemic problem of shallow high-resolution detail features being submerged by deep semantics in multi-scale feature interactions, significantly improving the recall rate and localization accuracy of the model for small and slender targets in remote sensing images. Furthermore, by directly regressing the rotated bounding box parameters, native modeling of targets in any orientation is achieved, avoiding the geometric errors introduced during the conversion from horizontal to rotated boxes, thus better adapting to application scenarios where remote sensing images exhibit a large target scale span and arbitrary directional distribution.

[0020] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application.

[0022] Figure 1 A flowchart illustrating a method for constructing a model for target detection in remote sensing images according to an example embodiment of this application is shown.

[0023] Figures 2A-2C This diagram illustrates a comparison of the detection performance of the model trained based on the embodiments of this application and the Grounding DINO model.

[0024] Figure 3 This diagram illustrates an apparatus for constructing a model for target detection in remote sensing images according to an example embodiment of this application.

[0025] Figure 4 A schematic diagram of an electronic device according to an example embodiment of this application is shown. Detailed Implementation

[0026] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0027] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of these specific details, or other methods, components, materials, apparatus, or operations may be employed. In these cases, well-known structures, methods, apparatuses, implementations, materials, or operations will not be shown or described in detail.

[0028] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0029] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0030] This application provides a method, apparatus, and device for constructing a model for target detection in remote sensing images. During the decoding process, a semantically guided cross-scale attention modulation mechanism is introduced, enabling the model to differentiate its focus on features at different scales based on textual semantic information. This avoids the weakening of shallow, fine-grained features during multi-scale interactions, thereby enhancing the modeling ability for small targets and local image information. Simultaneously, a direct regression method using rotated bounding box parameters is employed in the detection head to adapt to the arbitrary distribution of target orientations in remote sensing images. Combined with data augmentation strategies targeting small targets and rare categories, this synergistically improves the model's localization accuracy and robustness in complex scenes.

[0031] The following will describe in detail, with reference to the accompanying drawings, a method, apparatus, and device for constructing a model for target detection in remote sensing images according to embodiments of this application.

[0032] Figure 1 A flowchart illustrating a method for constructing a model for target detection in remote sensing images according to an example embodiment of this application is shown.

[0033] like Figure 1As shown, in step S100, the multi-scale visual features of the input remote sensing image and the text semantic features of the preset text prompt are obtained respectively to generate the semantically enhanced multi-scale visual features corresponding to the remote sensing image.

[0034] The device acquires the input remote sensing image and obtains the multi-scale visual features of the remote sensing image based on the preset feature dimensions.

[0035] According to some embodiments, the construction device can extract multi-scale visual features of remote sensing images through a visual backbone network (e.g., Swin-Tiny-224-1K), and project the extracted multi-scale visual features at each level onto feature dimensions pre-defined according to the task complexity. (e.g., 256 or 768) to obtain a multi-scale visual feature set. The multi-scale visual features at each level correspond to the hierarchical levels of the visual backbone network. Multi-scale visual feature set It can be expressed by the following formula.

[0036]

[0037] in, This indicates a hierarchical index, where features at different scales have different receptive fields. This represents the total number of feature levels. , They represent the first The height and width of the layer feature map. Shallow features retain rich details of small objects, while deep features contain global semantic information.

[0038] Meanwhile, the construction device obtains the corresponding text semantic features based on the preset target text prompts.

[0039] In some embodiments, the preset target text prompt is a string concatenated from the category names of the target to be detected. The construction device encodes the target text prompt using a language encoder (e.g., bert-base-uncased) to obtain the corresponding text semantic features. .

[0040]

[0041] in, This represents the total number of the smallest semantic units contained in a sequence of text semantic features. Indicates the first n The smallest semantic unit corresponds to Multi-dimensional feature vectors. Multi-scale visual features and textual semantic features are mapped to the same feature dimension space.

[0042] The device constructs a cross-modal enhancement system for multi-scale visual features and textual semantic features to generate semantically enhanced multi-scale visual features corresponding to remote sensing images.

[0043] According to some embodiments, the construction device inputs multi-scale visual features and textual semantic features into the cross-modal enhancement module of the encoder, and performs multi-layer visual-text interaction attention and visual self-attention updates through the encoder to obtain and output semantically enhanced multi-scale visual features. The semantically enhanced multi-scale visual features output by the encoder serve as input for cross-scale attention calculation and updates of the target query in the subsequent decoder, and can be expressed by the following formula.

[0044]

[0045] In step S200, based on semantically enhanced multi-scale visual features, cross-scale attention modulation is applied to the initial model to update the model's target query parameters.

[0046] The construction device obtains the first target query feature corresponding to the preset target query set.

[0047] According to some embodiments, the construction device obtains a target query set based on a predefined data structure and performs random initialization to detect whether a preset target exists in the remote sensing image through a decoder. The construction device will then... Layer target query set ={ The input decoder's self-attention module captures the spatial relationships and contextual dependencies between different targets, resulting in the first target query features updated through self-attention. Indicates the index of the target query ( ), Indicates the total number of target queries.

[0048] The construction device fuses the first target query features and text semantic features to obtain the second target query features.

[0049] According to some embodiments, the construction device combines the first target query features and the textual semantic features of the text prompt. The text cross-attention module of the input decoder is used to compute semantic information about the category of the fused target through cross-attention and obtain semantically enhanced second target query features. .

[0050] The construction device performs hierarchical weight prediction based on the second target query features to obtain the feature hierarchical weights corresponding to the semantic information of the target query set.

[0051] According to some embodiments, the construction device will construct the second target query features. The input decoder's hierarchical weight prediction network is used, and the feature hierarchical weights corresponding to the semantic information are calculated using the following formula to represent the current hierarchical weights. The first target corresponds to the The importance of layer feature maps.

[0052]

[0053] in, , The weight matrix is ​​a learnable matrix. The number of feature levels, This is a weight vector that reflects the current semantic preference for different feature levels.

[0054] Based on the second target query features, the device performs attention sampling on semantically enhanced multi-scale visual features to obtain multiple sampling points.

[0055] Furthermore, based on multiple sampling points, the device calculates the attention score corresponding to the semantically enhanced multi-scale visual features through the feature hierarchy weights corresponding to the semantic information.

[0056] According to some embodiments, the attention scores corresponding to the multi-scale visual features of semantic enhancement can be corrected by introducing feature level weights corresponding to semantic information, so that the model can determine the feature level most suitable for the current target, and then relatively enhance the weight of the feature level preferred by the semantics of the current target.

[0057] According to some embodiments, among multiple sampling points corresponding to the multi-scale visual features of semantic enhancement, let the first sampling point in the original attention be... The target query in the 1st At the feature level, the first The attention score for each sampling point is Then, after the feature level weights Corrected attention score It can be calculated and obtained through the following formula, to represent the first... On the layer feature map, the first The relevance of each sampling point to the current target query.

[0058]

[0059] in, To prevent the use of small, unstable constants, attention scores are calculated to allow the model to amplify the attention scores of important layers and suppress information from unimportant layers, based on the weights of the feature layers.

[0060] The construct aggregates attention scores with semantically enhanced multi-scale visual features in a weighted manner to obtain aggregated features, which are then output through a linear projection layer.

[0061] Furthermore, the construction device generates a third target query feature based on the aggregation feature and the second target query feature.

[0062] According to some embodiments, the construction apparatus can aggregate features and second target query features. Perform residual joins and normalization to obtain updated third target query features. .

[0063] The construction device performs nonlinear processing on the third target query features to obtain and update the target query parameters of the model.

[0064] According to some embodiments, the construction device will use third target query features The feedforward network (FFN) is input to the decoder to perform nonlinear transformation and integration on the third target query features, and the decoder's first-order feature is obtained through residual connections and layer normalization (LayerNorm). Target query parameters of the layer Target query parameters It can be obtained using the following formula.

[0065]

[0066] Among them, target query parameters It will be used as input to the next layer of the decoder, or fed into the detection head at the last layer of the decoder.

[0067] In step S300, the rotated bounding box parameters and the probability of the target category corresponding to the target query parameters are determined to obtain the output results of the model.

[0068] The construction device performs a rotation bounding box parameter regression operation on the target query parameters to obtain their corresponding rotation bounding box parameters.

[0069] According to some embodiments, let the set of target query parameter vectors output by the last layer of the decoder be... ,in This represents the total number of queries. The construction device uses each target query vector in this set. The bounding box regression branch network of the input model is used to predict the geometric offset of the current target relative to the reference point through a linear mapping, and then decoded into rotated bounding box parameters. Rotated bounding box parameters It can be expressed by the following formula.

[0070]

[0071] in, Indicates the center coordinates of the current target. and These represent the width and height of the current target, respectively. This indicates the current target's orientation angle. Through regression operations on the rotated bounding box parameters, accurate geometric modeling can be performed on remotely sensed targets with tilted, rotated, and elongated shapes.

[0072] The construction device obtains the category text vector by using the text semantic features of the target text prompt.

[0073] According to some embodiments, the construction apparatus constructs a sequence of textual semantic features containing all the smallest semantic units. The feature vectors corresponding to the smallest semantic units representing the names of each target category are selected and combined to form a set of category text vectors. Category text vector set It can be expressed by the following formula.

[0074]

[0075] in, This represents the total number of target categories contained in the set of category text vectors.

[0076] The device is constructed to calculate the dot product similarity between the target query parameters and the category text vectors to determine the probability of the target category.

[0077] According to some embodiments, based on target query parameters With category text vectors The construction device calculates the dot product similarity between the two using the following formula. This serves as the correlation score between the target query and each target category.

[0078]

[0079] Furthermore, the constructing device normalizes the score of each target query using softmax along the category dimension to obtain the probability distribution of each target category corresponding to the target query. That is, the first The target query belongs to the first The probability of each target category.

[0080] The construct combines the rotated bounding box parameters corresponding to the target query parameters with the probability of the target category to obtain the model's output, namely the text-guided target detection result.

[0081] In step S400, the training sample data for the model is constructed.

[0082] According to some embodiments, the training sample data includes first sample data and second sample data.

[0083] Construct a device to obtain training set data.

[0084] According to some embodiments, the training set data includes image data corresponding to multiple target categories.

[0085] The construction device determines the sampling probability of the first category of detection target based on the training set data to obtain the first sample data.

[0086] According to some embodiments, sampling weights can be set based on the number of samples of each category in the training set data. This allows for sampling of sample data according to the sampling weights when constructing training batches, thereby mitigating the impact of uneven distribution of the target category on model training. (The number of samples of each category in the training set data is mentioned twice.) The sampling weights can be expressed by the following formula.

[0087]

[0088] in, To prevent smoothing constants with a denominator of zero.

[0089] Furthermore, the device can improve the sampling probability of the first category of detected targets by adopting a category balancing or hierarchical sampling strategy based on the sampling weight.

[0090] According to some embodiments, the first category of detected targets includes targets with bounding boxes having an area less than 32. Small targets, typically 32 pixels in size. These targets are usually blurry in remote sensing images and difficult to detect.

[0091] According to some embodiments, the first category of detected targets also includes rare category targets. Based on the statistical distribution of the training set data, when the number of samples in a certain category is much lower than the average number of samples or a preset threshold, the construction device determines that the category is a rare category.

[0092] The device performs data augmentation processing on the first category of detection targets in the first sample data.

[0093] According to some embodiments, data augmentation processing includes operations such as cropping, rotating, or pasting the image region corresponding to the first category of detected targets to increase the frequency of occurrence of the first category of detected targets in the training samples.

[0094] For example, let the rotated bounding box of a certain first-class detected target be labeled as Apply a rotation angle to the area of ​​the target. Then, the center and orientation angle of the target can be updated according to the following expression.

[0095]

[0096] in, This indicates a coordinate rotation transformation operation centered at the origin. The construction device then pastes the enhanced region of the target to the new location. At that time, the center coordinates of the target are updated synchronously according to the following formula.

[0097]

[0098] While obtaining the first sample data, the construction device also stitches together multiple images from the training set data to generate a stitched image.

[0099] Then, the construction device rotates the stitched image to obtain second sample data.

[0100] According to some embodiments, the construction device stitches together several sub-region images from different images to form a stitched image, and applies an overall rotation angle to the stitched image. For any target in the stitched image, its bounding box orientation angle is updated synchronously. This allows for the construction of a second sample of data that includes targets in multiple directions and complex spatial layouts.

[0101] In step S500, the model is trained using training sample data based on the output results to optimize the model parameters.

[0102] Based on the text-guided target detection results output by the model, the construction device trains the model using training sample data until the model parameters converge, thereby obtaining a text-enhanced remote sensing target detection model.

[0103] According to some embodiments, the construction device optimizes the model parameters by minimizing the training objective function corresponding to the detection task until the model parameters converge.

[0104] For example, the AdamW optimizer can be used to optimize the model parameters, with an initial learning rate set to 5×10. -5 The weight decay coefficient was set to 0.01 to suppress overfitting while ensuring stable model convergence. The training batch size was set to 8, and the model was iteratively trained using remote sensing images and their corresponding text annotations from the DOTA dataset. The training process consisted of 30 rounds.

[0105] According to some embodiments, during the training process of the model, the construction device updates all learnable parameters of the model end-to-end through the backpropagation algorithm, and finally obtains a text-enhanced remote sensing target detection model.

[0106] To verify the effectiveness of the embodiments of this application, under the same dataset and training configuration, a comparative experiment was conducted between the model trained based on the embodiments of this application and the Grounding DINO model. The comparison diagram of their detection performance is shown below. Figures 2A-2C As shown. Among them, Figures 2A-2C The left side shows the detection results of the model trained based on the embodiments of this application, and the right side shows the detection results of the Grounding DINO model.

[0107] Experimental results show that, compared with the horizontal bounding box output by the Grounding DINO model, the rotated bounding box output by the model trained based on the embodiments of this application can more accurately locate the target referred to by the language description. Especially when the target has obvious rotation, tilt or small scale (such as densely arranged vehicles or ships), the embodiment of this application can effectively reduce background redundancy and show higher positioning accuracy and more stable detection effect.

[0108] Figure 3 This diagram illustrates an apparatus for constructing a model for target detection in remote sensing images according to an example embodiment of this application.

[0109] like Figure 3 As shown, the construction device 100 includes a feature extraction unit 110, an attention modulation unit 120, a detection output unit 130, a sample construction unit 140, and a model training unit 150.

[0110] The feature extraction unit 110 acquires the input remote sensing image and obtains multi-scale visual features of the remote sensing image based on the preset feature dimensions.

[0111] The feature extraction unit 110 obtains the corresponding text semantic features based on the preset target text prompt.

[0112] The feature extraction unit 110 performs cross-modal enhancement on multi-scale visual features and textual semantic features to generate semantically enhanced multi-scale visual features corresponding to the remote sensing image.

[0113] Attention modulation unit 120 acquires the first target query feature corresponding to the preset target query set.

[0114] The attention modulation unit 120 fuses the first target query features and text semantic features to obtain the second target query features.

[0115] The attention modulation unit 120 performs hierarchical weight prediction based on the second target query features to obtain the feature hierarchical weights corresponding to the semantic information of the target query set.

[0116] Based on the second target query features, the attention modulation unit 120 performs attention sampling on the semantically enhanced multi-scale visual features to obtain multiple sampling points.

[0117] Based on multiple sampling points, the attention modulation unit 120 calculates the attention score corresponding to the semantically enhanced multi-scale visual features through the feature level weights corresponding to the semantic information.

[0118] The attention modulation unit 120 weights and aggregates the attention score with the semantically enhanced multi-scale visual features to obtain aggregated features, which are then output through a linear projection layer.

[0119] Attention modulation unit 120 generates third target query features based on aggregated features and second target query features.

[0120] The attention modulation unit 120 performs nonlinear processing on the third target query features to obtain and update the target query parameters of the model.

[0121] The detection output unit 130 performs a rotation bounding box parameter regression operation on the target query parameters to obtain their corresponding rotation bounding box parameters.

[0122] The detection output unit 130 obtains the category text vector through the text semantic features of the target text prompt.

[0123] The detection output unit 130 calculates the dot product similarity between the target query parameters and the category text vector to determine the probability of the target category.

[0124] The detection output unit 130 combines the rotated bounding box parameters corresponding to the target query parameters with the probability of the target category to obtain the output result of the model.

[0125] The sample construction unit 140 acquires training set data and determines the sampling probability of the preset first category detection target based on the training set data to obtain the first sample data.

[0126] The sample construction unit 140 performs data augmentation processing on the first category of detection targets in the first sample data.

[0127] While obtaining the first sample data, the sample construction unit 140 also stitches together multiple images from the training set data to generate a stitched image.

[0128] The sample construction unit 140 rotates the stitched image to obtain second sample data.

[0129] Based on the text-guided target detection results output by the model, the model training unit 150 trains the model using training sample data until the model parameters converge, so as to obtain a text-enhanced remote sensing target detection model.

[0130] Figure 4 A schematic diagram of an electronic device according to an example embodiment of this application is shown.

[0131] like Figure 4 As shown, the electronic device 600 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0132] like Figure 4 As shown, the electronic device 600 is manifested in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc. The storage unit stores program code, which can be executed by the processing unit 610, causing the processing unit 610 to perform the methods described in this specification according to the various exemplary embodiments of this application. For example, the processing unit 610 can perform, for example... Figure 1 The method shown.

[0133] Storage unit 620 may include readable media in the form of volatile storage units, such as random access memory (RAM) 6201 and / or cache 6202, and may further include read-only memory (ROM) 6203.

[0134] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0135] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0136] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0137] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. The technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0138] Software products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0139] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0140] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0141] The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the computer-readable medium to perform the aforementioned functions.

[0142] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0143] The embodiments of this application have been described in detail above. These descriptions are solely for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, its specific implementation methods, and its application scope, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for constructing a model for target detection in remote sensing images, characterized in that, include: The multi-scale visual features of the input remote sensing image and the text semantic features of the preset text prompt are obtained respectively to generate the semantically enhanced multi-scale visual features corresponding to the remote sensing image. Based on the semantically enhanced multi-scale visual features, cross-scale attention modulation is applied to the initial model to update the target query parameters of the model. Determine the rotated bounding box parameters and the probability of the target category corresponding to the target query parameters to obtain the output result of the model; Construct the training sample data for the model; Based on the output results, the model is trained using the training sample data to optimize the model's parameters.

2. The method according to claim 1, characterized in that, The multi-scale visual features of the input remote sensing image and the textual semantic features of the preset text prompt are acquired respectively to generate semantically enhanced multi-scale visual features corresponding to the remote sensing image, including: The multi-scale visual features are obtained based on the preset feature dimensions; The semantic features of the text are obtained based on the preset target text prompts; Cross-modal enhancement is performed on the multi-scale visual features and the text semantic features to generate the semantically enhanced multi-scale visual features.

3. The method according to claim 1, characterized in that, Based on the semantically enhanced multi-scale visual features, cross-scale attention modulation is applied to the initial model to update the target query parameters of the model, including: Obtain the first target query feature corresponding to the preset target query set; Based on the first target query feature and the text semantic feature, obtain the feature level weights corresponding to the semantic information of the target query set; Based on the semantically enhanced multi-scale visual features, the model is subjected to cross-scale attention modulation through the feature hierarchy weights to obtain the third target query features; The third target query feature is subjected to non-linear processing to obtain and update the target query parameters.

4. The method according to claim 3, characterized in that, Based on the first target query feature and the text semantic feature, the feature level weights corresponding to the semantic information of the target query set are obtained, including: The first target query feature and the text semantic feature are fused together to obtain the second target query feature; Based on the second target query features, hierarchical weight prediction is performed to obtain the feature hierarchical weights.

5. The method according to claim 4, characterized in that, Based on the semantically enhanced multi-scale visual features, the model is subjected to cross-scale attention modulation through the feature hierarchy weights to obtain the third target query features, including: Based on the second target query feature, attention sampling is performed on the semantically enhanced multi-scale visual feature to obtain multiple sampling points; Based on the multiple sampling points, the attention score corresponding to the semantically enhanced multi-scale visual feature is calculated through the feature hierarchy weights; The attention score is weighted and aggregated with the semantically enhanced multi-scale visual features to obtain aggregated features; The third target query feature is generated based on the aggregated feature and the second target query feature.

6. The method according to claim 1, characterized in that, Determine the rotated bounding box parameters and the probability of the target category corresponding to the target query parameters to obtain the output of the model, including: Perform a rotated bounding box parameter regression operation on the target query parameters to obtain the rotated bounding box parameters; The category text vector is obtained through the aforementioned text semantic features; Calculate the dot product similarity between the target query parameter and the category text vector to determine the probability of the target category; The rotation bounding box parameters and the probability of the target category are combined to obtain the output of the model.

7. The method according to claim 1, characterized in that, The training sample data includes first sample data and second sample data; The training sample data used to construct the model includes: Acquire training set data, which includes image data corresponding to multiple target categories; The sampling probability of a preset first category of detection target is determined based on the training set data to obtain the first sample data; Data augmentation processing is performed on the first category of detected targets in the first sample data.

8. The method according to claim 7, characterized in that, The training sample data for constructing the model also includes: Multiple images from the training set data are stitched together to generate a stitched image; The stitched image is rotated to obtain the second sample data.

9. An apparatus for constructing a model for target detection in remote sensing images, characterized in that, include: The feature extraction unit is used to acquire the multi-scale visual features of the input remote sensing image and the text semantic features of the preset text prompt, respectively, so as to generate the semantically enhanced multi-scale visual features corresponding to the remote sensing image. An attention modulation unit is used to perform cross-scale attention modulation on the initial model based on the semantically enhanced multi-scale visual features in order to update the target query parameters of the model. The detection output unit is used to determine the rotated bounding box parameters and the probability of the target category corresponding to the target query parameters, so as to obtain the output result of the model; A sample construction unit is used to construct training sample data for the model. The model training unit is used to train the model based on the output results and the training sample data to optimize the parameters of the model.

10. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-8.

Images