Target positioning method and device, electronic equipment and storage medium

Abstract

The invention provides a target positioning method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: inputting a target remote sensing image and a target description text used for describing an interested target into a target positioning model, acquiring position information of an interested target in a target remote sensing image output by the target positioning model; the target positioning model is obtained by training based on a sample remote sensing image, a sample description text used for describing a sample target in the sample remote sensing image and position information of the sample target in the sample remote sensing image. According to the target positioning method and device, the electronic equipment and the storage medium provided by the invention, the interested target in the remote sensing image can be positioned more accurately and more efficiently according to the description of the text on the interested target, synchronous positioning of multiple interested targets in the remote sensing image can be realized, the flexibility of target positioning on the remote sensing image is higher, and the user experience is improved. Wide application prospects are realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a target positioning method, device, electronic device and storage medium. Background Art

[0002] In key areas such as maritime traffic management, environmental monitoring and disaster emergency response, remote sensing images are widely used to locate key targets such as aircraft, ships, stations and stadiums.

[0003] The traditional target positioning method in the related art can use computer vision technology to locate key targets in remote sensing images through traditional discriminant specialized models. However, the above traditional discriminant specialized models are difficult to achieve simultaneous positioning of multiple different types of targets in remote sensing images, and the flexibility of target positioning is not high.

[0004] Traditional target positioning methods in related technologies can also use traditional generative multimodal large models to locate key targets in remote sensing images by combining visual information with language descriptions. However, the above traditional generative multimodal large models have inherent defects in positioning accuracy. Therefore, how to improve the positioning accuracy and flexibility of locating targets in remote sensing images is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0005] The present invention provides a target positioning method, device, electronic device and storage medium, which are used to solve the defects of low flexibility and positioning accuracy in the prior art for positioning targets in remote sensing images, and to improve the positioning accuracy and flexibility of positioning targets in remote sensing images.

[0006] The present invention provides a target positioning method, comprising the following steps.

[0007] Obtaining target remote sensing images and target description text for describing the target of interest; Inputting the target remote sensing image and the target description text into a target positioning model, and obtaining the position information of the target of interest in the target remote sensing image output by the target positioning model; The target positioning model is obtained after training based on sample remote sensing images, sample description texts used to describe sample targets in the sample remote sensing images, and position information of the sample targets in the sample remote sensing images.

[0008] According to a target positioning method provided by the present invention, the target positioning model includes: a visual feature extraction module and a large language model; Inputting the target remote sensing image and the target description text into a target localization model to obtain the position information of the target of interest in the target remote sensing image output by the target localization model includes: Input the target remote sensing image into the visual feature extraction module to obtain the first visual feature information corresponding to the target remote sensing image output by the visual feature extraction module. Input the target description text into the tokenizer in the large language model to obtain the semantic feature information corresponding to the target description text output by the tokenizer. The first visual feature information and the semantic feature information are in the same dimensional space; Perform feature splicing on the first visual feature information and the semantic feature information to obtain target splicing feature information; Input the target splicing feature information into the backbone network in the large language model to obtain the hidden state information of each token in the target splicing feature information output by the backbone network; Input the hidden state information of each token in the target splicing feature information into the output layer in the large language model to obtain the position information of the target of interest in the target remote sensing image output by the output layer.

[0009] According to a target localization method provided by the present invention, the visual feature extraction module includes: a semantic encoder and a vision-language semantic processing module; The step of inputting the target remote sensing image into the visual feature extraction module to obtain the first visual feature information corresponding to the target remote sensing image output by the visual feature extraction module includes: Input the target remote sensing image into the semantic encoder, and the semantic encoder extracts features from the target remote sensing image, thereby obtaining the second visual feature information corresponding to the target remote sensing image output by the semantic encoder; Input the second visual feature information into the vision-language semantic processing module, and the vision-language semantic processing module compresses and performs dimensional conversion on the second visual feature information to obtain the first visual feature information output by the vision-language semantic processing module.

[0010] According to a target localization method provided by the present invention, the target localization model is trained based on the following steps: Obtain the sample remote sensing image, the sample description text, and the position information of the sample target in the sample remote sensing image; Construct an initial model, where the initial model includes a pre-trained semantic encoder, a vision-language semantic processing module, and a pre-trained large language model; Determine the initial model as the target localization model being trained, input the sample remote sensing image and the sample description text into the target localization model being trained, obtain the first predicted position information of the sample target in the sample remote sensing image output by the target localization model being trained, and obtain the hidden state information of each special token in the sample splicing feature information output by the backbone network of the large language model in the target localization model being trained; Input the hidden state information of each special token in the sample splicing feature information into the decoding model, and obtain the second predicted position information of the sample target in the sample remote sensing image output by the decoding model; Based on the first predicted position information of the sample target in the sample remote sensing image, the second predicted position information of the sample target in the sample remote sensing image, and the position information of the sample target in the sample remote sensing image, calculate the loss function value of the target localization model being trained; When it is determined that the loss function value of the target localization model being trained converges, determine that the target localization model being trained converges, and obtain the trained target localization model. When it is determined that the loss function value of the target localization model being trained does not converge, update the model parameters of the target localization model being trained, and return to execute the step of inputting the sample remote sensing image and the sample description text into the target localization model being trained until it is determined that the loss function value of the target localization model being trained converges.

[0011] According to a target localization method provided by the present invention, calculating the loss function value of the target localization model being trained based on the first predicted position information of the sample target in the sample remote sensing image, the second predicted position information of the sample target in the sample remote sensing image, and the position information of the sample target in the sample remote sensing image includes: Based on the first predicted position information of the sample target in the sample remote sensing image and the position information of the sample target in the sample remote sensing image, calculate the first loss function value of the target localization model being trained. Based on the first predicted position information of the sample target in the sample remote sensing image and the position information of the sample target in the sample remote sensing image, calculate the second loss function value of the target localization model being trained; Based on the first loss function value and the second loss function value of the target localization model being trained, calculate the loss function value of the target localization model being trained.

[0012] According to a target localization method provided by the present invention, the first loss function includes a cross-entropy loss function; The second loss function includes at least one of mean absolute error loss function, mean squared error loss function, smooth absolute error loss function, smooth quadratic error loss function, intersection over union loss function, generalized intersection over union loss function, distance intersection over union loss function, and complete intersection over union loss function.

[0013] The present invention also provides a target positioning device, including the following modules: A data acquisition module, configured to acquire a target remote sensing image and a target description text for describing an interested target. A target positioning module, configured to input the target remote sensing image and the target description text into a target positioning model, and acquire position information of the interested target in the target remote sensing image output by the target positioning model. Wherein, the target positioning model is obtained after being trained based on a sample remote sensing image, a sample description text for describing a sample target in the sample remote sensing image, and position information of the sample target in the sample remote sensing image.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the target positioning method as described in any one of the above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the target positioning method as described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the target positioning method as described in any one of the above is implemented.

[0017] The target positioning method, device, electronic device, and storage medium provided by the present invention, after inputting a target remote sensing image and a target description text for describing an interested target into a target positioning model, acquire position information of the interested target in the target remote sensing image output by the target positioning model. The above target positioning model is obtained after being trained based on a sample remote sensing image, a sample description text for describing a sample target in the sample remote sensing image, and position information of the sample target in the sample remote sensing image. It can more accurately and efficiently locate the interested target in the remote sensing image according to the text description of the interested target, can realize the synchronous positioning of multiple interested targets in the remote sensing image, has higher flexibility in target positioning of the remote sensing image, can better meet the actual needs of key fields such as maritime traffic management, environmental monitoring, and disaster emergency response, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 It is a schematic flowchart of the target positioning method provided by the present invention.

[0020] Figure 2 It is a schematic flowchart of the training process of the target positioning model in the target positioning method provided by the present invention.

[0021] Figure 3 It is a schematic diagram of the training framework of the target positioning model in the target positioning method provided by the present invention.

[0022] Figure 4 It is a schematic diagram of the structure of the target positioning model in the target positioning method provided by the present invention.

[0023] Figure 5 It is a schematic diagram of the structure of the target positioning device provided by the present invention.

[0024] Figure 6 It is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed implementation manners

[0025] To make the objectives, technical solutions and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention with reference to the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0026] In the description of the present invention, it should be noted that unless otherwise clearly defined and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0027] In the description of this application, the terms "first", "second", etc. are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of this application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually of the same category, and do not limit the number of objects. For example, the first object can be one or more. In addition, in the description of this application, "and / or" means at least one of the connected objects, and the character " / ", generally indicates that the related objects before and after are in an "or" relationship.

[0028] It should be noted that in the traditional target positioning method in the related art, when using computer vision technology to locate targets in remote sensing images, the traditional discriminative specialized model can only locate a certain type of target in the remote sensing image, or locate all targets in the remote sensing image. Although the above traditional discriminative specialized model performs well in the positioning task of a single target, its application scenario is limited and it cannot handle the positioning tasks of multiple different types of targets at the same time, thus unable to meet the synchronous positioning requirements of multiple different types of targets.

[0029] However, in practical applications, it is often necessary to handle the positioning tasks of multiple different types of targets at the same time. For example, in a monitoring scenario, it is not only necessary to accurately locate a target with a specific description in the remote sensing image (such as "a helicopter with a red fuselage"), but also necessary to accurately locate multiple targets belonging to a certain category (such as "parked ships").

[0030] In order to meet the need to handle the positioning tasks of multiple different types of targets at the same time in practical applications, the model needs to be more flexible. The model not only needs to be able to accurately locate any number of multiple types of targets in the remote sensing image, but also needs to have a deep understanding of the characteristics of complex targets to ensure seamless switching between the positioning tasks of different types of targets, so as to meet the requirements of real-time monitoring and efficient analysis of key targets in remote sensing images in practical applications.

[0031] Moreover, if the above traditional discriminative specialized model needs to locate a new type of target, it usually has to retrain the above traditional discriminative specialized model or adjust the above traditional discriminative specialized model, which not only requires a large amount of time and labor costs, but also increases the investment in computing and deployment costs.

[0032] With the development of multi-modal large language models, new technological breakthroughs have been brought to the understanding and analysis of remote sensing images. Traditional generative multi-modal large models can combine visual information with language descriptions and have the ability to process the alignment of images and texts. In related technologies, traditional object localization methods can also utilize the above-mentioned traditional generative multi-modal large models to localize key objects in remote sensing images by combining visual information with language descriptions.

[0033] However, the above-mentioned traditional generative multi-modal large models have inherent defects in accurate position prediction. Specifically, the above-mentioned traditional generative multi-modal large models can usually only align visual features with text at the image level and lack the ability to extract fine-grained information, making it difficult to accurately localize the positions of key objects. Moreover, since the above-mentioned traditional generative multi-modal large models treat the numbers representing positions (such as coordinates) as ordinary tokens for processing, the position prediction of key objects is reduced to a classification problem, and the distance between the prediction result and the true bounding box cannot be effectively measured, resulting in low accuracy of key object localization and inability to meet the high-precision object localization requirements of some key application scenarios. In addition, the above-mentioned traditional generative multi-modal large models have a large parameter scale, consume a large amount of computing resources during training and inference, and are difficult to be efficiently deployed in resource-constrained environments.

[0034] In summary, in the traditional object localization methods in related technologies, traditional discriminative specialized models are usually targeted at the localization tasks of single objects, lack flexibility, and cannot handle the localization tasks of multiple different types of objects at the same time. Moreover, the above-mentioned traditional discriminative specialized models need to be retrained or adjusted to achieve the localization of new types of objects, increasing the input of time cost, labor cost, computing cost, and deployment cost. The generative multi-modal large models have defects in localization accuracy, lack the ability to extract fine-grained information, resulting in inaccurate object position prediction, and the above-mentioned generative multi-modal large models consume a large amount of computing resources and are difficult to be efficiently deployed in resource-constrained environments.

[0035] To address the above technical problems, the present invention provides an object localization method for remote sensing images. Compared with the traditional object localization methods in related technologies that rely on traditional discriminative specialized models and traditional generative multi-modal large models, the object localization method provided by the present invention has achieved significant optimization in terms of localization accuracy, flexibility, and computing efficiency.

[0036] The object localization method provided by the present invention proposes a multi-modal large model. The above-mentioned multi-modal large model can achieve spatial fine-grained perception and support users to query objects through natural language input. Whether describing specific object features or inputting category names, the above-mentioned multi-modal large model can accurately detect and localize the objects that meet the conditions, avoiding the cumbersome processes of model switching and manual screening.

[0037] Through the optimization of the model structure, it is possible to ensure that the above-mentioned multimodal large model extracts fine-grained information during the alignment of visual features and text semantics, making the positioning of the target more accurate. In addition, by improving the position prediction strategy, the above-mentioned multimodal large model can better measure the distance between the predicted bounding box and the ground truth, significantly improving the accuracy and reliability of the positioning.

[0038] The target positioning method provided by the present invention improves the positioning accuracy while reducing the computational cost, can flexibly handle various complex positioning tasks, and has broad application prospects in key fields such as maritime traffic management, environmental monitoring, and disaster emergency response. The target positioning method provided by the present invention provides a more efficient, accurate, and flexible solution for remote sensing image analysis by organically combining visual perception and language understanding.

[0039] The following combines Figures 1-4 to describe the target positioning method provided by the present invention.

[0040] Figure 1 is a schematic flowchart of the target positioning method provided by the present invention. As Figure 1 shown, the method includes the following: Step 101, obtain a target remote sensing image and a target description text for describing the target of interest.

[0041] It should be noted that the execution subject of the embodiments of the present invention is a target positioning device. The above-mentioned target positioning device can be configured in electronic devices such as computers and servers.

[0042] Specifically, the target of interest in the target remote sensing image is the positioning object of the target positioning method provided by the present invention. Based on the target positioning method provided by the present invention, the target of interest in the target remote sensing image can be positioned, so as to obtain the position information of the target of interest in the target remote sensing image.

[0043] Among them, the position information of the target of interest in the target remote sensing image in the embodiments of the present invention may include the position coordinates of the target of interest in the target remote sensing image, and may also include the position coordinates of the target bounding box in the target remote sensing image. The above-mentioned target bounding box is the smallest square including the target of interest. Further, based on the position information of the target of interest in the target remote sensing image and the mapping relationship between the target remote sensing image and the real world, the position information of the target of interest in the real world can be obtained.

[0044] Optionally, the position information of the target of interest in the target remote sensing image includes the position coordinates of the target bounding box in the target remote sensing image, in the form of [x1, y1, x2, y2], where (x1, y1) represents the coordinates of the upper left point of the target bounding box, and (x2, y2) represents the coordinates of the lower right point of the target bounding box.

[0045] It can be understood that the target remote sensing image and the target of interest in the embodiments of the present invention can be determined based on actual needs. For example, the target of interest can include one or more artificial targets with important positioning value such as airplanes, ships, vehicles, bridges, oil storage tanks, stations, and sports fields. In the embodiments of the present invention, the target remote sensing image and the target of interest are not specifically limited.

[0046] It should be noted that the target remote sensing image in the embodiments of the present invention can be a remote sensing image taken by an extraterrestrial satellite.

[0047] It should be noted that in the embodiments of the present invention, the text used to describe the target of interest can be determined as the target description text. The target description text can include, but is not limited to, the category information and / or attribute information of the target of interest. The above attribute information can include at least one of, but is not limited to, color information, size information, position information, and mutual relationship information; the above mutual relationship information is used to describe the mutual relationship between the target of interest and other targets.

[0048] In the embodiments of the present invention, the target remote sensing image and the target description text can be obtained in various ways. For example, the target remote sensing image can be obtained based on data query, and the target description text can be obtained based on the input of the user. In the embodiments of the present invention, the specific ways of obtaining the target remote sensing image and the target description text are not limited.

[0049] Step 102: Input the target remote sensing image and the target description text into the target positioning model, and obtain the position information of the target of interest in the target remote sensing image output by the target positioning model.

[0050] Among them, the target positioning model is obtained after being trained based on the sample remote sensing image, the sample description text used to describe the sample target in the sample remote sensing image, and the position information of the sample target in the sample remote sensing image.

[0051] It should be noted that the sample target in the embodiments of the present invention can be determined based on prior knowledge and / or actual needs. For example, the sample target can include multiple artificial targets with important positioning value such as airplanes, ships, vehicles, bridges, oil storage tanks, stations, and sports fields. The sample remote sensing image in the embodiments of the present invention can be a remote sensing image taken by an extraterrestrial satellite containing the sample target.

[0052] In the embodiments of the present invention, the text used to describe the sample target can be determined as the sample description text. The sample description text may include, but is not limited to, the category information and / or attribute information of the sample target.

[0053] In the embodiments of the present invention, the position information of the sample target in the sample remote sensing image may include the position coordinates of the sample target in the sample remote sensing image, and may also include the position coordinates of the sample bounding box in the sample remote sensing image. The above sample bounding box is the smallest square including the sample target.

[0054] Optionally, the position information of the sample target in the sample remote sensing image includes the position coordinates of the sample bounding box in the sample remote sensing image, in the form of [X1, Y1, X2, Y2], where (X1, Y1) represents the coordinates of the upper left point of the sample bounding box, and (X2, Y2) represents the coordinates of the lower right point of the sample bounding box. The position coordinates of the sample bounding box in the sample remote sensing image are normalized by the image length and fixed within a specific range.

[0055] The target positioning model in the embodiments of the present invention is constructed based on deep learning technology and obtained after training based on the sample remote sensing image, the above sample description text, and the position information of the sample target in the sample remote sensing image.

[0056] After obtaining the target remote sensing image and the target description text, the target remote sensing image and the target description text can be input into the trained target positioning model.

[0057] The target positioning model can find the target matching the target description text in the target remote sensing image based on the target description text, as the target of interest in the target remote sensing image, and then can obtain and output the position information of the target of interest in the target remote sensing image.

[0058] In the embodiments of the present invention, after inputting the target remote sensing image and the target description text used to describe the target of interest into the target positioning model, the position information of the target of interest in the target remote sensing image output by the target positioning model is obtained. The above target positioning model is obtained after training based on the sample remote sensing image, the sample description text used to describe the sample target in the sample remote sensing image, and the position information of the sample target in the sample remote sensing image. It can more accurately and efficiently locate the target of interest in the remote sensing image according to the description of the target of interest in the text, can realize the synchronous positioning of multiple targets of interest in the remote sensing image, has higher flexibility in target positioning of the remote sensing image, can better meet the actual needs of key fields such as maritime traffic management, environmental monitoring, and disaster emergency response, and has broad application prospects.

[0059] Figure 2It is a schematic diagram of the training process of the target positioning model in the target positioning method provided by the present invention. As Figure 2 shown, as an optional embodiment, the target positioning model is trained based on the following steps: obtaining a sample remote sensing image, a sample description text, and the position information of a sample target in the sample remote sensing image.

[0060] Specifically, in the embodiments of the present invention, a large number of remote sensing images can be obtained from existing public data sets (such as DIOR and DOTA, etc.) as sample remote sensing images.

[0061] After obtaining the sample remote sensing image, a target detection annotation software can be used to annotate the sample target in the sample remote sensing image with a sample bounding box, and the position information of the sample target in the sample remote sensing image can be obtained.

[0062] In the embodiments of the present invention, the text for describing the above sample target can be obtained in various ways as the sample description text. For example, the above sample description text can be obtained from the existing public data sets of visual positioning tasks, or the above sample description text can be obtained through user input.

[0063] It should be noted that in the embodiments of the present invention, a data augmentation method with an unchanged sample description text can be adopted to expand the number of sample remote sensing images. Among them, the above data augmentation method can include but is not limited to color transformation and position transformation of the sample remote sensing image. The color transformation includes adjusting the contrast, sharpness, and brightness of the sample remote sensing image, etc.; the position transformation includes slightly rotating and translating the sample remote sensing image (without changing the position relationship between the targets in the image), and horizontal or vertical flipping combined with the sample description text. When flipping the sample remote sensing image, the position information in the sample description text is extracted, and the position description is equally transformed, and the above transformations are randomly selected according to a certain ratio.

[0064] It can be understood that there is a corresponding relationship between the position information of the sample target in the sample remote sensing image and the sample description text.

[0065] Optionally, in the embodiments of the present invention, based on the corresponding relationship between the position information of the sample target in the sample remote sensing image and the sample description text, the sample description text and the position information of the sample target in the sample remote sensing image can be converted into a question-and-answer form data set. Among them, the question part gives instructions, such as "locate the target that meets the sample description text", and attaches the above sample description text, and the answer part gives the position information of the sample target in the sample remote sensing image that meets the above sample description text.

[0066] Construct an initial model, and the initial model includes a pre-trained semantic encoder, a vision-language semantic processing module, and a pre-trained large language model.

[0067] The initial model includes a pre-trained semantic encoder, a pre-trained large language model, and a vision-language semantic processing module.

[0068] Specifically, the semantic encoder in the initial model is obtained through pre-training on a large amount of images. It is a semantic encoder suitable for fine-grained high-resolution remote sensing images, with powerful feature extraction and processing capabilities, and can encode the input high-resolution remote sensing images (1024×1024 or larger) into numerical representations.

[0069] The pre-training tasks for the above-mentioned semantic encoder in the embodiments of the present invention may include but are not limited to semantic segmentation, masked autoencoder (MAE), and object detection, etc.

[0070] The structure of the above-mentioned semantic encoder may include but is not limited to classical convolutional neural network (CNN) architectures, such as VGG and ResNet, and models built based on the attention mechanism, such as vision transformers (ViT) and Swin Transformer at multiple scales, etc. The above-mentioned semantic encoder adopts the shifted windows self-attention mechanism.

[0071] The size of the input image of the above-mentioned semantic encoder is 1024*1024. For input images with a size larger than 1024*1024, feature extraction can be performed through a 1024*1024 sliding window, and the sliding window stride is 1024. For input images with a size smaller than 1024*1024, dynamic interpolation can be performed on the position encoding to adapt to the size of the input image.

[0072] The large language model in the initial model is obtained through pre-training based on a large amount of text data, and has the ability to understand, generate, and reason natural language, and can map complex language information into useful numerical representations.

[0073] The above-mentioned large language model can perform basic language tasks (such as text classification, question answering, and translation), and meet the requirements of complex scenarios through multi-round conversations and context reasoning.

[0074] The structure of the above-mentioned large language model may include but is not limited to open-source large language models such as LLaMA (7B, 13B, 65B), Vicuna (7B, 13B), or OPT (125M, 350M, 1.3B, 6.7B, 13B, 175B), etc.

[0075] The above-mentioned large language model includes a tokenizer, a backbone network, and an output layer.

[0076] The visual language semantic processing module in the initial model can compress visual features and transform them into the dimensional space of the large language model while maintaining the semantic invariance of the visual features.

[0077] The structure of the above visual language semantic processing module can include but is not limited to structures such as a single-layer linear mapping layer, a multi-layer perceptron, or a Q-Former.

[0078] Determine the initial model as the target localization model during training. Input the sample remote sensing image and the sample description text into the target localization model during training, obtain the first predicted position information of the sample target in the sample remote sensing image output by the target localization model during training, and obtain the hidden state information of each special token in the sample concatenated feature information output by the backbone network of the large language model in the target localization model during training.

[0079] Figure 3 It is a schematic diagram of the training framework of the target localization model in the target localization method provided by the present invention. As Figure 3 shown, after constructing the initial model, the above initial model can be determined as the target localization model during training. Input the sample remote sensing image into the semantic encoder in the above target localization model during training, and input the sample description text into the tokenizer in the large language model in the above target localization model during training.

[0080] The semantic encoder in the above target localization model during training can perform visual feature extraction on the sample remote sensing image, and obtain and output the second visual feature information corresponding to the sample remote sensing image.

[0081] The tokenizer in the large language model in the above target localization model during training can perform semantic feature extraction on the sample description text, and obtain and output the semantic feature information corresponding to the sample description text.

[0082] After obtaining the second visual feature information corresponding to the sample remote sensing image output by the semantic encoder in the above target localization model during training, the second visual feature information corresponding to the sample remote sensing image can be input into the visual language semantic processing module in the above target localization model during training.

[0083] The visual language semantic processing module in the above target localization model during training can compress and perform dimensional conversion on the second visual feature information corresponding to the sample remote sensing image. On the basis of maintaining the semantic invariance of the second visual feature information corresponding to the sample remote sensing image, the second visual feature information corresponding to the sample remote sensing image is transformed into the dimensional space of the large language model, and then the first visual feature information corresponding to the sample remote sensing image is obtained and output.

[0084] After obtaining the first visual feature information corresponding to the sample remote sensing image output by the visual language semantic processing module in the target localization model in the above training, and the semantic feature information corresponding to the sample description text output by the tokenizer in the large language model in the target localization model in the above training, the first visual feature information corresponding to the sample remote sensing image and the semantic feature information corresponding to the sample description text can be feature concatenated to obtain sample concatenated feature information.

[0085] After obtaining the sample concatenated feature information, the sample concatenated feature information can be input into the backbone network in the large language model in the target localization model in the above training.

[0086] The backbone network in the large language model in the target localization model in the above training can obtain and output the hidden state information of each token in the sample concatenated feature information and the hidden state information of each special token in the sample concatenated feature information.

[0087] It should be noted that in a deep learning model, text can be segmented into multiple tokens (lexical units), and a token can be a word, sub-word, or character. The hidden state information of a token is the context-related vector representation generated by the deep learning model for each token when processing the input text, which is used to represent the semantic, syntactic, and relational information of the token in a specific context. The hidden state of a token not only contains its own information but also incorporates the context of its surrounding tokens. Moreover, as the number of model layers deepens, the hidden state information of the token will gradually become more abstract. Shallow layers may capture local syntax (such as part of speech), while deep layers may capture global semantics (such as sentiment or intention).

[0088] In the embodiments of the present invention, the hidden state information of each token in the sample concatenated feature information can be used as a feature to be input into the encoder in the large language model for aligning the sample target in the sample description text and the sample remote sensing image.

[0089] In a deep learning model, some special tokens are usually introduced, such as [CLS], [SEP], [PAD], etc. For example, in BERT, [CLS] is used for classification tasks, [SEP] is used for separating sentences, and [PAD] is used for padding. These special tokens also generate corresponding hidden states during model processing. For example, the hidden state of [CLS] is usually used as the aggregated representation of the entire sequence for classification tasks. And [SEP] may be used to separate different sentences or parts, and its hidden state may contain separation information. The hidden state information of special tokens is the vector representation generated by the deep learning model for special tokens. The hidden state information of special tokens can serve specific task or structural requirements.

[0090] In the embodiment of the present invention, the hidden state information of each special token in the sample splicing feature information can be used to represent the token corresponding to the sample bounding box in the sample remote sensing image. Correspondingly, the hidden state information of each special token in the sample splicing feature information can be decoded into the position coordinates of the sample bounding box in the sample remote sensing image.

[0091] After obtaining the hidden state information of each token in the sample splicing feature information output by the backbone network in the large language model of the target localization model in the above training, the hidden state information of each token in the sample splicing feature information can be input into the output layer in the large language model of the target localization model in the above training.

[0092] The output layer in the large language model of the target localization model in the above training can decode the hidden state information of each token in the sample splicing feature information, so as to obtain and output the first predicted position information of the sample target in the sample remote sensing image. Among them, the first predicted position information of the sample target in the sample remote sensing image includes the first predicted position coordinates of the sample bounding box in the sample remote sensing image.

[0093] Input the hidden state information of each special token in the sample splicing feature information into the decoding model, and obtain the second predicted position information of the sample target in the sample remote sensing image output by the decoding model.

[0094] After obtaining the hidden state information of each special token in the sample splicing feature information output by the backbone network in the large language model of the target localization model in the above training, the hidden state information of each special token in the sample splicing feature information can be input into the decoding module.

[0095] The above decoding module can decode the hidden state information of each special token in the sample splicing feature information, decode the hidden state information of each special token in the sample splicing feature information into high-precision values, so as to obtain and output the second predicted position information of the sample target in the sample remote sensing image. Among them, the second predicted position information of the sample target in the sample remote sensing image includes the second predicted position coordinates of the sample bounding box in the sample remote sensing image.

[0096] It should be noted that the structure of the decoding module in the embodiment of the present invention may include but is not limited to a multi-layer perceptron or a multi-layer Transformer model, etc.

[0097] Based on the first predicted position information of the sample target in the sample remote sensing image, the second predicted position information of the sample target in the sample remote sensing image, and the position information of the sample target in the sample remote sensing image, calculate the loss function value of the target localization model in the training.

[0098] Specifically, after obtaining the first predicted position information and the second predicted position information of the sample target in the sample remote sensing image, as well as the position information of the sample target in the sample remote sensing image, the loss function of the target localization model in the above training can be calculated through numerical calculation.

[0099] When it is determined that the loss function value of the target localization model in training converges, it is determined that the target localization model in training converges, and a trained target localization model is obtained. When it is determined that the loss function value of the target localization model in training does not converge, the model parameters of the target localization model in training are updated, and the step of inputting the sample remote sensing image and the sample description text into the target localization model in training is returned until it is determined that the loss function value of the target localization model in training converges.

[0100] It should be noted that during the training process of the target localization model, gradient descent can be performed on the target localization model in training based on the loss function value of the target localization model in training to update the model parameters of the target localization model in training.

[0101] As an optional embodiment, calculating the loss function value of the target localization model in training based on the first predicted position information of the sample target in the sample remote sensing image, the second predicted position information of the sample target in the sample remote sensing image, and the position information of the sample target in the sample remote sensing image includes: calculating the first loss function value of the target localization model in training based on the first predicted position information of the sample target in the sample remote sensing image and the position information of the sample target in the sample remote sensing image, and calculating the second loss function value of the target localization model in training based on the first predicted position information of the sample target in the sample remote sensing image and the position information of the sample target in the sample remote sensing image.

[0102] Based on the first loss function value and the second loss function value of the target localization model in training, the loss function value of the target localization model in training is calculated.

[0103] As an optional embodiment, the first loss function includes a cross-entropy loss function.

[0104] The second loss function includes at least one of a mean absolute error loss function, a mean squared error loss function, a smooth absolute error loss function, a smooth quadratic error loss function, an intersection over union loss function, a generalized intersection over union loss function, a distance intersection over union loss function, and a complete intersection over union loss function.

[0105] It should be noted that after obtaining the trained target localization model, the performance of the model can be evaluated using the accuracy of different IoU (Intersection over Union) thresholds. When the difference IoU between the predicted bounding box and the annotated bounding box is greater than the threshold, it is considered a correct prediction. The evaluation process aims to ensure the accuracy, reliability, and robustness of the model in practical applications.

[0106] Figure 4 It is a schematic structural diagram of the target localization model in the target localization method provided by the present invention. As Figure 4 shown, as an optional embodiment, the target localization model includes: a visual feature extraction module and a large language model.

[0107] Input the target remote sensing image and the target description text into the target localization model to obtain the position information of the target of interest in the target remote sensing image output by the target localization model, including: input the target remote sensing image into the visual feature extraction module to obtain the first visual feature information corresponding to the target remote sensing image output by the visual feature extraction module, input the target description text into the tokenizer in the large language model to obtain the semantic feature information corresponding to the target description text output by the tokenizer, and the first visual feature information and the semantic feature information are in the same dimensional space.

[0108] Perform feature splicing on the first visual feature information and the semantic feature information to obtain the target spliced feature information.

[0109] Input the target spliced feature information into the backbone network in the large language model to obtain the hidden state information of each token in the target spliced feature information output by the backbone network.

[0110] Input the hidden state information of each token in the target spliced feature information into the output layer in the large language model to obtain the position information of the target of interest in the target remote sensing image output by the output layer.

[0111] As an optional embodiment, the visual feature extraction module includes: a semantic encoder and a visual language semantic processing module.

[0112] Input the target remote sensing image into the visual feature extraction module to obtain the first visual feature information corresponding to the target remote sensing image output by the visual feature extraction module, including: input the target remote sensing image into the semantic encoder, and the semantic encoder extracts features from the target remote sensing image, and then obtains the second visual feature information corresponding to the target remote sensing image output by the semantic encoder.

[0113] Input the second visual feature information into the visual language semantic processing module, and the visual language semantic processing module compresses and converts the dimension of the second visual feature information to obtain the first visual feature information output by the visual language semantic processing module.

[0114] Specifically, after obtaining the trained target localization model, the target remote sensing image can be input into the semantic encoder in the above-trained target localization model, and the target description text can be input into the tokenizer in the large language model of the above-trained target localization model.

[0115] The semantic encoder in the above-trained target localization model can extract visual features from the target remote sensing image, and obtain and output the second visual feature information corresponding to the target remote sensing image.

[0116] The tokenizer in the large language model of the above-trained target localization model can extract semantic features from the target description text, and obtain and output the semantic feature information corresponding to the target description text.

[0117] After obtaining the second visual feature information corresponding to the target remote sensing image output by the semantic encoder in the above-trained target localization model, the second visual feature information corresponding to the target remote sensing image can be input into the vision-language semantic processing module in the above-trained target localization model.

[0118] The vision-language semantic processing module in the above-trained target localization model can compress and convert the dimension of the second visual feature information corresponding to the target remote sensing image. On the basis of keeping the semantics of the second visual feature information corresponding to the target remote sensing image unchanged, the second visual feature information corresponding to the target remote sensing image is converted to the dimension space of the large language model, and then the first visual feature information corresponding to the target remote sensing image is obtained and output.

[0119] After obtaining the first visual feature information corresponding to the target remote sensing image output by the vision-language semantic processing module in the above-trained target localization model, and the semantic feature information corresponding to the target description text output by the tokenizer in the large language model of the above-trained target localization model, the first visual feature information corresponding to the target remote sensing image and the semantic feature information corresponding to the target description text can be feature-stitched to obtain the target stitched feature information.

[0120] After obtaining the target stitched feature information, the target stitched feature information can be input into the backbone network in the large language model of the above-trained target localization model.

[0121] The target localization method provided by the present invention combines high-resolution target remote sensing images and target description texts for describing the target of interest, and accurately locates the target of interest in the target remote sensing image through a multi-modal target localization model, supporting flexible target query and localization. A large-scale remote sensing text-image dialogue dataset is used for model training. The dataset contains high-resolution images and their corresponding target coordinates and Q&A pairs, enhancing the model's understanding of target features and localization accuracy.

[0122] The target localization method provided by the present invention introduces a fine-grained semantic encoder and decoder module. By combining fine-grained visual features with a large language model, it improves the target localization accuracy and decoding ability. It also provides a refined localization enhancement method, optimizing the processing strategies of the image encoding end and the language decoding end, and improving the target localization precision.

[0123] The target localization method provided by the present invention uses a data augmentation method that keeps the description text unchanged to expand the training dataset, improving the robustness and generalization ability of the trained target localization model in small-sample scenarios. It proposes a combination strategy of multiple loss functions, including cross-entropy loss and regression loss, further enhancing the localization accuracy and stability of the trained target localization model.

[0124] Compared with the traditional target localization methods in related technologies, the target localization method provided by the present invention has the following technical advantages: First, enhanced task flexibility: Traditional discriminative specialized models usually can only complete the localization task of a single target and lack the ability to handle the localization tasks of multiple different types of targets simultaneously. In the multi-modal target localization model provided by the present invention, through supporting natural language input, it can more flexibly handle the localization tasks of multiple different types of targets, avoiding the cumbersome multi-model switching and manual screening processes, and improving the flexibility of target localization task processing.

[0125] Second, improved precise localization ability: Traditional generative multi-modal large models have accuracy problems in the alignment of images and texts and are difficult to achieve precise localization of target positions. However, in this application, by improving the position prediction strategy and combining the extraction of fine-grained semantic information, the target localization is more precise, and it can effectively measure the distance between the predicted bounding box and the true bounding box, enhancing the precision and reliability of target localization.

[0126] Third, optimized computational efficiency: Traditional generative multi-modal large models usually have a large number of parameters and consume a high amount of computing resources, making it difficult to be efficiently deployed in resource-constrained environments. The target localization model in this application uses a pre-trained semantic encoder and a pre-trained large language model, optimizing the computational efficiency while reducing the computational cost, ensuring that it can still operate efficiently under various complex tasks and is suitable for resource-limited application scenarios.

[0127] Fourth, multi-task processing ability: Compared with the limitation that traditional target localization models need to be retrained or fine-tuned when dealing with new tasks, the target localization model in this application can seamlessly switch between multiple tasks, flexibly meet the requirements of different target categories or task types, and significantly improve the efficiency and adaptability of remote sensing image analysis.

[0128] Figure 5It is a schematic structural diagram of the target positioning device provided by the present invention. The following will be combined with Figure 5 to describe the target positioning device provided by the present invention. The target positioning device described below can be correspondingly referred to the target positioning method provided by the present invention described above. As Figure 5 shown, there are a data acquisition module 501 and a target positioning module 502.

[0129] The data acquisition module 501 is used to acquire the target remote sensing image and the target description text for describing the target of interest.

[0130] The target positioning module 502 is used to input the target remote sensing image and the target description text into the target positioning model, and acquire the position information of the target of interest in the target remote sensing image output by the target positioning model.

[0131] Among them, the target positioning model is obtained after being trained based on the sample remote sensing image, the sample description text for describing the sample target in the sample remote sensing image, and the position information of the sample target in the sample remote sensing image.

[0132] Specifically, the data acquisition module 501 and the target positioning module 502 are electrically connected.

[0133] In the target positioning device in the embodiment of the present invention, after inputting the target remote sensing image and the target description text for describing the target of interest into the target positioning model, the position information of the target of interest in the target remote sensing image output by the target positioning model is acquired. The above target positioning model is obtained after being trained based on the sample remote sensing image, the sample description text for describing the sample target in the sample remote sensing image, and the position information of the sample target in the sample remote sensing image. It can more accurately and efficiently locate the target of interest in the remote sensing image according to the description of the target of interest in the text, can realize the synchronous positioning of multiple targets of interest in the remote sensing image, has higher flexibility in target positioning of the remote sensing image, can better meet the actual needs of key fields such as maritime traffic management, environmental monitoring, and disaster emergency response, and has broad application prospects.

[0134] Figure 6 Illustrates a schematic physical structure diagram of an electronic device, as Figure 6As shown in the figure, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640. Among them, the processor 610, the communications interface 620, and the memory 630 complete mutual communication through the communication bus 640. The processor 610 may call the logical instructions in the memory 630 to execute a target positioning method, which includes: obtaining a target remote sensing image and a target description text for describing the target of interest; inputting the target remote sensing image and the target description text into a target positioning model, and obtaining the position information of the target of interest in the target remote sensing image output by the target positioning model; wherein, the target positioning model is obtained by training based on a sample remote sensing image, a sample description text for describing the sample target in the sample remote sensing image, and the position information of the sample target in the sample remote sensing image.

[0135] In addition, when the logical instructions in the above-mentioned memory 630 are implemented in the form of software functional units and sold or used as an independent product, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0136] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target positioning method provided by the above-mentioned various methods. The method includes: obtaining a target remote sensing image and a target description text for describing the target of interest; inputting the target remote sensing image and the target description text into a target positioning model, and obtaining the position information of the target of interest in the target remote sensing image output by the target positioning model; wherein, the target positioning model is obtained by training based on a sample remote sensing image, a sample description text for describing the sample target in the sample remote sensing image, and the position information of the sample target in the sample remote sensing image.

[0137] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the target positioning method provided by the above-mentioned various methods. The method includes: obtaining a target remote sensing image and a target description text for describing the target of interest; inputting the target remote sensing image and the target description text into a target positioning model, and obtaining the position information of the target of interest in the target remote sensing image output by the target positioning model; wherein, the target positioning model is obtained after being trained based on a sample remote sensing image, a sample description text for describing the sample target in the sample remote sensing image, and the position information of the sample target in the sample remote sensing image.

[0138] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0139] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A target positioning method, characterized in that: include: Obtaining target remote sensing images and target description text for describing the target of interest; Inputting the target remote sensing image and the target description text into a target positioning model, and obtaining the position information of the target of interest in the target remote sensing image output by the target positioning model; The target positioning model is obtained after training based on sample remote sensing images, sample description texts used to describe sample targets in the sample remote sensing images, and position information of the sample targets in the sample remote sensing images.

2. The target positioning method according to claim 1, characterized in that: The target positioning model includes: a visual feature extraction module and a large language model; The step of inputting the target remote sensing image and the target description text into a target positioning model to obtain the position information of the target of interest in the target remote sensing image output by the target positioning model includes: Inputting the target remote sensing image into the visual feature extraction module, obtaining first visual feature information corresponding to the target remote sensing image output by the visual feature extraction module, inputting the target description text into the word segmenter in the large language model, obtaining semantic feature information corresponding to the target description text output by the word segmenter, wherein the first visual feature information and the semantic feature information are in the same dimensional space; Performing feature splicing on the first visual feature information and the semantic feature information to obtain target splicing feature information; Inputting the target concatenation feature information into the backbone network in the large language model, and obtaining hidden state information of each word unit token in the target concatenation feature information output by the backbone network; The hidden state information of each token in the target splicing feature information is input into the output layer in the large language model, and the position information of the target of interest in the target remote sensing image output by the output layer is obtained.

3. The target positioning method according to claim 2, characterized in that: The visual feature extraction module includes: a semantic encoder and a visual language semantic processing module; The step of inputting the target remote sensing image into the visual feature extraction module and acquiring first visual feature information corresponding to the target remote sensing image output by the visual feature extraction module comprises: Inputting the target remote sensing image into the semantic encoder, and having the semantic encoder perform feature extraction on the target remote sensing image, thereby obtaining second visual feature information corresponding to the target remote sensing image output by the semantic encoder; The second visual feature information is input into the visual language semantic processing module, and the visual language semantic processing module compresses and dimensionally converts the second visual feature information to obtain the first visual feature information output by the visual language semantic processing module.

4. The target positioning method according to claim 3, characterized in that: The target positioning model is trained based on the following steps: Acquire the sample remote sensing image, the sample description text, and the location information of the sample target in the sample remote sensing image; Constructing an initial model, wherein the initial model includes a pre-trained semantic encoder, a visual language semantic processing module, and a pre-trained large language model; Determine the initial model as the target positioning model in training, input the sample remote sensing image and the sample description text into the target positioning model in training, obtain the first predicted position information of the sample target in the sample remote sensing image output by the target positioning model in training, and obtain the hidden state information of each special token in the sample splicing feature information output by the backbone network of the large language model in the target positioning model in training; Inputting the hidden state information of each special token in the sample splicing feature information into a decoding model, and obtaining the second predicted position information of the sample target in the sample remote sensing image output by the decoding model; Based on the first predicted position information of the sample target in the sample remote sensing image, the second predicted position information of the sample target in the sample remote sensing image, and the position information of the sample target in the sample remote sensing image, a loss function value of the target positioning model in training is calculated; When it is determined that the loss function value of the target positioning model in training converges, the target positioning model in training converges to obtain a trained target positioning model; when it is determined that the loss function value of the target positioning model in training does not converge, the model parameters of the target positioning model in training are updated, and the step of inputting the sample remote sensing image and the sample description text into the target positioning model in training is returned to execute until it is determined that the loss function value of the target positioning model in training converges.

5. The target positioning method according to claim 4, characterized in that: The step of calculating the loss function value of the target positioning model in training based on the first predicted position information of the sample target in the sample remote sensing image, the second predicted position information of the sample target in the sample remote sensing image, and the position information of the sample target in the sample remote sensing image comprises: Based on the first predicted position information of the sample target in the sample remote sensing image and the position information of the sample target in the sample remote sensing image, a first loss function value of the target positioning model in training is calculated; based on the first predicted position information of the sample target in the sample remote sensing image and the position information of the sample target in the sample remote sensing image, a second loss function value of the target positioning model in training is calculated; Based on the first loss function value and the second loss function value of the target positioning model in training, the loss function value of the target positioning model in training is calculated.

6. The target positioning method according to claim 5, characterized in that: The first loss function includes a cross entropy loss function; The second loss function includes at least one of a mean absolute error loss function, a mean squared error loss function, a smoothed absolute error loss function, a smoothed quadratic error loss function, an intersection-over-union loss function, a generalized intersection-over-union loss function, a distance intersection-over-union loss function, and a complete intersection-over-union loss function.

7. A target positioning device, characterized in that: include: A data acquisition module is used to acquire a target remote sensing image and a target description text for describing the target of interest; A target positioning module, used for inputting the target remote sensing image and the target description text into a target positioning model, and obtaining the position information of the target of interest in the target remote sensing image output by the target positioning model; The target positioning model is obtained after training based on sample remote sensing images, sample description texts used to describe sample targets in the sample remote sensing images, and position information of the sample targets in the sample remote sensing images.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the target positioning method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the target positioning method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the target positioning method according to any one of claims 1 to 6 is implemented.

Images