Building vector polygon contour extraction method based on multi-modal large model

Abstract

The invention discloses a building vector polygon contour extraction method and device based on a multi-modal large model, a storage medium and electronic equipment. The method comprises the steps of obtaining a remote sensing image and a text instruction of a building; inputting the remote sensing image and the text instruction into a trained multi-modal large model to obtain a building polygon angular point coordinate sequence, and connecting angular points to form a building vector polygon contour; wherein the multi-modal large model is trained through a prediction-cutting cooperative training strategy. According to the invention, the method can meet the requirements of efficiency and robustness in a complex scene, and improves the capability of capturing the features of small-scale buildings.

Description

Technical Field

[0001] The present application relates to the technical field of image contour extraction, and in particular to a method, device, storage medium and electronic device for extracting vector polygon contours of buildings based on a multimodal large model. Background Art

[0002] Building extraction is a core task in remote sensing image interpretation and geographic information modeling, and plays a key role in urban planning, land resource management, and smart city construction. The current mainstream building extraction methods mainly follow three technical paths: mask-based segmentation network, contour-based detection model, and geometric modeling method based on primitives.

[0003] However, traditional methods face significant challenges when processing remote sensing images of high-resolution complex scenes: although mask-based methods can achieve pixel-level building area recognition, the generated binary masks need to rely on complex processes such as edge extraction and morphological post-processing to be converted into vector contours, which can easily produce boundary jagged edges and broken polygons; although contour-based methods can directly predict boundary points, they do not adequately model the adjacency relationships of dense building complexes, which can easily lead to contour intersections or topological errors; although primitive-based methods can generate vector vertices, they are less robust to small buildings and occluded scenes, and the vertex positioning accuracy is easily affected by noise.

[0004] On the other hand, with the breakthroughs of multimodal large models (MLLMs) in cross-modal understanding and generation tasks, their ability to combine visual-language representation provides new ideas for building extraction. However, existing research has mostly focused on the field of natural images, and there are significant limitations when directly applying MLLMs to remote sensing images of multiple buildings: the scales of buildings in remote sensing images vary greatly, and densely arranged small buildings are easily submerged by large-scale targets in global features, resulting in feature sparsity. Although some recent studies have attempted to improve the accuracy of small target detection through ROI cropping, the scale adaptation problem between the cropped area and the model input has not yet been solved, and direct scaling will lead to loss of details or computational redundancy. Therefore, studying and implementing end-to-end high-precision vector polygon generation while meeting the efficiency and robustness requirements in complex scenarios is still a technical difficulty that needs to be overcome. Summary of the invention

[0005] The embodiments of the present application provide a method, device, storage medium and electronic device for extracting vector polygon outlines of buildings based on a multimodal large model, which can meet the efficiency and robustness requirements in complex scenarios and improve the ability to capture features of small-scale buildings.

[0006] The embodiment of the present application provides a method for extracting building vector polygon outlines based on a multimodal large model, comprising: Obtain remote sensing images and text instructions of buildings; Input the remote sensing image and the text instruction into the trained multimodal large model to obtain a coordinate sequence of building polygon corner points, and connect the corner points to form a vector polygon outline of the building; The multimodal large model is trained using a prediction-clipping collaborative training strategy.

[0007] Furthermore, the above-mentioned method for extracting building vector polygon outlines based on a multimodal large model, wherein the multimodal large model includes a visual encoder, a text instruction encoder, a multimodal feature alignment module and a large language model; The step of inputting the remote sensing image and the text instruction into a trained multi-modal large model to obtain a coordinate sequence of building polygon corner points includes: Inputting the remote sensing image into the visual encoder to extract multi-scale features to obtain visual features; Inputting the text instruction into the text instruction encoder for deep semantic encoding to obtain text features; Inputting the visual features into the multimodal feature alignment module to map the visual features to a language space consistent with the large language model; The mapped visual features and the text features are input into the large language model to obtain a coordinate sequence of building polygon corner points.

[0008] Furthermore, the above-mentioned method for extracting building vector polygon outlines based on a multimodal large model, after the step of inputting the remote sensing image into the visual encoder for multi-scale feature extraction to obtain visual features, comprises: generating location embedding information based on the remote sensing image; The position embedding information is concatenated with the visual feature to obtain an enhanced visual feature.

[0009] Further, in the above-mentioned method for extracting building vector polygon outlines based on a multimodal large model, the text instruction encoder includes a subword segmenter and a plurality of self-attention layers; The step of inputting the text instruction into the text instruction encoder for deep semantic encoding to obtain text features includes: Inputting the text instruction into the subword segmenter to obtain a token sequence; The word sequence is sequentially input into multiple self-attention layers and then transformed by self-attention to obtain text features.

[0010] Furthermore, in the above-mentioned method for extracting building vector polygon outlines based on a multimodal large model, the step of mapping the visual features to a language space consistent with the large language model comprises: Mapping the visual features to a language space consistent with the large language model through a projector; wherein the projector is a two-layer multi-layer perceptron; The mapping process of the projector is expressed by the following formula:

[0011] in, , , , and are all projector parameters, is the activation function.

[0012] Furthermore, in the above-mentioned method for extracting building vector polygon outlines based on a multimodal large model, wherein the large language model includes a word segmenter, the method further includes: The word segmenter is expanded to add pixel feature tags and position feature tags, wherein the pixel feature tags are used to indicate the pixel features of the visual features, and the position feature tags are used to mark the position features of the remote sensing image.

[0013] Furthermore, in the above-mentioned method for extracting building vector polygon outlines based on a multimodal large model, the process of training the multimodal large model through the prediction-clipping collaborative training strategy includes: Label the center point and bounding box range of each building in the remote sensing image as the first training sample set; Inputting the images in the first training sample set into the multimodal large model, the first step is to predict the coordinates of the center point of each building on the remote sensing image, and the second step is to predict the range of the bounding box of each building based on the coordinates of the center point of the building; Based on the true value of the bounding box range of each building, the remote sensing image is cropped to obtain a local area containing only a single building, and a second training sample set is formed based on the local area containing the single building and the true value contour; Based on the true value contour coordinates corresponding to the second training sample set, the visual encoder parameters and text encoder parameters of the multimodal large model are frozen, and the remaining modules of the multimodal large model are trained until convergence.

[0014] The embodiment of the present application also provides a device for extracting building vector polygon outlines based on a multimodal large model, comprising: An acquisition module is used to obtain remote sensing images and text instructions of buildings; A vector polygon outline extraction module is used to input the remote sensing image and the text instruction into the trained multi-modal large model to obtain a coordinate sequence of building polygon corner points, and connect the corner points to form a vector polygon outline of the building; The multimodal large model is trained using a prediction-clipping collaborative training strategy.

[0015] An embodiment of the present application also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are suitable for being loaded by a processor to execute any of the above-mentioned methods for extracting vector polygon outlines of buildings based on a multimodal large model.

[0016] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used for the steps in any of the above-mentioned methods for extracting vector polygon contours of buildings based on a multimodal large model.

[0017] The present application provides a method, device, storage medium and electronic device for extracting vector polygon outlines of buildings based on a multimodal large model. The present application extracts vector polygon outlines of buildings through a multimodal large model, and trains the multimodal large model using a prediction-clipping collaborative training strategy. The following are achieved: (1) No complex post-processing is required. Through a simple, end-to-end trainable model, the morphological post-processing steps required for mask vectorization in traditional methods are avoided to complete the generation of the coordinate sequence of the building polygon corner points, significantly improving efficiency. (2) Strong semantic understanding ability, multimodal instruction drive, combined with visual encoder and text instruction encoder, cross-modal fusion is achieved through MLP feature alignment, so that the model can accurately understand the semantic instruction of "sequentially click on corner points" and imitate human drawing logic. (3) It is scalable. The trained neural network model can be adjusted and applied to other purposes, such as ground object change detection based on remote sensing images, and ground object target extraction of interest based on remote sensing images. (4) Strong robustness. It adopts an adaptive prediction-cropping strategy and a dynamic scaling and cropping mechanism to enhance the model's ability to capture the features of small-scale buildings. It can obtain good extraction results for various ground features in remote sensing images of dense building complexes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The technical solution and other beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0019] Figure 1 A flowchart of a method for extracting building vector polygon outlines based on a multimodal large model provided in an embodiment of the present application.

[0020] Figure 2 Another flowchart of the method for extracting building vector polygon outlines based on a multimodal large model provided in an embodiment of the present application.

[0021] Figure 3The prediction results of the method of the present invention and the currently best P2PFormer method on the WHU dataset are provided in the embodiments of the present application.

[0022] Figure 4 The method of the present invention provided in the embodiment of the present application predicts the building outline extraction method on the WHU and WHU-Mix data sets.

[0023] Figure 5 A schematic diagram of the structure of a building vector polygon outline extraction device based on a multimodal large model provided in an embodiment of the present application.

[0024] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0026] The embodiment of the present application provides a method, device, storage medium and electronic device for extracting vector polygon outlines of buildings based on a multimodal large model. The embodiment of the present application provides a device for extracting vector polygon outlines of buildings based on a multimodal large model, which can be integrated in an electronic device, which can be a terminal, a server and other devices, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC, Personal Computer), a micro processing box, or other devices.

[0027] See also Figure 1 and Figure 2 , Figure 1 A flowchart of a method for extracting building vector polygon outlines based on a multimodal large model provided in an embodiment of the present application, Figure 2 Another flow chart of a method for extracting building vector polygon outlines based on a multimodal large model provided in an embodiment of the present application, which is applied to an electronic device, and the method for extracting building vector polygon outlines based on a multimodal large model comprises the following steps: S1, obtain remote sensing images and text instructions of buildings.

[0028] S2, inputting the remote sensing image and the text instruction into the trained multimodal large model to obtain a coordinate sequence of building polygon corner points, and connecting the corner points to form a vector polygon outline of the building; The multimodal large model is trained using a prediction-clipping collaborative training strategy.

[0029] In one embodiment, the multimodal large model includes a visual encoder, a textual instruction encoder, a multimodal feature alignment module, and a large language model.

[0030] Step S2 includes the following steps: S21, inputting the remote sensing image into the visual encoder for multi-scale feature extraction to obtain visual features.

[0031] After step S21, the method further includes: Generate location embedding information based on remote sensing images; The position embedding information is concatenated with the visual feature to obtain an enhanced visual feature.

[0032] Specifically, the visual encoder uses the RADIO visual encoder based on the Transformer architecture to process the input remote sensing image and capture the spatial information and feature representation in the image. Its core is the multi-head self-attention mechanism, which can automatically capture the spatial relationship and high-level semantic features in the remote sensing image. In this process, first, the remote sensing image It is adjusted to a fixed size of 224×224 as the input of the visual encoder. After passing through the visual encoder, the visual features are obtained. , It is a high-level semantic feature with a downsampling factor of 16 (the input remote sensing image is divided into non-overlapping blocks of 16×16 pixels, and multi-scale features are extracted through a multi-layer self-attention mechanism, and the output is a high-level semantic feature map with a resolution of 1 / 16 downsampling). Subsequently, the learnable positional embeddings With visual features Splice to get enhanced visual feature representation This feature combines spatial information with visual features and has the ability to analyze and locate the target area.

[0033] S22, input the text instruction into the text instruction encoder for deep semantic encoding to obtain text features.

[0034] Step S22 includes the following steps: S221, input the text instruction into the subword segmenter to obtain a word token sequence; S222, input the word sequence into multiple self-attention layers in sequence to obtain text features after self-attention transformation.

[0035] The text instruction encoder encodes the input text instructions into high-level features. The encoder used in this module is Qwen2.5 tokenizer --- byte-level byte-pair encoding BBPE, which is built on a pre-trained language model and uses a multi-layer Transformer architecture to perform deep semantic encoding of task instructions.

[0036] Enter the given text command X q (such as "extracting the vertex coordinates of building polygons"), generate a token sequence through the subword segmentation Tokenizer (·), and then L After the layer self-attention transformation, the output text tokens ( ),in d is the hidden layer dimension, l is the sequence length.

[0037] S23, inputs the visual features into the multimodal feature alignment module, and maps the visual features to a language space consistent with the large language model.

[0038] Specifically, the multimodal feature alignment module plays the role of connecting the visual encoder and the language model in the model. It is responsible for mapping the visual features to a latent space consistent with the language model to ensure the effective alignment of multimodal information.

[0039] The visual features are mapped to a language space consistent with the large language model through a projector, so that the large language model can understand the image information; wherein the projector is a two-layer multi-layer perceptron (MLP).

[0040] The mapping process of the projector is expressed by the following formula:

[0041] in, , , , and are all projector parameters, is the activation function.

[0042] S24, inputting the mapped visual features and text features into the large language model to obtain a coordinate sequence of the building polygon corner points.

[0043] The large language model is pre-trained based on massive text data, and it has powerful language understanding, logical reasoning, and instruction-following capabilities. In multimodal tasks, the large language model can accurately parse the combined input of images and text through deep fusion with visual features. This model uses Qwen2.5-1.5B as the pre-trained large language model.

[0044] Furthermore, the large language model includes a word segmenter, which is extended to add pixel feature tags and position feature tags. The pixel feature tags are used to indicate pixel features of visual features, and the position feature tags are used to mark position features of remote sensing images.

[0045] Specifically, in order for the large language model to effectively understand and process the multimodal input from the visual encoder, its tokenizer needs to be extended by adding a series of specially designed special tags, mainly including: Encode structured text instructions such as "extract building polygon vertices in clockwise order"; add the following special tags by extending the vocabulary: <pixel>: Pixel feature markers used to indicate image pixel features from the visual encoder, helping the model recognize and process visual information.

[0046] <x>and <y>: Position feature editing, which represents the coordinate axis in the image and is used to mark the features at different positions in the image. <x0>arrive<x(W-1)> Represents the horizontal axis, from <y0>arrive<y(H-1)> Represents the ordinate, with a total of H+W tags. The embedding layer of the language model is adjusted to adapt to the newly added tags. After the introduction of the above special tags, the vocabulary of the model is expanded to ensure the alignment and fusion of multimodal features. In the building polygon extraction task in this invention, a pair of <x> <y>The tags are used to record the center point coordinates and corner point coordinates of the building. By predicting these coordinates in sequence, the accurate building polygon is obtained. The large language model performs reasoning based on the input visual features and text features, generates output and decodes it into the corresponding answer, including the center point coordinates, bounding box range, and building corner point coordinates.

[0047] Furthermore, the prediction-cropping strategy is to solve the problem of insufficient feature expression of small buildings. This strategy is completed under the same framework as the multimodal large model. We designed a three-step strategy. For a single building in an image, the multimodal large model is used to predict the center of each building. The dialogue is as follows: "Please extract the coordinates of the center point of each building in the image." " <x31> <y165> <x58> <y67> <x89> <y112> <x24> <y113> <x27> <y67> <x53> <y47> <x29> <y112> <x35> <y161>". Then, based on the predicted center point, the bounding box of each building is further predicted, its width and height are determined, and the scope of a single building is accurately located. The dialogue is: "Based on the center point of the building, give the x-axis length range and y-axis length range of the area where the building is located: <x31> <y56>(The two length ranges together form the building bounding box with the center point as the origin)" "x:24 y:13". Then, according to the predicted building bounding box, the original image is cropped to extract the local area containing a single building. In order to enhance the model's perception of small buildings, the cropped area is not fixed to the bounding box range, but is randomly enlarged by 1.5-2 times to ensure that the model can fully capture the target features. Then, the cropped image containing only a single building is adjusted to a fixed size of 224×224 and sent to the multimodal model for refined prediction. The coordinate sequence of the building polygon corner points is parsed and output from the remote sensing image, and the corner points are connected to form the vector polygon outline of the building. Finally, based on the bounding box information of the building in the complex image and the coordinates of the corner points in the cropped image, the spatial mapping of the building corner points from the local image to the original image is completed to ensure the accurate positioning of the prediction results in the original image.

[0048] The prediction-cropping collaborative training strategy achieves coarse-to-fine building extraction through three-step cascade of building center prediction, bounding box prediction, and single building outline accurate prediction. After training, the trained network model weights are loaded to predict new remote sensing images, which can achieve accurate vector polygon outline extraction of complex remote sensing images with multiple buildings.

[0049] The process of training a large multimodal model through the prediction-clipping co-training strategy includes: A1, annotates the center point and bounding box range of each building in the remote sensing image as the first training sample set.

[0050] Specifically, the original large-scale remote sensing images are cropped to a standard size supported by the network (224×224 pixels), the center point of each building is annotated, and a sample set for training building center point prediction is generated. Then, based on the true value of the center point, the local area containing a single building is cropped and scaled (randomly enlarged by 1.5-2 times) to generate refined polygon corner point coordinates and vector annotated true values ​​to form a bounding box prediction sample set.

[0051] A2, input the images in the first training sample set into the multimodal large model, the first step is to predict the coordinates of the center point of each building on the remote sensing image, and the second step is to predict the range of the bounding box of each building based on the coordinates of the center point of the building.

[0052] The first training sample set in step A1 (including center point prediction samples and bounding box prediction samples) is sequentially input into the multimodal large model, and the model is trained to output the center point coordinates and bounding box range of each building.

[0053] A3, based on the true value of the bounding box range of each building, the remote sensing image is cropped to obtain a local area containing only a single building, and the second training sample set is formed based on the local area containing the single building and the true value contour.

[0054] A4, based on the true value contour coordinates corresponding to the second training sample set, freeze the visual encoder parameters and text encoder parameters of the multimodal large model, and train the remaining modules of the multimodal large model until convergence.

[0055] Based on the samples generated in step A1, freeze the parameters of the visual encoder (RADIO) and the text encoder (Qwen2.5-1.5B), and only train the feature projector and center point prediction modules. Use the AdamW optimizer (learning rate 5e-5), and the loss function is the heat map mean square error (MSE). Fine-tune all linear layers in the language model and visual encoder using the LoRA technology to train them to output the coordinates of the building polygon corners until the overall model converges.

[0056] A specific embodiment is provided below: First, a building vector polygon outline extraction framework based on a multimodal large model is constructed according to the method of the present invention. Then, training sample data is obtained, and the network model is trained using the sample data. The sample data used in the embodiment is the WHU and WHU-Mix building extraction datasets. The WHU dataset contains 2793 training images, 627 verification images, and 2220 test images, with an image resolution of 1024×1024 pixels; the WHU-Mix dataset contains a large-scale aerial and satellite image dataset with COCO format building boundary annotations. The data comes from multiple regions around the world, including 43778 training images and 11675 test images (Test1) and 6011 cross-domain test images (Test2).

[0057] All input images are uniformly adjusted to 224×224 pixels to ensure data consistency and adapt to the input requirements of the visual encoder, and then input into the multimodal large model for iterative training until the model converges to obtain the optimal weight file. During the training process, the loss output by the language model is used as the loss function to ensure the target consistency of the model in multimodal learning. After the model training is completed, the test remote sensing image to be predicted is input into the multimodal large model with the training weights loaded to verify the model's performance in the building extraction task.

[0058] In order to verify the effectiveness and advancement of the method of the present invention, we compared the proposed method with the current mainstream building extraction algorithms. Including Mask-RCNN, QueryInst, YOLACT, SOLO, E2EC, P2Pformer and Line2Poly algorithms that have outstanding performance in building extraction tasks. All methods are trained on the same hardware environment (four NVIDIAV100 GPUs, 16GB of video memory per card, distributed training) using the same training dataset and evaluated on the WHU test set and WHU-Mix. The prediction results of all methods are based on the mean average precision (mAP), AP at IoU=0.5 (AP 50 ) and AP (AP) when IoU=0.75 75 ) are used as the main evaluation indicators for quantitative evaluation and are recorded in Table 1 (WHU dataset) and Table 2 (WHU-Mix dataset).

[0059] Table 1 Comparison of the accuracy of the proposed method and other advanced building extraction methods in the WHU dataset

[0060] Among them, "VECTOR" indicates whether the network can output vector building outlines; "PF" indicates whether post-processing is required.

[0061] Table 2 Comparison of the accuracy of the proposed method and other advanced building extraction methods in the WHU-Mix dataset

[0062] Among them, "VECTOR" indicates whether the network can output vector building outlines; "PF" indicates whether post-processing is required.

[0063] From Table 1, we can see that on the WHU test set, the mAP of the proposed method reaches 74.8%, which is 2.1% higher than the second best method P2PFormer (72.7%). 75 The indicator is improved by 1.7%; on the WHU-Mix test set, the overall prediction accuracy of the proposed method is improved by 3 AP on test1 and 0.7 AP on test2 compared with the suboptimal method P2Pformer. In addition, the AP under the high threshold matching standard is 75 In terms of indicators, our method improved by 12 points in test1 and 13.2 points in test2. Compared with these existing methods, it is proved that the method of the present invention has better robustness and can obtain more accurate building outline vector prediction results. Therefore, the method of the present invention has good engineering practical value.

[0064] Figure 3 The prediction results of the method of the present invention and the best P2PFormer method on the WHU dataset provided in the embodiment of this application. Figure 3 In , the first line is the building outline drawn by our method, and the second line is the result drawn by the P2PFormer method. Figure 3 It can be seen that the prediction results of the method of the present invention are more accurate, while the P2PFormer method is more prone to topological errors and missed building detection. Our method is more accurate in predicting the outline of buildings, has accurate topological relationships, fewer missed detections, and has more correct prediction results overall. This proves the innovation and effectiveness of the present invention and has good engineering practical value. Figure 4 The prediction results of the building outline extraction method provided by the present invention in the embodiment of the present application on the WHU and WHU-Mix data sets. Figure 4 It can be seen that our method has accurate prediction, accurate topological relationship, and few missed detections.

[0065] According to the method described in the above embodiment, this embodiment will be further described from the perspective of a building vector polygon outline extraction device based on a multimodal large model. The building vector polygon outline extraction device based on a multimodal large model can be implemented as an independent entity or integrated in an electronic device. The electronic device can be a terminal, a server, or other equipment. The terminal can include a tablet computer, a laptop computer, a personal computer (PC, Personal Computer), a micro processing box, or other equipment.

[0066] See also Figure 5 , Figure 5 The invention specifically describes a device for extracting a building vector polygon outline based on a multimodal large model provided in an embodiment of the present application, which is applied to an electronic device. The device for extracting a building vector polygon outline based on a multimodal large model may include: An acquisition module is used to obtain remote sensing images and text instructions of buildings; A vector polygon outline extraction module is used to input the remote sensing image and the text instruction into the trained multi-modal large model to obtain a coordinate sequence of building polygon corner points, and connect the corner points to form a vector polygon outline of the building; The multimodal large model is trained using a prediction-clipping collaborative training strategy.

[0067] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0068] In addition, the embodiment of the present application further provides an electronic device, which may be a computer, a tablet computer, or other device. The electronic device may implement the steps in any embodiment of the method for extracting building vector polygon outlines based on a multimodal large model provided in the embodiment of the present application, and therefore, may achieve the beneficial effects that can be achieved by any method for extracting building vector polygon outlines based on a multimodal large model provided in the embodiment of the present application, as detailed in the previous embodiment, which will not be repeated here.

[0069] Figure 6 The specific structural block diagram of the electronic device provided in the embodiment of the present invention is shown, and the electronic device can be used to implement the building vector polygon outline extraction method based on the multimodal large model provided in the above embodiment. The electronic device 500 can be a terminal, a server and other devices, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC, Personal Computer), a micro processing box, or other devices.

[0070] The RF circuit 510 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 510 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks may use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0071] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, that is, realizing the functions of taking pictures with the front camera, processing the captured images, and switching the display color of the display content on the display screen. The memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 520 may further include a memory remotely arranged relative to the processor 580, and these remote memories may be connected to the electronic device 500 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0072] The input unit 530 may be used to receive input digital or character information, and generate a keyboard and a mouse related to user settings and function control. The display unit 540 may be used to display information input by the user or information provided to the user and various graphical user interfaces, which may be composed of graphics, text, icons, videos, and any combination thereof. The display unit 540 may include a display panel 541, which may be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like.

[0073] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the electronic device 500. The audio circuit 560 can transmit the electrical signal converted from the received audio data to the speaker 561, which is converted into a sound signal for output; on the other hand, the microphone 562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 560 and converted into audio data, and then the audio data is output to the processor 580 for processing, and then sent to another terminal through the RF circuit 510, or the audio data is output to the memory 520 for further processing. The audio circuit 560 may also include an earplug jack to provide communication between an external headset and the electronic device 500.

[0074] The electronic device 500 can help the user receive requests, send information, etc. through the transmission module 570 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 570 is shown in the figure, it can be understood that it is not a necessary component of the electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0075] The processor 580 is the control center of the electronic device 500. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 520, and calling data stored in the memory 520, it executes various functions of the electronic device 500 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 580.

[0076] The electronic device 500 also includes a power supply 590 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 580 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 590 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0077] Although not shown, the electronic device 500 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors. One or more programs include instructions for performing the following operations: Obtain remote sensing images and text instructions of buildings; Input the remote sensing image and the text instruction into the trained multimodal large model to obtain a coordinate sequence of building polygon corner points, and connect the corner points to form a vector polygon outline of the building; The multimodal large model is trained using a prediction-clipping collaborative training strategy.

[0078] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0079] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores a plurality of instructions, and the instructions can be loaded by a processor to execute the steps of any embodiment of the method for extracting building vector polygon outlines based on a multi-modal large model provided in an embodiment of the present invention.

[0080] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0081] Since the instructions stored in the storage medium can execute the steps in any embodiment of the method for extracting the vector polygon outline of a building based on a multimodal large model provided in the embodiments of the present invention, the beneficial effects that can be achieved by any method for extracting the vector polygon outline of a building based on a multimodal large model provided in the embodiments of the present invention can be achieved. For details, please refer to the previous embodiments and will not be repeated here.

[0082] The above is a detailed introduction to a method, device, storage medium and electronic device for extracting vector polygon contours of buildings based on a multimodal large model provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and scope of application. In summary, the content of this specification should not be understood as a limitation on the present application. < / x31> < / x35> < / y112> < / x29> < / y47> < / x53> < / y67> < / x27> < / y113> < / x24> < / y112> < / x89> < / y67> < / x58> < / y165> < / x31> < / y> < / x> < / y> < / x> < / pixel>

Claims

1. A method for extracting building vector polygon outlines based on a multimodal large model, characterized in that: The method comprises: Obtain remote sensing images and text instructions of buildings; Input the remote sensing image and the text instruction into the trained multimodal large model to obtain a coordinate sequence of building polygon corner points, and connect the corner points to form a vector polygon outline of the building; Among them, the multimodal large model is trained through a prediction-cropping collaborative training strategy, and the prediction-cropping collaborative training strategy includes: predicting the training samples of the multimodal large model, obtaining the bounding box range of each building in the training samples, and cropping the training samples based on the bounding box range.

2. The method for extracting building vector polygon outlines based on a multimodal large model according to claim 1, characterized in that: The multimodal large model includes a visual encoder, a text instruction encoder, a multimodal feature alignment module and a large language model; The step of inputting the remote sensing image and the text instruction into a trained multi-modal large model to obtain a coordinate sequence of building polygon corner points includes: Inputting the remote sensing image into the visual encoder to extract multi-scale features to obtain visual features; Inputting the text instruction into the text instruction encoder for deep semantic encoding to obtain text features; Inputting the visual features into the multimodal feature alignment module to map the visual features to a language space consistent with the large language model; The mapped visual features and the text features are input into the large language model to obtain a coordinate sequence of building polygon corner points.

3. The method for extracting building vector polygon outlines based on a multimodal large model according to claim 2, characterized in that: After the step of inputting the remote sensing image into the visual encoder to extract multi-scale features and obtain visual features, the method further comprises: generating location embedding information based on the remote sensing image; The position embedding information is concatenated with the visual feature to obtain an enhanced visual feature.

4. The method for extracting building vector polygon outlines based on a multimodal large model according to claim 2, characterized in that: The text instruction encoder includes a subword segmenter and a plurality of self-attention layers; The step of inputting the text instruction into the text instruction encoder for deep semantic encoding to obtain text features includes: Inputting the text instruction into the subword segmenter to obtain a token sequence; The word sequence is sequentially input into multiple self-attention layers and then transformed by self-attention to obtain text features.

5. The method for extracting building vector polygon outlines based on a multimodal large model according to claim 2, characterized in that: Mapping the visual features to a language space consistent with the large language model includes: Mapping the visual features to a language space consistent with the large language model through a projector; wherein the projector is a two-layer multi-layer perceptron; The mapping process of the projector is expressed by the following formula: in, , , , and are all projector parameters, is the activation function.

6. The method for extracting building vector polygon outlines based on a multimodal large model according to claim 2, characterized in that: The large language model includes a word segmenter, and the method further includes: The word segmenter is expanded to add pixel feature tags and position feature tags, wherein the pixel feature tags are used to indicate the pixel features of the visual features, and the position feature tags are used to mark the position features of the remote sensing image.

7. The method for extracting building vector polygon outlines based on a multimodal large model according to claim 2, characterized in that: The process of training the multimodal large model through the prediction-clipping collaborative training strategy includes: Label the center point and bounding box range of each building in the remote sensing image as the first training sample set; Inputting the images in the first training sample set into the multimodal large model, the first step is to predict the coordinates of the center point of each building on the remote sensing image, and the second step is to predict the range of the bounding box of each building based on the coordinates of the center point of the building; Based on the true value of the bounding box range of each building, the remote sensing image is cropped to obtain a local area containing only a single building, and a second training sample set is formed based on the local area containing the single building and the true value contour; Based on the true value contour coordinates corresponding to the second training sample set, the visual encoder parameters and text encoder parameters of the multimodal large model are frozen, and the remaining modules of the multimodal large model are trained until convergence.

8. A device for extracting building vector polygon outlines based on a multimodal large model, characterized in that: include: An acquisition module is used to obtain remote sensing images and text instructions of buildings; A vector polygon outline extraction module is used to input the remote sensing image and the text instruction into the trained multi-modal large model to obtain a coordinate sequence of building polygon corner points, and connect the corner points to form a vector polygon outline of the building; Among them, the multimodal large model is trained through a prediction-cropping collaborative training strategy, and the prediction-cropping collaborative training strategy includes: predicting the training samples of the multimodal large model, obtaining the bounding box range of each building in the training samples, and cropping the training samples based on the bounding box range.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the method for extracting building vector polygon outlines based on a multimodal large model as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes a processor and a memory, the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute the steps in the method for extracting building vector polygon outlines based on a multimodal large model as described in any one of claims 1 to 7.

Images