Cross-modal building image extraction method and device based on multi-feature synergistic effect, electronic equipment and medium

Abstract

The invention relates to the field of computer vision, in particular to a cross-modal building image extraction method and device based on a multi-feature synergistic effect, electronic equipment and a medium. The method comprises the following steps: constructing a single building instance segmentation network based on a Mask RCNN framework, and adding a direction parameter to a position prediction branch of the network to obtain a building image; then, a receptive field module is embedded into a segmentation branch, Shuffle Net is adopted as a network trunk to extract building features, and ROI-Align is utilized to obtain a feature map of a building detection frame; performing vertex prediction on the building contour by using a long-short term memory network based on an attention mechanism; and then based on the generative adversarial network, building transformation detection is completed by adopting an end-to-end double-branch structure detection method. According to the invention, the precision of cross-modal building image extraction can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a cross-modal building image extraction method, device, electronic device and medium based on multi-feature synergy. Background Art

[0002] With the acceleration of urbanization, the demand for efficient and accurate building information extraction is growing. Traditional building image extraction methods often rely on single modality data, such as optical images or synthetic aperture radar data, which to some extent limits the extraction accuracy and application scope.

[0003] In recent years, the rapid development of deep learning technology has provided new possibilities for cross-modal building image extraction. In the existing technology, convolutional neural networks have become the mainstream tool for image feature extraction and classification. However, there are still deficiencies in the current research on the collaborative processing of multimodal data, especially how to effectively integrate data obtained by different sensors to improve the accuracy and robustness of building detection. Although some researchers have tried to enhance the integration effect of multimodal data through strategies such as feature-level fusion and decision-level fusion, these methods often fail to fully utilize the complementary information between the modal data, or perform poorly when dealing with building recognition problems in complex environments.

[0004] In addition, there is insufficient research on the recovery mechanism of remote sensing blurred information in the existing technology, and a complete solution has not yet been formed to deal with the common blur problems in remote sensing images, which further limits the application prospects of cross-modal building image extraction technology.

[0005] Therefore, studying a new cross-modal building image extraction method based on deep learning technology can not only improve the accuracy and robustness of building recognition in complex environments, but also meet the urgent needs for efficient and accurate building information extraction in the current urbanization process. Summary of the invention

[0006] The purpose of the present invention is to provide a cross-modal building image extraction method, device, electronic device and medium based on multi-feature synergy, which improves the accuracy of cross-modal building image extraction by fully identifying and extracting information in building images.

[0007] To achieve the above object, the first aspect of the present invention provides a cross-modal building image extraction method based on multi-feature synergy, which adopts the following technical solution:

[0008] A cross-modal building image extraction method based on multi-feature synergy, comprising:

[0009] A single building instance segmentation network is constructed based on the MaskRCNN framework; a direction parameter is added to the position prediction branch of the single building instance segmentation network to obtain the building image; the receptive field module is embedded in the prediction branch, ShuffleNet is used as the network backbone to extract building features, and ROI-Align is used to obtain the feature map of the building detection frame; a long short-term memory network based on the attention mechanism is used to predict the vertices of the building contour; and a detection method based on a generative adversarial network and an end-to-end dual-branch structure is used to complete the building transformation detection.

[0010] Specifically, this method constructs a single building instance segmentation network based on the MaskRCNN framework, and adds direction parameters to the location prediction branch of the network to obtain the building image.

[0011] Furthermore, the process of the single building instance segmentation network includes: building a single building instance segmentation network based on the MaskRCNN framework, building a region proposal network, and proposing candidate object bounding boxes. For region proposals, the proposed feature map is ROI pooled according to the region, and through the remaining network, the category, bounding box and binary mask are output, which can group highly overlapping boxes of the same category and select only the prediction with the highest confidence, avoiding repeated box selection of the same target and improving the accuracy of building instance segmentation.

[0012] Furthermore, the process of adding the direction parameter to the location prediction branch of the network includes: outputting multiple bounding boxes and warping them into a fixed dimension; then inputting the warped features into the fully connected layer, using softmax for classification, and using the regression model to further refine the bounding box predictions. The warped features are then input into the Mask classifier, which consists of two CNNs and outputs a binary mask for each ROI. The mask classifier allows the network to generate a mask for each class without competing between classes; then a multi-task loss function is defined for the direction parameter:

[0013] L=L cls +L box +L mask

[0014] Where, L cls represents the classification loss, L box represents the bounding box loss, L mask represents the binary mask loss. This mask branch outputs Km for each ROI 2 , which are K binary masks of m×m resolution, representing K classes.

[0015] Furthermore, the receptive field module is embedded into the segmentation branch, and the feature map extracted from the network is divided into two branches here, and one branch is divided into multiple sub-regions; then the channel size is adjusted by 1×1 convolution, and the size before pooling is obtained by bilinear interpolation; then the two branches are merged. Then the extracted feature map (channel = N) is pooled to obtain a feature pyramid; 1×1 deep convolution is used to reduce the channel to obtain 1×1, 2×2, 4×4, 6×6 feature maps = 1 / N feature maps, and the feature maps are bilinearly interpolated and filled to upsample to the original feature map size, and the channels are spliced ​​with the feature maps to obtain a feature map with a channel number increased by 1 times, and the spliced ​​feature maps are deeply convolved and reduced using a 1×1 convolution kernel to obtain the final semantic segmentation prediction result consistent with the number of channels of the input feature map. By fusing the features of different receptive fields and the features of sub-regions to enhance the feature representation ability, the recognition efficiency of edge features of different buildings is improved.

[0016] Furthermore, Shuffle Net is used as the network backbone to extract building features. The prediction results after each group convolution are grouped within the group; then the subgroups within each group are exchanged with each other; Channel Shuffle is used to allow group convolution to obtain input data from different groups, thereby reducing the computational cost while maintaining the effectiveness and accuracy of building feature extraction.

[0017] Furthermore, ROI-Align is used to obtain the feature map of the building detection frame, and each ROI candidate area is traversed, and the ROI candidate area is divided into several small grids. For the predicted building detection frame, it is mapped to the output feature map. The coordinates of the detection frame are divided by the size of the input image, and then multiplied by the size of the feature map to obtain the corresponding area on the feature map; then the corresponding area on the feature map is divided into sub-windows of fixed size; then the floating point boundary of the sub-window is kept without quantization, and the coordinates of the center point of the sub-window are added with 0.5 and rounded; then bilinear interpolation is performed in each grid to find four integer coordinate points around the coordinates of the sub-window, and the value of the sub-window is calculated according to the value of the coordinate point on the feature map and the distance weight; then the values ​​of all sub-windows are arranged according to their positions on the feature map to obtain a fixed-size feature map, thereby converting the entire feature aggregation process into a continuous operation process and improving the rate of building image extraction.

[0018] Further, the building outline data is obtained, and the building outline data is used to represent the outline information of the building; the outline information refers to a set of two-dimensional outline points or vertices of a two-dimensional model; the outline information of the building should be converted into a sequence form suitable for processing by a long short-term memory network; then the building outline data is normalized and the coordinate data is scaled to a standard range; the long short-term memory network includes: inputting the building outline data specifically includes: adding a cell state as a repository for storing historical information after the input, and controlling the input and output of its information by three gating units. The gating unit includes: an input gate, a forget gate, and an output gate. Then the calculation formula for the input gate, the forget gate, and the output gate is defined:

[0019] i t =σ(W xi x t +W hi h t-1 +b i )

[0020] f t =σ(W xf x t +W hf h t-1 +b f )

[0021] o t =σ(W xo x t +W ho h t-1 +b o )

[0022] In the formula, x t represents the tth element of the input sequence, h t-1 represents the hidden state of the previous moment; W xi represents the weight matrix of the input gate, W hi Represents the weight matrix of the hidden state, b i represents the bias vector; W xf Represents the weight matrix of the forget gate; W hf The weight matrix representing the hidden state; b f represents the bias vector; W xo Represents the weight matrix of the output gate, W ho Represents the weight matrix of the hidden state, b o represents the bias vector; σ represents the sigmoid function.

[0023] The calculation formula for the cell state is defined as:

[0024] c t =ft c t-1 +i t tanh(W xc x t +W hc h t-1 +b c )

[0025] Among them, c t-1 Indicates the cell state at the last moment, W xc The weight matrix representing the cell state, W hc Represents the weight matrix of the hidden state, b c represents the bias vector, and tanh represents the hyperbolic tangent function.

[0026] The attention mechanism is constructed based on the long short-term memory network, specifically including:

[0027] For the elements in the cell state, the similarity or correlation between the cell state and each instruction is calculated, and the instruction is the weight coefficient corresponding to each element; then the weight coefficients are weighted and summed to obtain the final value.

[0028] The method adopts the attention mechanism to define the calculation formula:

[0029]

[0030] In the formula, Source represents the cell state, Query represents the element of the cell state, and Key i Instructions representing cell states, Value i Represents the weight coefficient corresponding to each element of the cell state; Similarity(Query, Keyi) represents the weighted sum of the weight coefficients, L x =||Source||, indicating the length of Source. By obtaining the cell state used to represent the building outline data in each output state, the boundary information of the fuzzy building outline is effectively fitted, thereby improving the extraction accuracy of the building outline.

[0031] Furthermore, a generative adversarial network with two branches is constructed based on the method, wherein the two branches of the generative adversarial network share a feature extraction layer; the two branches of the generative adversarial network have independent output layers to process images of different categories; the generative adversarial network learns the mapping from the input image to the output image in an end-to-end manner; the first branch of the generative adversarial network is used to detect the image before the transformation, and the second branch is used to detect the image after the transformation; the generator of the generative adversarial network converts the random noise vector into a realistic output image; the discriminator of the generative adversarial network distinguishes between the real image and the forged image generated by the generator. The generator learns the mapping from the input to the output image, and learns the feature representation of the building before and after the transformation in the feature space. The generative adversarial network uses a pre-trained network to detect the building transformation. The image before the change is input into the generative adversarial network, and then the difference between the network output and the actual image after the transformation is observed to detect the transformation of the building, so as to achieve accurate recognition of the building transformation process.

[0032] The second aspect of the present invention provides a cross-modal building image extraction device based on multi-feature synergy, which adopts the following technical solution:

[0033] A cross-modal building image extraction device based on multi-feature synergy, comprising:

[0034] A single building instance segmentation module for building a region proposal network that outputs categories, bounding boxes, and binary masks;

[0035] A building image acquisition module, for adding a direction parameter based on the location prediction branch of the single building instance segmentation network, the direction parameter is used to refine the bounding box prediction and then input the distorted features into a Mask classifier and output a binary mask;

[0036] A building feature extraction module is used to embed a receptive field module into the segmentation branch. The receptive field module allows grouped convolution to obtain input data from different groups through Channel Shuffle. The Channel Shuffle uses ROI-Align to obtain a feature map of the building detection box;

[0037] A building outline vertex prediction module is used for the long short-term memory network to normalize the building outline data, scale the coordinate data to a standard range, and the long short-term memory network based on the attention mechanism obtains the final value according to the input coordinate data;

[0038] The building transformation detection module is used to observe the difference between the network output and the actual image after transformation using an end-to-end dual-branch generative adversarial network.

[0039] A third aspect of the present invention provides an electronic device, which adopts the following technical solution:

[0040] An electronic device, comprising:

[0041] at least one processor;

[0042] Memory;

[0043] at least one application, wherein the at least one application is stored in the memory and configured to be executed by at least

[0044] A processor executes, and the at least one application is configured to: execute the above-mentioned building image visual recognition method.

[0045] The fourth aspect of the present invention provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the technical solution provided in the first aspect, a cross-modal building image extraction method based on multi-feature synergy.

[0046] The significant advantages of the present invention compared with the prior art are:

[0047] 1. The present invention constructs a single building instance segmentation network based on the Mask RCNN framework, adds a direction parameter to the location prediction branch of the network to obtain the building image, can group highly overlapping boxes of the same type, and only select the prediction with the highest confidence, avoiding repeated box selection of the same target and improving the accuracy of building instance segmentation.

[0048] 2. The present invention embeds the receptive field module into the segmentation branch, performs bilinear interpolation padding and upsamples the feature map to the original feature map size, and enhances the feature representation capability by fusing the features of different receptive fields and the features of sub-regions, thereby improving the recognition efficiency of edge features of different buildings.

[0049] 3. The present invention adopts Shuffle Net as the network backbone to extract building features, groups the prediction results after each group convolution within the group, and allows group convolution to obtain input data from different groups through Channel Shuffle, thereby reducing the computational cost while maintaining the effectiveness and accuracy of building feature extraction.

[0050] 4. The present invention uses ROI-Align to obtain the feature map of the building detection frame, traverses each ROI candidate area, maps the predicted building detection frame to the output feature map, converts the entire feature aggregation process into a continuous operation process, and improves the rate of building image extraction.

[0051] 5. After inputting the building outline data, the present invention adds the cell state as a repository for storing historical information, and three gating units are used to control the input and output of its information. By obtaining the cell state used to represent the building outline data in each output state, the boundary information of the fuzzy building outline is effectively fitted, thereby improving the extraction accuracy of the building outline.

[0052] 6. The present invention constructs a generative adversarial network with two branches and uses a pre-trained network to detect building transformation. The image before the change is input into the generative adversarial network, and then the difference between the network output and the actual image after the transformation is observed to detect the transformation of the building, thereby achieving accurate recognition of the building transformation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0054] Figure 1 It is a flow chart of a cross-modal building image extraction method based on multi-feature synergy proposed in an embodiment of the present invention;

[0055] Figure 2 is a block diagram of a cross-modal building image extraction device based on multi-feature synergy proposed in an embodiment of the present invention;

[0056] Figure 3 Schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0059] The embodiment of the present invention provides a cross-modal building image extraction method based on multi-feature synergy, which is executed by an electronic device, which can be a server or a terminal device, wherein the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a virtual machine, a desktop computer, etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present invention. Figure 1 As shown, the method includes:

[0060] Step S10, obtaining a cross-modal building image.

[0061] For the embodiments of the present invention, the modal parameters of building image acquisition include building instance segmentation accuracy, building edge feature recognition rate, building image extraction calculation rate, building image extraction recognition rate, building contour extraction accuracy, building transformation recognition accuracy, etc. When the electronic device acquires a cross-modal building image, the staff uploads the modal parameters acquired from the building image to the recognition device. After receiving the modal parameters, the recognition device executes a cross-modal building image extraction method based on multi-feature synergy provided by the embodiments of the present invention on the modality acquired from the building image. Then the recognition device encapsulates the above information and sends it to the electronic device.

[0062] Step S11, performing building instance segmentation on the cross-modal building image.

[0063] In an embodiment of the present invention, a Mask RCNN framework is used to construct a single building instance segmentation network, and a direction parameter is added to the location prediction branch of the network to perform building instance segmentation.

[0064] Specifically, a single building instance segmentation network based on the Mask RCNN framework is used to construct a region proposal network to propose candidate object bounding boxes. For region proposals, the proposed feature map is ROI pooled according to the region, and the category, bounding box, and binary mask are output through the remaining network. For example: an 8×8 input feature map is obtained through the convolution layer, and a 7×5 region is used as the input of the ROI to output a 2×2 feature map. Each box selection rectangle is rounded to have an integer length for subsequent pooling. For each value of the output feature map, only the maximum value of each box selection rectangle is selected, so that highly overlapping boxes of the same class can be grouped and only the prediction with the highest confidence is selected, avoiding repeated box selection of the same target.

[0065] Preferably, based on a single building instance segmentation network, a direction parameter is added to the location prediction branch of the network, including: outputting multiple bounding boxes and warping them into a fixed dimension; then inputting the warped features into a fully connected layer, using softmax for classification, and using a regression model to further refine the bounding box predictions. Then the warped features are input into a Mask classifier, which consists of two CNNs and outputs a binary mask for each ROI. The mask classifier allows the network to generate a mask for each class without competing between classes; then a multi-task loss function is defined for the direction parameter:

[0067] L= +L box +L mask

[0068] In the formula, represents the classification loss, represents the bounding box loss, and represents the binary mask loss. The mask branch outputs Km for each ROI 2 , they are K binary masks of m×m resolution, representing K classes. For example: ROI output is 40*52, which means they are 40 binary masks of 5×5 resolution, representing 40 classes.

[0069] Step S12: performing building edge feature recognition on the cross-modal building image.

[0070] In an embodiment of the present invention, the receptive field module is embedded in the segmentation branch, and the first feature map extracted from the single building instance segmentation network is divided into two branches here, and one branch is divided into multiple sub-regions; then the channel size is adjusted by 1×1 convolution, and the size before pooling is obtained by bilinear interpolation; then the two branches are merged. Then the extracted first feature map (channel = N) is pooled to obtain a feature pyramid; then the first feature map = 1 / N second feature map of 1×1, 2×2, 4×4, 6×6 is obtained by 1×1 depth convolution and channel reduction respectively; then the second feature map is bilinearly interpolated and filled and upsampled to the size of the first feature map, and the channel is spliced ​​with the first feature map to obtain a third feature map with a channel number increased by 1 times; then the third feature map is deeply convolved and channel reduced using a 1×1 convolution kernel to obtain the final semantic segmentation prediction result consistent with the number of channels of the input second feature map, and the feature map representation ability is enhanced by merging the feature maps of different receptive fields and the feature maps of the sub-regions, thereby improving the recognition efficiency of the edge features of different buildings.

[0071] Step S13, reducing the computational cost of building image extraction in the cross-modal building image recognition process.

[0072] Specifically, Shuffle Net is used as the network backbone to extract the first feature map, and the prediction results after each group convolution are grouped within the group. Channel Shuffle is used to allow group convolution to obtain input data from different groups, thereby reducing the computational cost while maintaining the effectiveness and accuracy of building feature extraction.

[0073] Step S14, improving the building image extraction and recognition rate in the cross-modal building image recognition process.

[0074] Specifically, ROI-Align is used to obtain the feature map of the building detection frame, and then each ROI candidate region is traversed, and the ROI candidate region is divided into several small grids. For the predicted building detection frame, it is mapped to the output feature map, and then the coordinates of the detection frame are divided by the size of the input image, and then multiplied by the size of the feature map to obtain the corresponding area on the feature map, and then the corresponding area on the feature map is divided into sub-windows of fixed size; then the floating point boundary of the sub-window is kept without quantization, and the center point coordinates of the sub-window are added with 0.5 and rounded to the nearest integer; then bilinear interpolation is performed in each grid to find four integer coordinate points around the sub-window coordinates, and the value of the sub-window is calculated according to the value of the coordinate point on the feature map and the distance weight; then the values ​​of all sub-windows are arranged according to their positions on the feature map to obtain a fixed-size feature map. For example: if the center point coordinates of the sub-window are (0.6, 0.5), the rounded result is (1, 1); if the center point coordinates of the sub-window are (0.9, 1.5), the rounded result is (1, 2).

[0075] Step S15: extracting building contours from the cross-modal building image.

[0076] In an embodiment of the present invention, a long short-term memory network based on an attention mechanism is used to predict vertices of a building outline, and then building outline data is obtained. The building outline data is used to represent outline information containing the building, and the outline information includes: a two-dimensional outline point set or vertices of a two-dimensional model; the building outline information should be converted into a sequence form suitable for processing by the long short-term memory network; then the building outline data is normalized to scale the coordinate data to within a standard range.

[0077] Specifically, the building outline data is input into the long short-term memory network, and the cell state is added after the input as a repository for storing historical information. The input and output of its information are controlled by three gate control units, including: input gate, forget gate and output gate. The calculation formulas for the input gate, forget gate and output gate are as follows:

[0078] i t =σ(W xi xt +W hi h t-1 +b i )

[0079] f t =σ(W xf x t +W hf h t-1 +b f )

[0080] o t =σ(W xo x t +W ho h t-1 +b o )

[0081] In the formula, x t represents the tth element of the input sequence, h t-1 represents the hidden state of the previous moment; W xi represents the weight matrix of the input gate, W hi Represents the weight matrix of the hidden state, b i represents the bias vector; W xf Represents the weight matrix of the forget gate; W hf The weight matrix representing the hidden state; b f represents the bias vector; W xo Represents the weight matrix of the output gate, W ho Represents the weight matrix of the hidden state, b o represents the bias vector; σ represents the sigmoid function.

[0082] The calculation formula of cell state is as follows:

[0083] c t =f t c t-1 +i t tanh(W xc x t +W hc h t-1 +b c )

[0084] Among them, represents the cell state at the previous moment, represents the weight matrix of the cell state, represents the weight matrix of the hidden state, represents the bias vector, and tanh represents the hyperbolic tangent function.

[0085] Specifically, an attention mechanism is constructed based on the long short-term memory network. For the elements in the cell state, the similarity or correlation between the cell state and the weight coefficient corresponding to each element is calculated, and then the weight coefficient is weighted and summed to obtain the final value. The weighted calculation formula is as follows:

[0086]

[0087] In the formula, Source represents the cell state, Query represents the element of the cell state, and Key i Instructions representing cell states, Value i Represents the weight coefficient corresponding to each element of the cell state; Similarity(Query, Keyi) represents the weighted sum of the weight coefficients, L x =||Source||, indicating the length of Source.

[0088] Specifically, the weighted calculation process is divided into three stages. In the first stage, the dot product calculation and Cosin similarity calculation are performed on the elements of the cell state and each instruction, using the following formula:

[0089] Among them, Similarity(Query,Key i )=Query*Key i For dot product calculation and formula;

[0090] Among them, Similarity(Query,Key i )=Query×Key i is the Cosin similarity calculation formula;

[0091] The second stage normalizes the original weights obtained in the first stage and maps the range of values ​​to between 0 and 1 using the following formula:

[0092]

[0093] In the third stage, the Value is weighted and summed according to the weight coefficient to obtain the final attention value using the following formula:

[0094]

[0095] Preferably, by acquiring the cell state for representing the building outline data in each output state, the boundary information of the fuzzy building outline is effectively fitted, thereby improving the extraction accuracy of the building outline.

[0096] Step S16, accurately identifying the building transformation process of the cross-modal building image.

[0097] In an embodiment of the present invention, a detection method based on a generative adversarial network and an end-to-end dual-branch structure is used to complete building transformation detection, and a generative adversarial network with two branches is constructed. The two branches share a feature extraction layer, and the two branches have independent output layers to process images of different categories; the generative adversarial network learns the mapping from the input image to the output image in an end-to-end manner; the first branch is used to detect the image before the transformation, and the second branch is used to detect the image after the transformation; the generator of the generative adversarial network converts a random noise vector into a realistic output image; the discriminator distinguishes between real images and forged images generated by the generator, and the generator learns the mapping from the input to the output image, and learns the feature representation of the building before and after the transformation in the feature space, and then uses the pre-trained network to perform building transformation detection.

[0098] Preferably, the image before the change is input into a generative adversarial network, and then the difference between the network output and the actual image after the transformation is observed to detect the transformation of the building, thereby achieving accurate identification of the building transformation process.

[0099] The above-mentioned embodiment introduces a cross-modal building image extraction method based on the synergy of multiple features from the perspective of method flow. The following embodiment introduces a cross-modal building image extraction device based on the synergy of multiple features from the perspective of a virtual module or a virtual unit. Please refer to the following embodiments for details.

[0100] The embodiment of the present invention provides a cross-modal building image extraction device based on multi-feature synergy, such as Figure 2 As shown, a cross-modal building image extraction device 20 based on multi-feature synergy may specifically include: a single building instance segmentation module 21, a building image acquisition module 22, a building feature extraction module 23, a building contour vertex prediction module 24, and a building transformation detection module 25.

[0101] The single building instance segmentation module 21 is used to construct a region proposal network, and the proposal network is used to output categories, bounding boxes and binary masks;

[0102] A building image acquisition module 22, for adding a direction parameter based on the location prediction branch of the single building instance segmentation network, the direction parameter is used to refine the bounding box prediction and then input the distorted features into a Mask classifier and output a binary mask;

[0103] A building feature extraction module 23, used to embed a receptive field module into the segmentation branch, wherein the receptive field module allows grouped convolution to obtain input data from different groups through Channel Shuffle, and the Channel Shuffle uses ROI-Align to obtain a feature map of the building detection frame;

[0104] The building outline vertex prediction module 24 is used for the long short-term memory network to normalize the building outline data, scale the coordinate data to a standard range, and obtain the final value based on the input coordinate data by the long short-term memory network based on the attention mechanism;

[0105] The building transformation detection module 25 is used to use an end-to-end dual-branch structured generative adversarial network to observe the difference between the network output and the actual image after transformation.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0107] The embodiment of the present invention also introduces an electronic device from the perspective of a physical device, such as Figure 3 As shown, Figure 3 The electronic device 300 shown includes, in addition to conventional configuration devices: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, such as through a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of the present invention.

[0108] The processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor) or other programmable logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. The processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0109] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0110] The memory 303 may be a ROM (Read Only Memory) or a memory that can store static information and instructions.

[0111] It may be other types of static storage devices, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0112] The memory 303 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the contents shown in the above method embodiment.

[0113] The electronic device includes, but is not limited to, fixed terminals such as virtual machines and desktop computers, and may also be a server. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0114] The embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment. Compared with the prior art, in the embodiment of the present invention, by constructing a single building instance segmentation network based on the Mask RCNN framework, a direction parameter is added to the position prediction branch of the network to obtain a building image, and highly overlapping boxes of the same type can be grouped, and only the prediction with the highest confidence is selected, thereby avoiding repeated box selection of the same target and improving the accuracy of building instance segmentation. Then, the receptive field module is embedded in the segmentation branch, and the feature map is bilinearly interpolated and padded and upsampled to the original feature map size. By fusing the features of different receptive fields and the features of sub-regions to enhance the feature representation capability, the recognition efficiency of edge features of different buildings is improved. Then, Shuffle Net is used as the network backbone to extract building features, and the prediction results after each group convolution are grouped within the group. Through Channel Shuffle allows grouped convolution to obtain input data from different groups, reducing the computational cost while maintaining the effectiveness and accuracy of building feature extraction. Then, ROI-Align is used to obtain the feature map of the building detection frame, and each ROI candidate area is traversed. For the predicted building detection frame, it is mapped to the output feature map, and the entire feature aggregation process is converted into a continuous operation process to improve the rate of building image extraction. Then, after the building outline data is input, the cell state is added as a repository for storing historical information, and the input and output of its information are controlled by three gated units. By obtaining the cell state used to represent the building outline data under each output state, the boundary information of the fuzzy building outline is effectively fitted, and the extraction accuracy of the building outline is improved. Then, a generative adversarial network with two branches is constructed, and the pre-trained network is used to detect building transformation. The image before the change is input into the generative adversarial network based on the above-mentioned network, and then the difference between the network output and the actual image after the transformation is observed to detect the transformation of the building, so as to achieve accurate recognition of the building transformation process.

[0115] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0116] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cross-modal building image extraction method based on multi-feature synergy, characterized by: By constructing a single building instance segmentation network based on the MaskRCNN framework, the direction parameter is added to the location prediction branch of the network to obtain the building image; then the receptive field module is embedded in the segmentation branch, ShuffleNet is used as the network backbone to extract building features, and ROI-Align is used to obtain the feature map of the building detection frame; then the long short-term memory network based on the attention mechanism is used to predict the vertices of the building outline; Then, based on the generative adversarial network, an end-to-end dual-branch structure detection method is adopted to complete the building transformation detection.

2. The cross-modal building image extraction method based on multi-feature synergy according to claim 1, characterized in that: A single building instance segmentation network based on the MaskRCNN framework is constructed, and a direction parameter is added to the location prediction branch of the network to obtain a building image, including: constructing a single building instance segmentation network based on the Mask RCNN framework, constructing a region proposal network, and proposing a candidate object bounding box. For the region proposal, the proposed feature map is ROI pooled according to the region, and the category, bounding box, and binary mask are output through the remaining network. Based on the single building instance segmentation network, a direction parameter is added to the location prediction branch of the network, including: outputting multiple bounding boxes and warping them into a fixed dimension; then the warped features are input into the fully connected layer, classified using softmax, and the bounding box prediction is further refined using a regression model. Then the warped features are input into a Mask classifier, which consists of two CNNs and outputs a binary mask for each ROI. The mask classifier allows the network to generate a mask for each class without competing between classes; then a multi-task loss function is defined for the direction parameter: L= +L box +L mask Where, L cls represents the classification loss, L box represents the bounding box loss, L mask represents the binary mask loss. This mask branch outputs Km2 for each ROI, which are K binary masks of m×m resolution, representing K classes.

3. The cross-modal building image extraction method based on multi-feature synergy according to claim 2, characterized in that: The receptive field module is embedded into the segmentation branch, including: the feature map extracted from the network is divided into two branches here, and one branch is divided into multiple sub-regions; then the channel size is adjusted by 1×1 convolution, and the size before pooling is obtained by bilinear interpolation; then the two branches are merged. Then the extracted feature map (channel = N) is pooled to obtain a feature pyramid; then the channel is reduced by 1×1 depth convolution to obtain 1×1, 2×2, 4×4, 6×6 feature maps = 1 / N feature maps respectively; then the feature map is upsampled to the original feature map size by bilinear interpolation filling, and the channel is spliced ​​with the feature map to obtain a feature map with a channel number doubled; then the spliced ​​feature map is deep convolved and channel reduced by 1×1 convolution kernel to obtain the final semantic segmentation prediction result consistent with the number of channels of the input feature map.

4. The cross-modal building image extraction method based on multi-feature synergy according to claim 2, characterized in that: Shuffle Net is used as the network backbone to extract building features, including: grouping the prediction results after each group convolution within the group; then exchanging the subgroups within each group; and then allowing the group convolution to obtain input data from different groups through Channel Shuffle.

5. The cross-modal building image extraction method based on multi-feature synergy according to claim 2, characterized in that: The feature map of the building detection frame is obtained by using ROI-Align, including: traversing each ROI candidate area, dividing the ROI candidate area into several small grids, and mapping the predicted building detection frame to the output feature map. The coordinates of the detection frame are divided by the size of the input image, and then multiplied by the size of the feature map to obtain the corresponding area on the feature map; then the corresponding area on the feature map is divided into sub-windows of fixed size; then the floating point number boundaries of the sub-window are kept unquantized, and the coordinates of the center point of the sub-window are added with 0.5 and rounded; then bilinear interpolation is performed in each grid to find four integer coordinate points around the coordinates of the sub-window, and the value of the sub-window is calculated according to the value of the coordinate point on the feature map and the distance weight; then the values ​​of all sub-windows are arranged according to their positions on the feature map to obtain a fixed-size feature map.

6. The cross-modal building image extraction method based on multi-feature synergy according to claim 5, characterized in that: The vertex prediction of the building outline is performed using a long short-term memory network based on an attention mechanism, including: obtaining building outline data, the building outline data is used to represent the outline information of the building; the outline information refers to a set of two-dimensional outline points or vertices of a two-dimensional model; the building outline information should be converted into a sequence form suitable for processing by a long short-term memory network; then the building outline data is normalized to scale the coordinate data to a standard range; the long short-term memory network includes: inputting the building outline data specifically includes: adding a cell state as a repository for storing historical information after the input, and controlling the input and output of its information by three gating units. The gating unit includes: an input gate, a forget gate, and an output gate. Then, the calculation formula for the input gate, the forget gate, and the output gate is defined: i t =σ(W xi x t +W hi h t-1 +b i ) f t =σ(W xf x t +W hf h t-1 +b f ) o t =σ(W xo x t +W ho h t-1 +b o ) In the formula, x t represents the tth element of the input sequence, h t-1 represents the hidden state of the previous moment; W xi represents the weight matrix of the input gate, W hi The weight matrix representing the hidden state, b i represents the bias vector; W xf Represents the weight matrix of the forget gate; W hf The weight matrix representing the hidden state; b f represents the bias vector; W xo Represents the weight matrix of the output gate, W ho The weight matrix representing the hidden state, b o represents the bias vector; σ represents the sigmoid function. The calculation formula for the cell state is defined as: c t =f t c t-1 +i t tanh(W xc x t +W hc h t-1 +b c ) Among them, c t-1 Indicates the cell state at the last moment, W xc The weight matrix representing the cell state, W hc The weight matrix representing the hidden state, b c represents the bias vector, and tanh represents the hyperbolic tangent function. The attention mechanism is constructed based on the long short-term memory network, specifically including: For the elements in the cell state, the similarity or correlation between the cell state and each instruction is calculated, and the instruction is the weight coefficient corresponding to each element; then the weight coefficients are weighted and summed to obtain the final value. The method adopts the attention mechanism to define the calculation formula: In the formula, Source represents the cell state, Query represents the element of the cell state, and Key i Instructions representing cell states, Value i Represents the weight coefficient corresponding to each element of the cell state; Similarity(Query, Keyi) represents the weighted sum of the weight coefficients, L x =||Source||, indicating the length of Source.

7. The cross-modal building image extraction method based on multi-feature synergy according to claim 6 is characterized in that: A detection method based on a generative adversarial network and an end-to-end dual-branch structure is used to complete building transformation detection, including: constructing a generative adversarial network with two branches based on the method, the two branches of the generative adversarial network share a feature extraction layer; the two branches of the generative adversarial network have independent output layers to process images of different categories; the generative adversarial network learns the mapping from the input image to the output image in an end-to-end manner; the first branch of the generative adversarial network is used to detect the image before the transformation, and the second branch is used to detect the image after the transformation; the generator of the generative adversarial network converts a random noise vector into a realistic output image; the discriminator of the generative adversarial network distinguishes between a real image and a forged image generated by the generator. The generator learns the mapping from the input to the output image, and learns the feature representation of the building before and after the transformation in the feature space. The generative adversarial network uses a pre-trained network to detect building transformation. The image before the change is input into the generative adversarial network, and then the difference between the network output and the actual image after the transformation is observed to detect the transformation of the building.

8. A cross-modal building image extraction device based on multi-feature synergy, characterized in that: A module for executing the method according to any one of claims 1 to 7, comprising: A single building instance segmentation module for building a region proposal network that outputs categories, bounding boxes, and binary masks; A building image acquisition module, for adding a direction parameter based on the location prediction branch of the single building instance segmentation network, the direction parameter is used to refine the bounding box prediction and then input the distorted features into a Mask classifier and output a binary mask; A building feature extraction module is used to embed a receptive field module into the segmentation branch. The receptive field module allows grouped convolution to obtain input data from different groups through Channel Shuffle. The Channel Shuffle uses ROI-Align to obtain a feature map of the building detection box; A building outline vertex prediction module is used for the long short-term memory network to normalize the building outline data, scale the coordinate data to a standard range, and the long short-term memory network based on the attention mechanism obtains the final value according to the input coordinate data; The building transformation detection module is used to observe the difference between the network output and the actual image after transformation using an end-to-end dual-branch generative adversarial network.

9. An electronic device, characterized in that: The electronic device includes: at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the cross-modal building image extraction method based on multi-feature synergy as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed in a computer, the computer is caused to execute the cross-modal building image extraction method based on multi-feature synergy as described in any one of claims 1 to 7.

Images