Image processing method, device, equipment and storage medium

Abstract

The present application proposes an image processing method, apparatus, device and storage medium. The method comprises obtaining a target image containing a building. Extracting a roof area of ​​the building from the target image, and determining a predicted offset between the roof and the base of the building based on the target image. According to the predicted offset, transforming the roof area to obtain the base area of ​​the building.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] At present, in the field of image processing, it is usually necessary to extract buildings from images for urban planning, map drawing, building change detection, etc. One of the important tasks in building extraction is to extract the building base.

[0003] However, since the target image is usually a remote sensing image taken by a satellite or an airplane, the building base in the image may be partially blocked, resulting in unclear visual features, which affects the extraction accuracy of the building base. Summary of the invention

[0004] In view of this, the present application discloses at least one image processing method, which includes: obtaining a target image including a building; extracting a roof area of ​​the building from the target image, and determining a predicted offset between the roof and the base of the building based on the target image; and transforming the roof area according to the predicted offset to obtain the base area of ​​the building.

[0005] In some examples shown, the method further includes:

[0006] The target image is subjected to target detection by using a building bounding box prediction submodel to obtain a bounding box of the building; the roof area of ​​the building is extracted from the target image, comprising: determining the roof area of ​​the building included in the bounding box based on the bounding box, the target image and the roof area prediction submodel; the predicted offset between the roof and the base of the building based on the target image is determined, comprising: determining the predicted offset between the roof and the base of the building included in the bounding box based on the bounding box, the target image and the offset prediction submodel.

[0007] In some of the examples shown, the roof area prediction submodel and the offset prediction submodel share the same regional feature extraction unit; the method also includes: determining the building features corresponding to the building included in the bounding box based on the bounding box, the target image and the regional feature extraction unit; the determining the roof area of ​​the building included in the bounding box based on the bounding box, the target image and the roof area prediction submodel includes: performing a first convolution processing on the building features using the first convolution processing unit included in the roof area prediction submodel to obtain the roof area of ​​the building; the determining the predicted offset between the roof and the base of the building included in the bounding box based on the bounding box, the target image and the offset prediction submodel includes: performing a second convolution processing on the building features using the second convolution processing unit included in the offset prediction submodel to obtain the predicted offset between the roof and the base of the building included in the bounding box.

[0008] In some of the examples shown, the roof area is transformed according to the predicted offset to obtain the base area corresponding to the building, including: determining the base area corresponding to the building based on the predicted offset, the building features corresponding to the building and the base area prediction sub-model.

[0009] In some of the examples shown, the base area corresponding to the building is determined based on the predicted offset, the building features corresponding to the building and the base area prediction sub-model, including: using the spatial transformation network included in the base area prediction sub-model to translate the building features corresponding to the roof area to obtain the base features corresponding to the building; wherein the spatial transformation parameters corresponding to the spatial transformation network include parameters determined based on the predicted offset; and using the base area prediction sub-model to perform a third convolution process on the base features to obtain the base area corresponding to the building.

[0010] In some of the examples shown, the above-mentioned spatial transformation network includes a sampler constructed based on an interpolation method; wherein the above-mentioned sampler includes a sampling grid constructed based on the above-mentioned prediction offset; the above-mentioned spatial transformation network included in the above-mentioned base area prediction submodel is used to perform a translation transformation on the building features corresponding to the above-mentioned roof area to obtain the base features corresponding to the above-mentioned building, including: using the above-mentioned sampler, according to the coordinate information of each pixel point of the base feature, each pixel point corresponding to the base feature is used as the current pixel point in turn, and the pixel point corresponding to the above-mentioned current pixel point is determined through the above-mentioned sampling grid among the pixels included in the roof area, and the determined pixel point is translated based on the interpolation method to obtain the pixel value corresponding to the above-mentioned current pixel point.

[0011] In some of the examples shown, a roof contour prediction submodel is also included that shares the same area feature extraction unit with the roof area prediction submodel and the offset prediction submodel; the method also includes: performing contour regression on the building features using the roof contour prediction submodel to determine the roof contour of the building; performing a translation transformation on the roof contour according to the predicted offset to obtain the base contour of the building; and adjusting the base area based on the base contour to obtain the final base area corresponding to the building.

[0012] In some of the examples shown, the roof outline prediction submodel is used to perform outline regression on the building features to determine the roof outline of the building, including: extracting multiple connection points from the building features; combining at least some of the multiple connection points in pairs to obtain multiple line segments; predicting the multiple line segments to obtain prediction scores corresponding to each line segment; wherein the prediction scores are used to indicate the probability that the line segment corresponding to the score belongs to the roof outline; and combining the line segments whose corresponding prediction scores are greater than a preset threshold among the multiple line segments to obtain the roof outline of the building.

[0013] In some of the examples shown, the above method is obtained using an image processing model; wherein the above image processing model includes a building bounding box prediction sub-model, a roof area prediction sub-model, an offset prediction sub-model, a roof contour prediction sub-model and a base area prediction sub-model.

[0014] In some of the examples shown, the training method of the above-mentioned image processing model includes: obtaining multiple training samples including annotation information; wherein the above-mentioned annotation information includes a building bounding box, a building roof area, a building roof outline, a predicted offset between the building roof and the base, and a building base area; based on the loss information corresponding to each sub-model included in the above-mentioned image processing model, constructing joint learning loss information; based on the above-mentioned joint learning loss information and the above-mentioned training samples, jointly training the sub-models included in the above-mentioned image processing model until the above-mentioned sub-models converge.

[0015] The present application also proposes an image processing device, which includes: an acquisition module, used to acquire a target image containing a building; an extraction module, used to extract the roof area of ​​the building from the target image, and determine the predicted offset between the roof and the base of the building based on the target image; and a transformation module, used to transform the roof area according to the predicted offset to obtain the base area of ​​the building.

[0016] In some of the examples shown, the bounding box determination module is used to perform target detection on the above-mentioned target image using the building bounding box prediction sub-model to obtain the bounding box of the above-mentioned building; the above-mentioned extraction module includes: a roof area determination module, which is used to determine the roof area of ​​the building included in the above-mentioned bounding box based on the above-mentioned bounding box, the above-mentioned target image and the roof area prediction sub-model; an offset determination module is used to determine the predicted offset between the roof and the base of the building included in the above-mentioned bounding box based on the above-mentioned bounding box, the above-mentioned target image and the offset prediction sub-model.

[0017] In some of the examples shown, the roof area prediction submodel and the offset prediction submodel share the same regional feature extraction unit; the device also includes: a building feature determination module, which is used to determine the building features corresponding to the building included in the bounding box based on the bounding box, the target image and the regional feature extraction unit; the roof area determination module includes: a first convolution processing module, which is used to perform a first convolution processing on the building features using the first convolution processing unit included in the roof area prediction submodel to obtain the roof area of ​​the building; the offset determination module includes: a second convolution processing module, which is used to perform a second convolution processing on the building features using the second convolution processing unit included in the offset prediction submodel to obtain the predicted offset between the roof and the base of the building included in the bounding box. In some of the examples shown, the transformation module is specifically used to determine the base area corresponding to the building based on the predicted offset, the building features corresponding to the building and the base area prediction submodel.

[0018] In some of the examples shown, the above-mentioned transformation module includes: a first translation transformation module, which is used to use the spatial transformation network included in the above-mentioned base area prediction sub-model to perform a translation transformation on the building features corresponding to the above-mentioned roof area, so as to obtain the base features corresponding to the above-mentioned building; wherein the spatial transformation parameters corresponding to the above-mentioned spatial transformation network include parameters determined based on the above-mentioned prediction offset; a third convolution processing module, which is used to perform a third convolution processing on the above-mentioned base features using the above-mentioned base area prediction sub-model, so as to obtain the base area corresponding to the above-mentioned building.

[0019] In some of the examples shown, the above-mentioned spatial transformation network includes a sampler constructed based on an interpolation method; wherein the above-mentioned sampler includes a sampling grid constructed based on the above-mentioned predicted offset; the above-mentioned first translation transformation module is specifically used to: use the above-mentioned sampler, according to the coordinate information of each pixel point of the base feature, take each pixel point corresponding to the base feature as the current pixel point in turn, determine the pixel point corresponding to the above-mentioned current pixel point among the pixels included in the roof area through the above-mentioned sampling grid, and perform a translation transformation on the determined pixel point based on the interpolation method to obtain the pixel value corresponding to the above-mentioned current pixel point.

[0020] In some of the examples shown, a roof contour prediction submodel is also included that shares the same area feature extraction unit with the roof area prediction submodel and the offset prediction submodel; the above-mentioned device also includes: a contour regression module, which is used to use the roof contour prediction submodel to perform contour regression on the above-mentioned building features to determine the roof contour of the above-mentioned building; a second translation transformation module, which is used to perform a translation transformation on the above-mentioned roof contour according to the above-mentioned predicted offset to obtain the base contour of the above-mentioned building; and a final base area determination module, which is used to adjust the above-mentioned base area based on the above-mentioned base contour to obtain the final base area corresponding to the above-mentioned building.

[0021] In some of the examples shown, the above-mentioned contour regression module is specifically used to: extract multiple connection points from the above-mentioned building features; combine at least some of the above-mentioned multiple connection points in pairs to obtain multiple line segments; predict the above-mentioned multiple line segments to obtain prediction scores corresponding to each line segment; wherein the above-mentioned prediction scores are used to indicate the probability that the line segment corresponding to the score belongs to the roof contour; combine the line segments whose corresponding prediction scores are greater than a preset threshold among the above-mentioned multiple line segments to obtain the roof contour of the above-mentioned building.

[0022] In some of the examples shown, the above-mentioned extraction module is specifically used to: perform image processing on the target image using an image processing model; wherein the image processing model includes a building bounding box prediction sub-model, a roof area prediction sub-model, an offset prediction sub-model, a roof outline prediction sub-model and a base area prediction sub-model.

[0023] In some of the examples shown, the training device corresponding to the training method of the above-mentioned image processing model includes: a training sample acquisition module, used to obtain multiple training samples including annotation information; wherein the above-mentioned annotation information includes a building bounding box, a building roof area, a building roof outline, a predicted offset between the building roof and the base, and a building base area; a loss information determination module, used to construct joint learning loss information based on the loss information corresponding to each sub-model included in the above-mentioned image processing model; a joint training module, used to jointly train the sub-models included in the above-mentioned image processing model based on the above-mentioned joint learning loss information and the above-mentioned training samples until the above-mentioned sub-models converge.

[0024] The present application also proposes an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to call the executable instructions stored in the memory to implement the image processing method shown in any of the above embodiments.

[0025] The present application also proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the image processing method shown in any of the above embodiments.

[0026] In the above scheme, since the building roof area with more obvious features and the predicted offset between the roof and the base can be extracted from the acquired target image, and then the roof area with higher precision is transformed based on the predicted offset, a building base area with higher precision can be obtained. Therefore, in the process of predicting the building base, there is no need to rely on the base features included in the target image, so that even if the building base features included in the target image are blocked, a building base with higher precision can be obtained.

[0027] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in one or more embodiments of the present application or related technologies, the drawings required for use in the embodiments or related technology descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 A method flow chart of an image processing method shown in this application;

[0030] Figure 2A schematic diagram of a process of extracting a base through an image processing model shown in this application;

[0031] Figure 3 A schematic diagram of a method for predicting a roof area by an image processing unit shown in the present application;

[0032] Figure 4 A schematic flow chart of a method for predicting an offset by an image processing unit shown in the present application;

[0033] Figure 5 A schematic flow chart of a method for performing offset prediction and roof area prediction by an image processing unit shown in the present application;

[0034] Figure 6 A schematic flow chart of a method for performing base prediction using a base area prediction sub-model shown in this application;

[0035] Figure 7 A flowchart of a final base area prediction method shown in this application;

[0036] Figure 8 A schematic diagram of a method for predicting roof contours through a bounding box analysis network shown in this application;

[0037] Fig. 9 A schematic diagram of a base area prediction process shown in this application;

[0038] Fig.10 A task-model correspondence diagram for this application;

[0039] Fig.11 A method flow chart of an image processing model training method shown in this application;

[0040] Fig.12 A schematic diagram of an image processing device shown in the present application;

[0041] Fig.13 A hardware structure diagram of a type shown in this application. DETAILED DESCRIPTION

[0042] The exemplary embodiments will be described in detail below, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0043] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "above", and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. It should also be understood that the word "if" used in this article, depending on the context, can be interpreted as "at the time of" or "when" or "in response to determining".

[0044] The present application aims to propose an image processing method. The method extracts a building roof area with obvious features and a predicted offset between the roof and the base from an acquired target image, and then transforms the roof area with higher accuracy based on the predicted offset to obtain a building base area with higher accuracy. In this way, in the process of predicting the building base, it is not necessary to rely on the base features included in the target image, so that even if the building base features included in the target image are blocked, a building base with higher accuracy can be obtained.

[0045] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of an image processing method shown in this application. Figure 1 As shown, the above method may include:

[0046] S102, acquiring a target image containing a building.

[0047] S104, extracting the roof area of ​​the building from the target image, and determining the predicted offset between the roof and the base of the building based on the target image.

[0048] S106: According to the predicted offset, transform the roof area to obtain the base area of ​​the building.

[0049] The above image processing method can be applied to electronic devices. The above electronic devices can execute the above image processing method by carrying a software system corresponding to the image processing method. In the embodiment of the present application, the type of the above electronic device can be a laptop, a computer, a server, a mobile phone, a PAD terminal, etc., which is not particularly limited in the present application.

[0050] It is understandable that the above-mentioned image processing method can be executed by the terminal device or the server device alone, or can be executed by the terminal device and the server device in cooperation.

[0051] For example, the above image processing method may be integrated into a client. After receiving an image processing request, a terminal device equipped with the client may provide computing power through its own hardware environment to execute the above image processing method.

[0052] For another example, the above image processing method may be integrated into a system platform. After receiving an image processing request, a server device equipped with the system platform may provide computing power through its own hardware environment to execute the above image processing method.

[0053] For another example, the image processing method can be divided into two tasks: acquiring a target image and processing the target image. The acquisition task can be integrated into the client and mounted on the terminal device. The processing task can be integrated into the server and mounted on the server device. The terminal device can initiate an image processing request to the server device after acquiring the target image. After receiving the image processing request, the server device can execute the method on the target image in response to the request.

[0054] The following description is made by taking an electronic device (hereinafter referred to as device) as an example.

[0055] The target image is an image including multiple buildings. For example, the target image may be a remote sensing image taken by an airplane, a drone, a satellite or other equipment.

[0056] In one case, when acquiring the target image, the above device can complete the input of the target image by interacting with the user. For example, the above device can provide the user with a window for inputting the target image to be processed through its interface, so that the user can input the image. The user can complete the input of the target image based on the window. After acquiring the target image, the above device can input the image into the image processing model for calculation.

[0057] In another case, the above device can directly obtain the remote sensing image output by the image acquisition system. For example, the above device can pre-establish a certain protocol with the remote sensing image acquisition system. When the image acquisition system generates the remote sensing image, it can be sent to the above device for image processing.

[0058] In some examples, the above-mentioned device can be equipped with an image processing model to perform the above-mentioned image processing.

[0059] Specifically, the above-mentioned device can use the image processing model to perform image processing on the above-mentioned target image to extract the roof area of ​​the building and the predicted offset between the roof and the base of the above-mentioned building from the above-mentioned target image, and according to the above-mentioned predicted offset, transform the above-mentioned roof area to obtain the base area corresponding to the above-mentioned building.

[0060] The above-mentioned image processing model is specifically an end-to-end image processing model for extracting the building base based on the target image. In some examples, the image processing model can be a pre-trained neural network model.

[0061] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of a base extraction process using an image processing model as shown in this application. Figure 2 As shown, the image processing model may include an image processing unit and a region transformation unit. The input of the image processing unit is the target image. The output of the image processing unit is the roof area of ​​the building and the predicted offset between the roof and the base. The input of the region transformation unit is the output of the image processing unit. The output of the region transformation unit is the base area.

[0062] The above-mentioned image processing unit can specifically be a model constructed based on a deep neural network for predicting the roof area and the offset between the roof and the base.

[0063] In some examples, the image processing unit may further include a building bounding box prediction sub-model and a roof area prediction sub-model.

[0064] The building bounding box prediction sub-model is used to extract the building bounding box in the target image and provide input for the roof area prediction sub-model. The building bounding box prediction sub-model can be a neural network model trained based on multiple training samples with labeled bounding boxes.

[0065] The roof area prediction sub-model is used to extract the building boundary box included in the target image, input the roof area prediction sub-model based on the building boundary box, and predict the roof area for each building. The roof area prediction sub-model can be a neural network model trained based on multiple training samples with roof areas labeled.

[0066] It can be understood that, at this point, the roof area prediction sub-model can be considered to include the building bounding box prediction sub-model.

[0067] See also Figure 3 , Figure 3 This is a schematic flow chart of a method for predicting a roof area by using an image processing unit as shown in the present application.

[0068] like Figure 3As shown, the image processing unit may include a building bounding box prediction submodel and a roof area prediction submodel. Among them, the above-mentioned building bounding box prediction submodel can be a regression model constructed based on RPN (Region Proposal Network). The above-mentioned roof area prediction submodel can be a regression model constructed based on regional feature extraction units such as Rol Align (Region of interest Align) network or Rol pooling (Region of interest pooling) network. The above-mentioned roof area prediction submodel includes the above-mentioned building bounding box prediction submodel. In an embodiment of the present application, Figure 3 This is only for illustrative purposes, and some intermediate layers such as convolutional layers, spatial pyramid layers, and fully connected layers may be added according to actual circumstances.

[0069] When predicting the roof of a building, the building bounding box prediction sub-model may be used to perform target detection on the target image to obtain the bounding box of the building.

[0070] For example, in some examples, the target image can be input into the building bounding box prediction sub-model to perform target detection to obtain a bounding box corresponding to the building.

[0071] After obtaining the bounding boxes of the buildings included in the target image, the roof area of ​​the building included in the bounding box may be determined based on the bounding box, the target image and the roof area prediction sub-model.

[0072] For example, in some examples, the above-mentioned bounding box and the above-mentioned target image can be input into the above-mentioned roof area prediction sub-model to obtain the roof area corresponding to the building included in the above-mentioned bounding box.

[0073] It can be understood that in order to increase roof area supervision when training the backbone network, thereby improving the accuracy of feature extraction of the backbone network, the target image used in the step of inputting the above-mentioned bounding box and the above-mentioned target image into the above-mentioned roof area prediction sub-model to obtain the roof area of ​​the building included in the above-mentioned bounding box can be a target feature map obtained after the target image is processed by the backbone network.

[0074] In the above scheme, the roof area prediction sub-model uses the output of the building bounding box prediction sub-model. Therefore, when training the building bounding box prediction sub-model, the roof area can be used as one of the true values ​​for supervised training, and the relevant information of the roof area is introduced as the supervision information, so that the building bounding box prediction sub-model can learn the relevant features required to predict the roof area, thereby improving the building prediction accuracy and further improving the base extraction accuracy.

[0075] The image processing unit may also be equipped with a sub-model for predicting the offset between the roof and the base (hereinafter referred to as the "offset prediction sub-model"), which is used to extract the predicted offset between the roof and the base of the building included in the target image (hereinafter referred to as the "predicted offset"). When predicting the offset, the target image may be input into the sub-model for predicting the offset between the roof and the base to predict the predicted offset.

[0076] In some examples, since the offset prediction sub-model and the roof area prediction sub-model both perform feature extraction on buildings included in the target image, in order to reduce the amount of model calculation, the offset prediction sub-model and the roof area prediction sub-model can share the building bounding box prediction sub-model.

[0077] See also Figure 4 , Figure 4 A flowchart of a method for predicting an offset by an image processing unit is shown in the present application.

[0078] like Figure 4 As shown, the image processing unit may include a building bounding box prediction submodel and an offset prediction submodel. Among them, the above-mentioned building bounding box prediction submodel can be a regression model constructed based on RPN (Region Proposal Network). The above-mentioned offset prediction submodel can be a regression model constructed based on regional feature extraction units such as Rol Align (Region of interest Align) network or Rol pooling (Region of interest pooling) network. The above-mentioned offset prediction submodel shares the above-mentioned building bounding box prediction submodel with the above-mentioned roof area prediction submodel. In an embodiment of the present application, Figure 4 This is only for illustrative purposes, and some intermediate layers such as accumulation layers, spatial pyramid layers, and fully connected layers can be added according to actual situations.

[0079] When predicting the offset, the predicted offset between the roof and the base of the building included in the bounding box may be determined based on the bounding box, the target image, and the offset prediction sub-model.

[0080] For example, in some examples, the bounding box of the building output by the building bounding box prediction sub-model and the above-mentioned target image can be input into the above-mentioned offset prediction sub-model between the roof and the base to obtain the predicted offset between the roof and the base of the building included in the above-mentioned bounding box.

[0081] It can be understood that in order to increase the prediction offset supervision when training the backbone network, thereby improving the accuracy of feature extraction of the backbone network, the target image used in the step of inputting the bounding box and the above-mentioned target image into the above-mentioned roof-base offset prediction sub-model to obtain the predicted offset between the roof and the base of the building included in the above-mentioned bounding box can be a target feature map obtained after the target image is processed by the backbone network.

[0082] In the above scheme, on the one hand, the input of the offset prediction sub-model is the output of the building bounding box prediction sub-model, that is, the offset prediction sub-model includes the building bounding box prediction sub-model. Therefore, when training the building bounding box prediction sub-model, the predicted offset can be used as one of the true values ​​for supervised training, and the information related to the predicted offset is introduced as the supervisory information, so that the building bounding box prediction sub-model can learn the relevant features required for the predicted offset, thereby improving the building prediction accuracy and further improving the base extraction accuracy.

[0083] On the other hand, the offset prediction sub-model and the roof area prediction sub-model share the building bounding box prediction sub-model to reduce the amount of model calculation.

[0084] In some embodiments, the roof area prediction sub-model and the roof-to-base offset prediction sub-model may include the same regional feature extraction unit.

[0085] Among them, the above-mentioned regional feature extraction unit can be a regional feature extraction unit constructed based on a Rol Align (Region of interest Align, region of interest feature alignment) unit or a Rol pooling (Region of interest pooling, region of interest feature pooling) unit.

[0086] See also Figure 5 , Figure 5 A flowchart of a method for predicting offset and roof area through an image processing unit shown in the present application.

[0087] like Figure 5 As shown, the above processing flow includes two sub-branches, wherein the first sub-branch is a roof area prediction sub-branch, and the other sub-branch is an offset prediction sub-branch.

[0088] The above two sub-branches can share a regional feature extraction unit. When performing offset prediction and roof area prediction, S502 can be executed first to determine the building features corresponding to the buildings included in the above bounding box based on the above bounding box, the above target image (which can be a target feature map after the target image is processed by the backbone network) and the above regional feature extraction unit.

[0089] In the embodiment of the present application, in the present application, the above target image may include multiple buildings. It is understandable that, in the above situation, the solution recorded in the present application can extract the bounding boxes of multiple buildings respectively, and perform the above S502 step for each building bounding box. The number of buildings included in the target image is not limited in the present application.

[0090] After obtaining the building features corresponding to the building in the frame, in the roof area prediction sub-branch, S504 can be executed to perform a first convolution processing on the building features using the first convolution processing unit included in the roof area prediction sub-model to obtain the roof area of ​​the building.

[0091] When performing the first convolution process, the building features can be input into Figure 5 The roof area is obtained by calculation in the first convolution processing unit shown.

[0092] After obtaining the building features corresponding to the building in the frame, S506 can also be executed in the offset prediction sub-branch, and the second convolution processing unit included in the above-mentioned offset prediction sub-model is used to perform a second convolution processing on the above-mentioned building features to obtain the predicted offset between the roof and the base of the building included in the above-mentioned boundary box.

[0093] When performing the second convolution process, the building features may be input into Figure 5 The roof area is obtained by calculation in the second convolution processing unit shown.

[0094] In the embodiments of the present application, on the one hand, the present application does not limit the structure of the first convolution unit and the second convolution unit. The structure of the first convolution unit and the second convolution unit can be set according to actual needs. Figure 5 The model structure shown is only a schematic diagram. Figure 5 Conventional structures such as upsampling, downsampling, and pooling operations are not shown. The conventional structures can be set according to actual situations.

[0095] In the above scheme, the above roof area prediction sub-model and the above roof-to-base offset prediction sub-model may include the same regional feature extraction unit. Therefore, on the one hand, when training the above regional feature extraction unit, the predicted offset and the roof area may be used as one of the true values ​​for supervised training, and relevant information such as the predicted offset and the roof area may be introduced as supervisory information, so that the regional feature extraction unit can learn the predicted offset and the relevant features required for the roof area, thereby improving the accuracy of building feature extraction and further improving the accuracy of base extraction. On the other hand, the model structure is streamlined and the amount of model calculation is reduced.

[0096] Please continue to see Figure 2 After the roof area and the predicted offset are obtained through the image processing unit, the obtained roof area and the predicted offset can be input into the area change unit to obtain the base area.

[0097] The area change unit may be a mapping unit. For example, a mapping function y=f(x1, x2) may be constructed. Where x1 is the predicted offset between the roof area and the base area, and x2 is the roof area. y is the base area. f is a mapping function for obtaining the base area from the predicted offset and the roof area. By constructing the above mapping function, the base area may be obtained based on the predicted offset and the roof area.

[0098] In the embodiment of the present application, the above-mentioned predicted offset may include a rotation predicted offset and a translation predicted offset. The specific meaning of the predicted offset is not limited in the present application. The following is an example of the roof and the base being obtained by only translation.

[0099] In some embodiments, in order to improve the base extraction accuracy, when the roof area is transformed according to the predicted offset to obtain the base area corresponding to the building, the base features corresponding to the base area can be transformed according to the predicted offset and the roof features corresponding to the roof area. After obtaining the base features, the base features are further refined to obtain the base area.

[0100] In the above embodiment, when performing the translation transformation, the translation transformation is not performed on the roof area, but directly on the roof features. Since the roof area is obtained based on the roof features through convolution and / or upsampling, the above operation can avoid introducing other errors introduced by the process of convolution and / or upsampling of the roof features to the roof area during the translation transformation, thereby improving the base lifting accuracy.

[0101] In the embodiments of this application, please refer to Figure 5 , the roof area is obtained based on the building features, and the building features are obtained from the region extraction network. Since the supervised training is performed with the roof area as the true value when training the region extraction network, the feature response in the roof area of ​​the above-mentioned building features will be very high. At this time, the building features can be used directly as the above-mentioned roof features. Of course, in this application, the above-mentioned building features can also be convolved multiple times to obtain the above-mentioned roof features, and then the steps are performed. The following is an example of using the building features directly as the above-mentioned roof features.

[0102] In some embodiments, the region transformation unit may be a unit constructed based on a neural network. That is, the image processing model further includes a base region prediction sub-model constructed based on a neural network.

[0103] At this time, when the roof area is transformed according to the predicted offset to obtain the base area corresponding to the building, the base area corresponding to the building can be determined based on the predicted offset, the building features corresponding to the building and the base area prediction sub-model.

[0104] For example, in some examples, the predicted offset and the building features corresponding to the building may be input into the base area prediction sub-model to obtain the base area corresponding to the building.

[0105] In some embodiments, the base area prediction sub-model may include a space transformation network, wherein the space transformation parameters corresponding to the space transformation network include parameters determined based on the prediction offset.

[0106] See also Figure 6 , Figure 6 A schematic flow chart of a method for performing base prediction using a base area prediction sub-model as shown in the present application.

[0107] like Figure 6 As shown, after obtaining the building features corresponding to the roof area, the spatial transformation network included in the above-mentioned base area prediction sub-model can be used to perform a translation transformation on the building features corresponding to the above-mentioned roof area to obtain the predicted offset of the base features corresponding to the above-mentioned building.

[0108] After obtaining the base features, the base features can be input into the multiple convolutional layers ( Figure 6 The third convolution unit shown in the figure performs the third convolution processing to obtain the base area corresponding to the above-mentioned building.

[0109] In the embodiments of the present application, on the one hand, the present application does not limit the structure of the third convolution unit. The structure of the third convolution unit can be set according to actual needs. Figure 6 The model structure shown is only a schematic diagram. Figure 6 Conventional structures such as upsampling, downsampling, and pooling operations are not shown. The above conventional structures can be set according to actual situations.

[0110] The above-mentioned spatial transformation network may include a sampler (Sampler) constructed based on an interpolation method, wherein the above-mentioned sampler includes a sampling grid prediction offset (Grid generator) constructed based on the above-mentioned prediction offset.

[0111] The sampling grid is specifically a transformation function constructed based on the predicted offset. The sampling grid can indicate the mapping relationship between each pixel point included in the roof feature and each pixel point included in the base feature. For example, based on the sampling grid, it can be determined which pixels included in the roof feature constitute a certain pixel point corresponding to the base feature.

[0112] The above-mentioned sampler is specifically a mapping unit constructed based on an interpolation method. The above-mentioned sampler can map the feature position (feature position) and the feature value (feature score) of the original feature (building feature) and the feature after translation transformation (base feature) based on the interpolation method, so as to obtain the base feature. Among them, the above-mentioned interpolation method can be bilinear interpolation, linear difference, parabolic interpolation, etc. In this application, the bilinear interpolation method can be used.

[0113] At this time, when the building features corresponding to the above-mentioned roof area and the above-mentioned predicted offset are input into the spatial transformation network included in the above-mentioned base area prediction sub-model for translation transformation, and the base features corresponding to the above-mentioned building are obtained, the above-mentioned sampler can be used to take the pixel points corresponding to the base features as the current pixel points in turn according to the coordinate information of each pixel point of the base features, and determine the pixel points corresponding to the above-mentioned current pixel points among the pixel points included in the roof area through the above-mentioned sampling grid, and perform translation transformation on the determined pixel points based on the interpolation method to obtain the pixel value corresponding to the above-mentioned current pixel point.

[0114] In the above embodiment, on the one hand, since the above base area prediction sub-model includes a spatial transformation network that can be back-propagated and a third convolutional unit, compared with the method of mapping building features to base features through non-neural network forms such as RT transformation (rotation, translation transformation), the base area (base feature) can be used as the true value to perform supervised training on the base area prediction sub-model (including the spatial transformation network and the third convolutional layer), thereby introducing the prediction error between the base area predicted based on the roof area and the true value of the base area as supervision information, so that the shared network between the base area prediction sub-model and the offset prediction sub-model can be trained based on the above error, thereby improving the accuracy of offset prediction and further improving the accuracy of base area prediction.

[0115] On the other hand, since the base area prediction sub-model, the roof area prediction sub-model, and the offset prediction sub-model share the building features output by the regional feature extraction unit, the supervision information can be shared during the training of each sub-model, thereby accelerating the model convergence and improving the performance of each sub-model.

[0116] In some embodiments, in order to further improve the accuracy of base area prediction, a roof outline that is closer to reality than the edges included in the roof area extracted by the roof area prediction sub-model can be extracted from the target image, and the roof area obtained by the base prediction sub-model can be corrected based on the above roof outline to obtain the final base area.

[0117] See also Figure 7 , Figure 7 This is a flow chart of a final base area prediction method shown in this application.

[0118] like Figure 7 As shown, after obtaining the building features, S702 may be executed to perform contour regression on the building features using the roof contour prediction sub-model to determine the roof contour of the building.

[0119] In some examples, the building features may be input into the roof profile prediction sub-model to obtain the roof profile of the building.

[0120] The roof outline prediction sub-model may be a model constructed based on a wireframe parsing network. The wireframe parsing model can be used to extract a relatively accurate roof outline from a target image.

[0121] See also Figure 8 , Figure 8 This is a schematic flow chart of a method for predicting roof contours through a bounding box analysis network shown in the present application.

[0122] like Figure 8 As shown, after the building features are obtained, S802 may be executed to extract multiple connection points from the building features.

[0123] When executing S802, the building features can be input into the fourth convolution unit for multiple convolution operations and smoothing to obtain a heat map including multiple connection points. When training the fourth convolution unit, a plurality of training samples can be obtained by annotating the true value (i.e., when the pixel block includes a connection point, it is marked as 1, otherwise it is marked as 0) for each pixel block in the heat map (for example, the resolution of the heat map is 14*14, and the heat map includes 196 pixel blocks), and then the fourth convolution unit can be trained based on the training samples with the cross entropy loss information as the objective function, so that the fourth convolution unit can predict the connection points for each pixel block in the target heat map.

[0124] After obtaining a plurality of connection points, S804 may be executed to perform line segment sampling, that is, at least some of the plurality of connection points are combined in pairs to obtain a plurality of line segments.

[0125] After obtaining multiple line segments, S806 can be executed to perform line segment verification, that is, predict the multiple line segments to obtain the prediction score corresponding to each line segment, and filter out the line segments whose prediction score is greater than a preset threshold, wherein the prediction score is used to indicate the probability that the line segment corresponding to the score belongs to the roof outline.

[0126] The above-mentioned preset threshold may be a threshold set based on experience.

[0127] When executing S806, the above-mentioned multiple line segments may be input into a line segment verification network to obtain a prediction score corresponding to each line segment, and then line segments having a prediction score greater than the above-mentioned preset threshold may be screened out.

[0128] The line segment verification network may include a line segment feature extraction network and a classification score prediction network. The line segment feature extraction network is used to extract line segment features corresponding to the constructed line segment from the building features. After the line segment features are obtained, the classification score corresponding to the line segment can be predicted based on the line segment features.

[0129] In the embodiment of the present application, in order to improve the prediction accuracy of the line segment verification network, the same number of positive samples and negative samples can be set when constructing the training samples, so that the above-mentioned line segment verification network can learn the line segment features corresponding to the positive samples and the negative samples respectively, so as to achieve accurate prediction of the line segment classification score. Among them, the positive sample refers to the line segment pair with high similarity in the image. The negative sample refers to the line segment pair with low similarity.

[0130] After filtering out the line segments whose classification scores are greater than the preset threshold, S808 may be executed to combine the line segments whose corresponding prediction scores are greater than the preset threshold among the plurality of line segments to obtain the roof outline of the building.

[0131] After the roof outline is obtained, S704 may be executed to perform a translation transformation on the roof outline according to the predicted offset to obtain the base outline of the building.

[0132] In some embodiments, the above translation transformation may be mapping the roof contour to the base contour through a preset transformation function (eg, RT transformation).

[0133] In some embodiments, in order to improve the accuracy of the change, the roof contour can be translated and transformed through a spatial transformation network to obtain the base contour.

[0134] See also Fig. 9 , Fig. 9 A schematic diagram of a base area prediction process shown in this application.

[0135] like Fig. 9As shown, after obtaining the roof area, the roof area and the predicted offset can be input into the spatial transformation network for translation transformation to obtain the base contour. The relevant introduction of the spatial transformation network can be found in the above content and will not be described in detail here.

[0136] In the embodiment of the present application, the space transformation network used to predict the base contour and the base area can be the same network or different networks. Of course, it is understandable that in order to reduce the amount of calculation, the space transformation network used to predict the base contour and the base area can be the same network.

[0137] After the base contour is obtained, S706 may be executed to adjust the base area based on the base contour to obtain a final base area corresponding to the building.

[0138] In this step, the base contour can be fused with the preliminarily predicted base area, and the edges corresponding to the preliminarily predicted base area can be corrected by fusion technology to obtain a base contour that is more in line with the actual situation. After that, the base contour can be fused with the original target image to obtain the final base area. The image fusion process can be referred to in the relevant technology and will not be described in detail here.

[0139] In the above scheme, the roof outline is first obtained from the target image using the bounding box analysis network. Then, the accurate base outline is obtained based on the roof outline. Finally, the initially predicted base area is corrected based on the base outline to obtain the final base area. Since the above roof outline is more accurate and more in line with the real building roof outline than the roof area predicted by the roof area prediction sub-model, the final base area predicted based on the above base outline correction will be more accurate.

[0140] The above is an introduction to the building base prediction solution shown in this application. The following introduces the training method of the image processing model.

[0141] In the present application, the image processing model used in the building base prediction scheme may include a building bounding box prediction sub-model, a roof area prediction sub-model, an offset prediction sub-model, a roof outline prediction sub-model, and a base area prediction sub-model.

[0142] In order to improve the prediction accuracy of the image processing model for the base area and the generalization ability of the model, a multi-task joint training method is adopted when training the image processing model.

[0143] See also Fig.10 , Fig.10 A task-model correspondence diagram shown in this application.

[0144] like Fig.10As shown, Fig. 9 By decomposing the base area prediction process shown, it can be obtained that the base prediction needs to include at least a building bounding box prediction subtask, a roof area prediction subtask, an offset prediction subtask between the roof and the base (hereinafter referred to as the "offset prediction subtask"), a roof contour prediction subtask, and a base area prediction subtask. Among them, the above-mentioned building bounding box prediction subtask corresponds to a building bounding box prediction submodel. The above-mentioned roof area prediction subtask corresponds to a roof area prediction submodel. The above-mentioned offset prediction subtask corresponds to an offset prediction submodel. The above-mentioned roof contour prediction subtask corresponds to a roof contour prediction submodel. The above-mentioned base area prediction subtask corresponds to a base area prediction submodel.

[0145] The following introduction Fig.10 The training process of the image processing model is shown in Figure 2. Fig.11 , Fig.11 A method flow chart of an image processing model training method shown in this application.

[0146] Among them, the above-mentioned image processing model includes a building bounding box prediction sub-model, a roof area prediction sub-model, an offset prediction sub-model, a roof contour prediction sub-model and a base area prediction sub-model.

[0147] like Fig.11 As shown, the method includes:

[0148] S1102, obtaining a plurality of training samples including annotation information; wherein the annotation information includes a building boundary box, a building roof area, a building roof outline, a predicted offset between the building roof and the base, and a building base area.

[0149] When performing this step, the original image can be annotated with true values ​​by manual annotation or machine-assisted annotation. For example, after acquiring the original image, the image annotation software can be used to annotate the building bounding box, building roof area, building roof outline, predicted offset between the building roof and the base, and building base area included in the original image to obtain multiple training samples. In the embodiment of the present application, one-hot encoding and other methods can be used for encoding when encoding the training samples, and the present application does not limit the specific encoding method.

[0150] S1104: Construct joint learning loss information based on the loss information corresponding to each sub-model included in the above-mentioned image processing model.

[0151] When executing this step, the loss information corresponding to each sub-model can be determined first. In order to improve the prediction accuracy of the sub-model, in this application, the loss information corresponding to the above-mentioned building boundary box prediction sub-model is Smooth L1; the loss information corresponding to the above-mentioned roof area prediction sub-model, the above-mentioned roof outline prediction sub-model, and the above-mentioned base area prediction sub-model is cross entropy loss information; the loss information corresponding to the above-mentioned offset prediction sub-model between the roof and the base is MSE loss information.

[0152] After determining the loss information corresponding to each sub-model, the joint learning loss information can be constructed based on the loss information corresponding to each sub-model included in the above image processing model. For example, the loss information corresponding to each sub-model can be added to obtain the above joint learning loss information.

[0153] In the embodiment of the present application, a regularization term may be added to the above-mentioned joint learning loss information in the present application, which is not particularly limited here.

[0154] After determining the joint learning loss information and the training samples, S1106 may be executed to jointly train the sub-models included in the image processing model based on the joint learning loss information and the training samples until the sub-models converge.

[0155] When training the model, hyperparameters such as learning rate, number of training cycles, etc. may be specified first. After the hyperparameters are determined, supervised training may be performed on the image processing model based on the training samples labeled with true values.

[0156] In the supervised training process, after forward propagation is performed on the image processing model to obtain the calculation result, the error between the true value and the above calculation result can be evaluated based on the constructed joint learning loss information. After obtaining the error, the stochastic gradient descent method can be used to determine the descent gradient. After determining the descent gradient, the model parameters corresponding to the above image processing model can be updated based on back propagation. Repeat the above process until the above sub-models converge. In the embodiment of the present application, the conditions for convergence of the multi-model are not specifically limited in the present application.

[0157] When training the image processing model, due to the use of a supervised joint training method, the five sub-models included in the image processing can be trained simultaneously, so that the sub-models can both constrain and promote each other during the training process, thereby improving the convergence efficiency of the image processing model on the one hand; on the other hand, the backbone network shared by the sub-models can extract features that are more beneficial to the prediction of the base area, thereby improving the model prediction accuracy.

[0158] Corresponding to any of the above embodiments, the present application also proposes an image processing device.

[0159] See also Fig.12 , Fig.12 This is a schematic diagram of an image processing device shown in the present application.

[0160] like Fig.12 As shown, the above-mentioned device 1200 includes:

[0161] The acquisition module 1210 is used to acquire a target image containing a building; the extraction module 1220 is used to extract the roof area of ​​the building from the target image, and determine the predicted offset between the roof and the base of the building based on the target image; the transformation module 1230 is used to transform the roof area according to the predicted offset to obtain the base area of ​​the building.

[0162] In some of the examples shown, the bounding box determination module is used to perform target detection on the target image using the building bounding box prediction sub-model to obtain the bounding box of the building; the extraction module 1220 includes: a roof area determination module, which is used to determine the roof area of ​​the building included in the bounding box based on the bounding box, the target image and the roof area prediction sub-model; an offset determination module, which is used to determine the predicted offset between the roof and the base of the building included in the bounding box based on the bounding box, the target image and the offset prediction sub-model.

[0163] In some of the examples shown, the roof area prediction submodel and the offset prediction submodel share the same regional feature extraction unit; the device also includes: a building feature determination module, which is used to determine the building features corresponding to the building included in the bounding box based on the bounding box, the target image and the regional feature extraction unit; the roof area determination module includes: a first convolution processing module, which is used to perform a first convolution processing on the building features using the first convolution processing unit included in the roof area prediction submodel to obtain the roof area of ​​the building; the offset determination module includes: a second convolution processing module, which is used to perform a second convolution processing on the building features using the second convolution processing unit included in the offset prediction submodel to obtain the predicted offset between the roof and the base of the building included in the bounding box. In some of the examples shown, the transformation module 1230 is specifically used to determine the base area corresponding to the building based on the predicted offset, the building features corresponding to the building and the base area prediction submodel.

[0164] In some of the examples shown, the transformation module 1230 includes: a first translation transformation module 1230, which is used to use the spatial transformation network included in the base area prediction sub-model to perform a translation transformation on the building features corresponding to the roof area to obtain the base features corresponding to the building; wherein the spatial transformation parameters corresponding to the spatial transformation network include parameters determined based on the prediction offset; and a third convolution processing module, which is used to perform a third convolution processing on the base features using the base area prediction sub-model to obtain the base area corresponding to the building.

[0165] In some of the examples shown, the above-mentioned spatial transformation network includes a sampler constructed based on an interpolation method; wherein the above-mentioned sampler includes a sampling grid constructed based on the above-mentioned predicted offset; the above-mentioned first translation transformation module 1230 is specifically used to: use the above-mentioned sampler, according to the coordinate information of each pixel point of the base feature, take each pixel point corresponding to the base feature as the current pixel point in turn, determine the pixel point corresponding to the above-mentioned current pixel point among the pixels included in the roof area through the above-mentioned sampling grid, and perform a translation transformation on the determined pixel point based on the interpolation method to obtain the pixel value corresponding to the above-mentioned current pixel point.

[0166] In some of the examples shown, the roof area prediction submodel and the offset prediction submodel share the same area feature extraction unit, and also include a roof contour prediction submodel; the above-mentioned device also includes: a contour regression module, which is used to use the roof contour prediction submodel to perform contour regression on the above-mentioned building features to determine the roof contour of the above-mentioned building; a second translation transformation module 1230, which is used to perform a translation transformation on the above-mentioned roof contour according to the above-mentioned predicted offset to obtain the base contour of the above-mentioned building; a final base area determination module, which is used to adjust the above-mentioned base area based on the above-mentioned base contour to obtain the final base area corresponding to the above-mentioned building.

[0167] In some of the examples shown, the above-mentioned contour regression module is specifically used to: extract multiple connection points from the above-mentioned building features; combine at least some of the above-mentioned multiple connection points in pairs to obtain multiple line segments; predict the above-mentioned multiple line segments to obtain prediction scores corresponding to each line segment; wherein the above-mentioned prediction scores are used to indicate the probability that the line segment corresponding to the score belongs to the roof contour; combine the line segments whose corresponding prediction scores are greater than a preset threshold among the above-mentioned multiple line segments to obtain the roof contour of the above-mentioned building.

[0168] In some of the examples shown, the extraction module 1220 is specifically used to: perform image processing on the target image using an image processing model; wherein the image processing model includes a building bounding box prediction sub-model, a roof area prediction sub-model, an offset prediction sub-model, a roof outline prediction sub-model, and a base area prediction sub-model.

[0169] In some examples shown, the training device 1300 corresponding to the training method of the above-mentioned image processing model includes:

[0170] The training sample acquisition module 1310 is used to acquire a plurality of training samples including annotation information; wherein the annotation information includes a building boundary box, a building roof area, a building roof outline, a predicted offset between the building roof and the base, and a building base area;

[0171] A loss information determination module 1320, configured to construct joint learning loss information based on the loss information corresponding to each sub-model included in the above-mentioned image processing model;

[0172] The joint training module 1330 is used to jointly train the sub-models included in the above-mentioned image processing model based on the above-mentioned joint learning loss information and the above-mentioned training samples until the above-mentioned sub-models converge.

[0173] The embodiment of the image processing device shown in the present application can be applied to an electronic device. Accordingly, the present application discloses an electronic device, which may include: a processor.

[0174] Memory used to store processor-executable instructions.

[0175] The processor is configured to call the executable instructions stored in the memory to implement the image processing method as shown in any of the above embodiments.

[0176] See also Fig.13 , Fig.13 This is a hardware structure diagram of an electronic device shown in this application.

[0177] like Fig.13 As shown, the electronic device may include a processor for executing instructions, a network interface for network connection, a memory for storing operating data for the processor, and a non-volatile memory for storing instructions corresponding to the image processing device.

[0178] The image processing device embodiment can be implemented by software, hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the electronic device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, in addition to Fig.13 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device in the embodiment is located may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0179] It is understandable that, in order to improve the processing speed, the corresponding instructions of the image processing device can also be directly stored in the memory, which is not limited here. The present application proposes a computer-readable storage medium, the storage medium stores a computer program, and the computer program is used to execute the image processing method shown in any of the above embodiments.

[0180] Those skilled in the art will appreciate that one or more embodiments of the present application may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (which may include but are not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0181] The term "and / or" in the present application means at least one of the two. For example, "A and / or B" may include three options: A, B, and "A and B".

[0182] Each embodiment in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0183] The above describes a specific embodiment of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0184] The embodiments of the subject matter and functional operations described in this application can be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware that may include the structures disclosed in this application and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this application can be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0185] The processes and logic flows described in this application can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The above processes and logic flows can also be performed by special-purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special-purpose logic circuits.

[0186] A computer suitable for executing a computer program may include, for example, a general and / or special purpose microprocessor, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer may include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, a computer does not necessarily have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0187] Computer readable media suitable for storing computer program instructions and data may include all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0188] Although the application includes many specific implementation details, these should not be interpreted as limiting any disclosed scope or the scope of protection claimed, but are mainly used to describe the features of specific disclosed specific embodiments. Certain features described in multiple embodiments in the application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented with any suitable sub-combination. In addition, although the feature can work as mentioned above in some combinations and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0189] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0190] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0191] The above are only preferred embodiments of one or more embodiments of the present application and are not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of one or more embodiments of the present application.

Claims

1. An image processing method, It is characterized in that The method comprises: Get a target image containing a building; Performing target detection on the target image using a building bounding box prediction sub-model to obtain a bounding box of the building; Determine building features corresponding to the buildings included in the bounding box based on the bounding box, the target image and a regional feature extraction unit; Extracting the roof area of ​​the building from the target image, and determining a predicted offset between the roof and the base of the building based on the target image; transforming the roof area according to the predicted offset to obtain the base area of ​​the building; Performing a profile regression on the building features using a roof profile prediction sub-model to determine the roof profile of the building; According to the predicted offset, the roof profile is translated to obtain the base profile of the building; The base area is adjusted based on the base contour to obtain a final base area corresponding to the building.

2. The method according to claim 1, It is characterized in that The step of extracting a roof area of ​​a building from the target image comprises: Determine a roof area of ​​the building included in the bounding box based on the bounding box, the target image, and the roof area prediction sub-model; The step of determining a predicted offset between a roof and a base of the building based on the target image comprises: Based on the bounding box, the target image, and the offset prediction sub-model, a predicted offset between a roof and a base of a building included in the bounding box is determined.

3. The method according to claim 2, It is characterized in that The roof area prediction sub-model and the offset prediction sub-model share the same area feature extraction unit; The determining the roof area of ​​the building included in the bounding box based on the bounding box, the target image and the roof area prediction sub-model comprises: performing a first convolution processing on the building feature using a first convolution processing unit included in the roof area prediction sub-model to obtain the roof area of ​​the building; The method of determining the predicted offset between the roof and the base of the building included in the bounding box based on the bounding box, the target image and the offset prediction sub-model includes: performing a second convolution processing on the building feature using a second convolution processing unit included in the offset prediction sub-model to obtain the predicted offset between the roof and the base of the building included in the bounding box.

4. The method according to claim 3, It is characterized in that The step of transforming the roof area according to the predicted offset to obtain a base area corresponding to the building includes: determining the base area corresponding to the building based on the predicted offset, building features corresponding to the building, and a base area prediction sub-model.

5. The method according to claim 4, It is characterized in that The determining the base area corresponding to the building based on the predicted offset, the building feature corresponding to the building, and the base area prediction sub-model includes: Using the spatial transformation network included in the base area prediction submodel, the building features corresponding to the roof area are translated and transformed to obtain the base features corresponding to the building; wherein the spatial transformation parameters corresponding to the spatial transformation network include parameters determined based on the predicted offset; The base area prediction sub-model is used to perform a third convolution process on the base feature to obtain a base area corresponding to the building.

6. The method according to claim 5, It is characterized in that The spatial transformation network includes a sampler constructed based on an interpolation method; wherein the sampler includes a sampling grid constructed based on the predicted offset; The utilizing the spatial transformation network included in the base area prediction sub-model to perform translation transformation on the building features corresponding to the roof area to obtain the base features corresponding to the building includes: Utilizing the sampler, according to the coordinate information of each pixel point of the base feature, each pixel point corresponding to the base feature is taken as the current pixel point in turn, and the pixel point corresponding to the current pixel point among the pixels included in the roof area is determined through the sampling grid, and the determined pixel point is translated based on the interpolation method to obtain the pixel value corresponding to the current pixel point.

7. The method according to any one of claims 3 to 6, It is characterized in that The roof area prediction sub-model and the offset prediction sub-model share the same area feature extraction unit, and also include a roof outline prediction sub-model.

8. The method according to claim 7, It is characterized in that The step of performing contour regression on the building features using the roof contour prediction sub-model to determine the roof contour of the building includes: extracting a plurality of connection points from the building features; Combining at least some of the multiple connection points in pairs to obtain multiple line segments; Predicting the multiple line segments to obtain a prediction score corresponding to each line segment; wherein the prediction score is used to indicate the probability that the line segment corresponding to the score belongs to the roof outline; The line segments whose corresponding prediction scores are greater than a preset threshold value among the multiple line segments are combined to obtain the roof outline of the building.

9. The method according to any one of claims 1 to 6 and claim 8, It is characterized in that The method is implemented using an image processing model; wherein the image processing model includes a building bounding box prediction sub-model, a roof area prediction sub-model, an offset prediction sub-model, a roof contour prediction sub-model and a base area prediction sub-model.

10. The method according to claim 9, It is characterized in that The training method of the image processing model includes: Acquire a plurality of training samples including annotation information; wherein the annotation information includes a building bounding box, a building roof area, a building roof outline, a predicted offset between the building roof and the base, and a building base area; Constructing joint learning loss information based on the loss information corresponding to each sub-model included in the image processing model; Based on the joint learning loss information and the training samples, each sub-model included in the image processing model is jointly trained until each sub-model converges.

11. An image processing device, It is characterized in that The device comprises: An acquisition module, used for acquiring a target image containing a building; A bounding box determination module, used to perform target detection on the target image using a building bounding box prediction sub-model to obtain a bounding box of the building; A building feature determination module, configured to determine building features corresponding to the buildings included in the bounding box based on the bounding box, the target image and the regional feature extraction unit; an extraction module, configured to extract a roof area of ​​the building from the target image, and to determine a predicted offset between the roof and the base of the building based on the target image; A transformation module, configured to transform the roof area according to the predicted offset to obtain a base area of ​​the building; A profile regression module, used to perform profile regression on the building features using a roof profile prediction sub-model to determine the roof profile of the building; A second translation transformation module is used to perform a translation transformation on the roof outline according to the predicted offset to obtain a base outline of the building; The final base area determination module is used to adjust the base area based on the base contour to obtain a final base area corresponding to the building.

12. An electronic device, It is characterized in that The device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to call the executable instructions stored in the memory to implement the image processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, It is characterized in that The storage medium stores a computer program for executing the image processing method according to any one of claims 1 to 10.

Images