Image processing method, device, equipment and storage medium

Abstract

The present application proposes an image processing method, device, equipment and storage medium. The above method includes obtaining a target image containing a building. Performing image processing on the above target image to determine the roof area of ​​the above building, the side bottom edge and the side top edge of the above building, and the offset angle between the roof and the base of the above building. According to the above offset angle, determine the offset between the side bottom edge and the side top edge. According to the above offset, transform the roof contour corresponding to the above roof area to obtain the base contour.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] At present, in the field of image processing, it is usually necessary to predict the outline of buildings in images for urban planning, map drawing, building change detection, residential area management, etc. The more important tasks in building prediction include the prediction of building bases and building heights.

[0003] However, since the images captured are usually non-orthophoto remote sensing images taken by satellites or aircraft, the base of the building in the image may be partially blocked, resulting in its visual features not being obvious, thus affecting the prediction accuracy of the building base. Summary of the invention

[0004] In view of this, the present application discloses at least one image processing method, the method comprising:

[0005] Get a target image containing a building;

[0006] Performing image processing on the target image to determine the roof area of ​​the building, the side bottom edge and the side top edge of the building, and the offset angle between the roof and the base of the building;

[0007] Determine the offset between the bottom edge of the side surface and the top edge of the side surface according to the offset angle;

[0008] According to the above offset, the roof contour corresponding to the above roof area is transformed to obtain the base contour.

[0009] In some examples shown, the method further includes:

[0010] Based on the offset and a predetermined ratio between the building height and the offset, the building height corresponding to the building is determined.

[0011] In some examples shown, the method further includes:

[0012] Performing image processing on the target image to determine the edge directions corresponding to the pixel points included in the roof outline of the building;

[0013] The above method further includes:

[0014] Based on the edge direction, the roof outline is regularized to obtain a roof polygon corresponding to the building;

[0015] The step of transforming the roof contour corresponding to the roof area according to the offset to obtain the base contour includes: transforming the roof polygon according to the offset to obtain the base contour.

[0016] In some examples shown, the roof outline is regularized based on the edge direction to obtain the roof polygon corresponding to the building, including:

[0017] Taking any pixel point among the pixels included in the roof outline as a target pixel point, determining the difference between the edge direction corresponding to the target pixel point and the edge direction corresponding to the adjacent pixel point of the target pixel point;

[0018] When the difference between the edge direction corresponding to the target pixel point and the edge direction corresponding to the adjacent pixel point reaches a first preset threshold, the target pixel point is determined as the vertex of the roof polygon corresponding to the building;

[0019] Based on the determined vertices of the roof polygon, a roof polygon corresponding to the building is obtained.

[0020] In some examples shown, the method further includes:

[0021] Divide the preset angle into N angle intervals; where N is a positive integer;

[0022] The above-mentioned determining the difference between the edge direction corresponding to the target pixel point and the edge direction corresponding to the adjacent pixel points of the target pixel point includes:

[0023] Determine the first angle interval to which the edge direction corresponding to the target pixel point belongs;

[0024] Determine a second angle interval to which the edge direction corresponding to the adjacent pixel points of the target pixel point belongs;

[0025] The difference between the identification value corresponding to the first angle interval and the identification value corresponding to the second angle interval is determined as the difference between the edge direction corresponding to the target pixel point and the edge direction corresponding to the adjacent pixel point of the target pixel point.

[0026] In some examples shown, N is a positive integer less than or equal to the second preset threshold.

[0027] In some examples shown, the method further includes:

[0028] Based on the vertex correction model, the vertices of the above-mentioned roof polygon are corrected to obtain a corrected roof polygon; wherein the above-mentioned vertex correction model is a model determined based on a graph neural network.

[0029] In some examples shown, determining the offset between the bottom edge of the side surface and the top edge of the side surface according to the offset angle includes:

[0030] Determine the amount of change in the position of the bottom edge of the side surface when the bottom edge of the side surface moves to the top edge of the side surface in the direction of the offset angle, and use the amount of change in the position of the bottom edge of the side surface as the offset.

[0031] In some examples shown, determining the change in the position of the bottom edge of the side surface when the bottom edge of the side surface moves to the top edge of the side surface in the direction of the offset angle, and using the change in the position of the bottom edge of the side surface as the offset, includes:

[0032] Based on a preset border corresponding to the side top edge, cropping the side top edge probability map corresponding to the side top edge to obtain a first cropping result;

[0033] According to a preset step length and a preset maximum offset, the side bottom edge is moved multiple times in the direction of the offset angle, and after each movement, the side bottom edge probability map corresponding to the side bottom edge is cropped based on the preset border to obtain multiple second cropping results;

[0034] Among the plurality of second cropping results, a target cropping result matching the first cropping result is determined, and a change in the position of the bottom edge of the side surface when the target cropping result is obtained is determined as the offset.

[0035] In some examples shown, the method further includes:

[0036] Determine the circumscribed border corresponding to the top edge of the side surface as the preset border; or,

[0037] The circumscribed frame corresponding to the combined edge obtained by combining the multiple side top edges included in the above roof outline is determined as the above preset frame.

[0038] In some examples shown, performing image processing on the target image to determine the roof area of ​​the building, the side bottom edge and the side top edge of the building, and the offset angle between the roof and the base of the building include:

[0039] Performing image processing on the target image using an image processing model to determine the roof area of ​​the building, the side bottom edge and the side top edge of the building, and the offset angle between the roof and the base of the building;

[0040] The above-mentioned image processing model includes a roof area prediction sub-model for outputting the above-mentioned roof area, a building edge prediction sub-model for outputting the above-mentioned side bottom edge and the above-mentioned side top edge, a building edge direction prediction sub-model for outputting the above-mentioned edge direction, and an offset angle prediction sub-model for outputting the above-mentioned offset angle.

[0041] In some examples shown, the training method of the above image processing model includes:

[0042] Acquire a plurality of training samples including annotation information; wherein the annotation information includes the roof area and the side area of ​​the building, the edges included in the building outline, the edge direction corresponding to each pixel point included in the building, and the offset angle between the roof and the base;

[0043] Based on the loss information corresponding to each sub-model included in the above image processing model, construct joint learning loss information;

[0044] Based on the joint learning loss information and the training samples, each sub-model included in the image processing model is jointly trained until each sub-model converges.

[0045] The present application also proposes an image processing device, the device comprising:

[0046] An acquisition module, used for acquiring a target image containing a building;

[0047] An image processing module is used to perform image processing on the target image to determine the roof area of ​​the building, the side bottom edge and the side top edge of the building, and the offset angle between the roof and the base of the building;

[0048] An offset determination module, used to determine an offset between the bottom edge of the side surface and the top edge of the side surface according to the offset angle;

[0049] The transformation module is used to transform the roof contour corresponding to the above-mentioned roof area according to the above-mentioned offset to obtain the base contour.

[0050] In some examples shown, the apparatus further comprises:

[0051] The building height determination module is used to determine the building height corresponding to the above-mentioned building based on the above-mentioned offset and a predetermined scale between the building height and the offset.

[0052] In some examples shown, the apparatus further comprises:

[0053] An edge direction determination module is used to perform image processing on the target image to determine the edge directions corresponding to the pixel points included in the roof outline of the building;

[0054] The above device also includes:

[0055] A regularization processing module performs regularization processing on the roof outline based on the edge direction to obtain a roof polygon corresponding to the building;

[0056] The above-mentioned transformation module is specifically used to: transform the above-mentioned roof polygon according to the above-mentioned offset to obtain the above-mentioned base contour.

[0057] In some examples shown, the regularization processing module includes:

[0058] A first determination submodule is used to take any pixel point among the pixels included in the roof outline as a target pixel point, and determine the difference between the edge direction corresponding to the target pixel point and the edge direction corresponding to the adjacent pixel point of the target pixel point;

[0059] A second determination submodule is used to determine the target pixel as a vertex of a roof polygon corresponding to the building when the difference between the edge direction corresponding to the target pixel and the edge direction corresponding to the adjacent pixel reaches a first preset threshold;

[0060] The roof polygon determination submodule obtains the roof polygon corresponding to the above-mentioned building based on the vertices of the determined roof polygon.

[0061] In some examples shown, the apparatus further comprises:

[0062] Division module: Divide the preset angle into N angle intervals; where N is a positive integer;

[0063] The first determination submodule is specifically used for:

[0064] Determine the first angle interval to which the edge direction corresponding to the target pixel point belongs;

[0065] Determine a second angle interval to which the edge direction corresponding to the adjacent pixel points of the target pixel point belongs;

[0066] The difference between the identification value corresponding to the first angle interval and the identification value corresponding to the second angle interval is determined as the difference between the edge direction corresponding to the target pixel point and the edge direction corresponding to the adjacent pixel point of the target pixel point.

[0067] In some examples shown, N is a positive integer less than or equal to the second preset threshold.

[0068] In some examples shown, the apparatus further comprises:

[0069] The vertex correction module corrects the vertices of the above-mentioned roof polygon based on the vertex correction model to obtain a corrected roof polygon; wherein the above-mentioned vertex correction model is a model determined based on a graph neural network.

[0070] In some examples shown, the offset determination module includes:

[0071] The offset determination submodule is used to determine the change in the position of the bottom edge of the side surface when the bottom edge of the side surface moves to the top edge of the side surface in the direction of the offset angle, and use the change in the position of the bottom edge of the side surface as the offset.

[0072] In some examples shown, the above offset determination submodule is specifically used to:

[0073] Based on a preset border corresponding to the side top edge, cropping the side top edge probability map corresponding to the side top edge to obtain a first cropping result;

[0074] According to a preset step length and a preset maximum offset, the side bottom edge is moved multiple times in the direction of the offset angle, and after each movement, the side bottom edge probability map corresponding to the side bottom edge is cropped based on the preset border to obtain multiple second cropping results;

[0075] Among the plurality of second cropping results, a target cropping result matching the first cropping result is determined, and a change in the position of the bottom edge of the side surface when the target cropping result is obtained is determined as the offset.

[0076] In some examples shown, the apparatus further comprises:

[0077] Determine the circumscribed border corresponding to the top edge of the side surface as the preset border; or,

[0078] The circumscribed frame corresponding to the combined edge obtained by combining the multiple side top edges included in the above roof outline is determined as the above preset frame.

[0079] In some examples shown, the above image processing module is specifically used for:

[0080] Performing image processing on the target image using an image processing model to determine the roof area of ​​the building, the side bottom edge and the side top edge of the building, and the offset angle between the roof and the base of the building;

[0081] The above-mentioned image processing model includes a roof area prediction sub-model for outputting the above-mentioned roof area, a building edge prediction sub-model for outputting the above-mentioned side bottom edge and the above-mentioned side top edge, a building edge direction prediction sub-model for outputting the above-mentioned edge direction, and an offset angle prediction sub-model for outputting the above-mentioned offset angle.

[0082] In some examples shown, the training device corresponding to the training method of the above-mentioned image processing model includes:

[0083] A training sample acquisition module, used to acquire a plurality of training samples including annotation information; wherein the annotation information includes the roof area and the side area of ​​the building, the edges included in the building outline, the edge direction corresponding to each pixel point included in the building, and the offset angle between the roof and the base;

[0084] A loss information determination module, used to construct joint learning loss information based on the loss information corresponding to each sub-model included in the above-mentioned image processing model;

[0085] The joint training module is used to jointly train the sub-models included in the above-mentioned image processing model based on the above-mentioned joint learning loss information and the above-mentioned training samples until the above-mentioned sub-models converge.

[0086] The present application also proposes an electronic device, the device comprising:

[0087] processor;

[0088] A memory for storing the above processor executable instructions;

[0089] The processor is configured to call the executable instructions stored in the memory to implement the image processing method shown in any of the above embodiments.

[0090] The present application also proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the image processing method shown in any of the above embodiments.

[0091] In the above scheme, the roof area corresponding to the building, the side bottom edge and the side top edge corresponding to the building, and the offset angle between the roof and the base corresponding to the building are determined from the acquired target image. Then, according to the offset angle, the offset between the side bottom edge and the side top edge is determined. Finally, according to the offset, the roof contour corresponding to the roof area is transformed to obtain the base contour, so that in the process of predicting the building base, there is no need to rely on the base features included in the target image, so that even if the building base features included in the target image are blocked, the building base with high accuracy can be obtained.

[0092] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] In order to more clearly illustrate the technical solutions in one or more embodiments of the present application or related technologies, the drawings required for use in the embodiments or related technology descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0094] Figure 1 A method flow chart of an image processing method shown in this application;

[0095] Figure 2 A schematic diagram of a base profile prediction process shown in this application;

[0096] Figure 3 A schematic diagram of a process of performing image processing on a target image shown in the present application;

[0097] Figure 4 A schematic diagram of an offset angle shown in this application;

[0098] Figure 5 A schematic diagram of the offset between the roof and the base shown in this application;

[0099] Figure 6 A method flow chart of an offset determination method shown in the present application;

[0100] Figure 7 A schematic diagram of the movement of the side bottom edge shown in the present application;

[0101] Figure 8 A schematic diagram of a base profile prediction process shown in this application;

[0102] Fig. 9 A schematic diagram of an edge direction shown in this application;

[0103] Fig.10 A schematic diagram of a process of performing image processing on a target image shown in the present application;

[0104] Fig.11 A method flow chart of an image processing model training method shown in this application;

[0105] Fig.12 A schematic diagram of an image processing device shown in the present application;

[0106] Fig.13 This is a hardware structure diagram of an electronic device shown in this application. DETAILED DESCRIPTION

[0107] The exemplary embodiments will be described in detail below, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0108] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "above", and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. It should also be understood that the word "if" used in this article, depending on the context, can be interpreted as "at the time of" or "when" or "in response to determining".

[0109] The present application aims to propose an image processing method. The method predicts the roof area corresponding to the building from the target image, as well as the relevant features for determining the offset between the roof and the base of the building, and determines the offset according to the relevant features. After determining the offset, the roof contour corresponding to the roof area is transformed based on the offset to obtain the base contour, so that in the process of predicting the building base, there is no need to rely on the base features included in the target image, so that even if the building base features included in the target image are blocked, the building base with high accuracy can be obtained.

[0110] See also Figure 1 , Figure 1 FIG. 1 is a flowchart of an image processing method shown in this application. Figure 1 As shown, the above method may include:

[0111] S102, acquiring a target image containing a building.

[0112] S104, performing image processing on the target image to determine the roof area of ​​the building, the side bottom edge and the side top edge of the building, and the offset angle between the roof and the base of the building.

[0113] S106: Determine the offset between the bottom edge of the side surface and the top edge of the side surface according to the offset angle.

[0114] S108: According to the offset, the roof contour corresponding to the roof area is transformed to obtain a base contour.

[0115] The above image processing method can be applied to electronic devices. The above electronic devices can execute the above image processing method by carrying a software system corresponding to the image processing method. In the embodiment of the present application, the type of the above electronic device can be a laptop, a computer, a server, a mobile phone, a PAD terminal, etc., which is not particularly limited in the present application.

[0116] It is understandable that the above-mentioned image processing method can be executed by the terminal device or the server device alone, or can be executed by the terminal device and the server device in cooperation.

[0117] For example, the above image processing method may be integrated into a client. After receiving an image processing request, a terminal device equipped with the client may provide computing power through its own hardware environment to execute the above image processing method.

[0118] For another example, the above image processing method may be integrated into a system platform. After receiving an image processing request, a server device equipped with the system platform may provide computing power through its own hardware environment to execute the above image processing method.

[0119] For another example, the image processing method can be divided into two tasks: acquiring a target image and processing the target image. The acquisition task can be integrated into the client and mounted on the terminal device. The processing task can be integrated into the server and mounted on the server device. The terminal device can initiate an image processing request to the server device after acquiring the target image. After receiving the image processing request, the server device can execute the method on the target image in response to the request.

[0120] The following description is made by taking an electronic device (hereinafter referred to as device) as an example.

[0121] See also Figure 2 , Figure 2 A schematic diagram of a base contour prediction process shown in this application.

[0122] like Figure 2 As shown, after acquiring the target image, the above-mentioned device can execute S104 to perform image processing on the above-mentioned target image to determine the roof area of ​​the above-mentioned building, the side bottom edge and the side top edge of the above-mentioned building, and the offset angle between the roof and the base of the above-mentioned building.

[0123] The target image is an image including multiple buildings. For example, the target image may be a remote sensing image taken by an airplane, a drone, a satellite or other equipment.

[0124] In one case, when acquiring the target image, the above device can complete the input of the target image by interacting with the user. For example, the above device can provide the user with a window for inputting the target image to be processed through its interface, so that the user can input the image. The user can complete the input of the target image based on the window. After acquiring the target image, the above device can input the image into the image processing model for calculation.

[0125] In another case, the above device can directly obtain the remote sensing image output by the image acquisition system. For example, the above device can pre-establish a certain protocol with the remote sensing image acquisition system. When the image acquisition system generates the remote sensing image, it can be sent to the above device for image processing.

[0126] In some examples, the target image may be processed using an image processing model to extract from the target image a roof area corresponding to the building, a side bottom edge and a side top edge corresponding to the building, and an offset angle between the roof and the base corresponding to the building.

[0127] The above-mentioned image processing model can specifically be an image processing model for predicting the building roof area and the above-mentioned related features for the target image. In practical applications, the image processing model can be a pre-trained neural network model.

[0128] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of image processing for a target image shown in the present application. Figure 3 As shown, the above image processing model may include three branches, and the three branches may share the same backbone network. Among them, branch one may be used to predict the roof area, branch two may be used to predict the side edge (including the top side edge and the bottom side edge), and branch three may be used to predict the offset angle between the roof and the base corresponding to the building. In the embodiment of the present application, Figure 3 The structure of the image processing model shown in the figure is only for illustrative purposes. In practical applications, the structure of the model can be constructed according to the actual situation.

[0129] The image processing model may be trained based on a plurality of training samples annotated with annotation information, wherein the annotation information may include a roof area, a top edge of a building side, a bottom edge of a building side, and an offset angle between a roof and a base of the building.

[0130] The backbone network is specifically used to predict features for a target image. For example, the backbone network may be a feature prediction network such as VGG, ResNet, etc., which is not particularly limited here. After the target feature map corresponding to the target image is predicted by the backbone network, the target feature map may be respectively input into the three branches for further prediction.

[0131] The above branch 1 and branch 2 may be pixel-level segmentation networks. The above branch 1 may classify each pixel included in the target image as a roof or background. For example, the target image includes pixel A. If the above branch 1 predicts that the pixel A is a pixel included in the roof area, the pixel A may be classified as a roof.

[0132] The above branch 2 can classify each pixel point included in the target image as one of the side top edge, the side bottom edge or the background. For example, the target image includes pixel point B. If the above branch 2 predicts that the pixel point B is a pixel point included in the side top edge, the pixel point B can be classified as the side top edge.

[0133] The above branch three may specifically be an image-level segmentation network. The above branch three may predict the offset angle between the base and the roof of the building included in the target image. The above offset angle may refer to the angular offset between the base and the roof. Through the offset angle, the relationship between the change in the y-axis direction and the change in the x-axis direction of the roof position and the base position may be determined.

[0134] Exemplarily, the offset angle may be the angle between the side hypotenuse and the vertical direction (for example, the angle between the side hypotenuse and the vertical direction in the clockwise direction). The tangent value corresponding to the angle obtained by subtracting 90 degrees from the angle is the ratio of the change in the roof position and the base position in the y-axis direction to the change in the x-axis direction.

[0135] See also Figure 4 , Figure 4 This is a schematic diagram of an offset angle shown in this application.

[0136] Figure 4 The coordinate system shown is a rectangular coordinate system constructed with the lower left corner of the target image as the coordinate circle center. The side hypotenuse is an edge connecting the base and the roof. Angle α is the above-mentioned offset angle.

[0137] In the embodiment of the present application, on the one hand, since the target image is usually a remote sensing image taken by a satellite or an airplane far away from the building, the above-mentioned offset angle corresponding to each building in the target image that can be taken is roughly the same. On the other hand, the above-mentioned offset angle can also be other angles defined by the developer according to the actual situation that can indicate the angle between the base and the roof of the building. For example, the above-mentioned offset angle can also be the angle between the side hypotenuse and the vertical upward direction or the horizontal rightward direction, etc.

[0138] It is understood that the side bottom edge is usually one of the edges included in the building base outline, and the side top edge is usually one of the edges included in the building roof outline.

[0139] Please continue to see Figure 2 After determining the offset angle, the top edge of the building side surface and the bottom edge of the building side surface, S106 may be performed to determine the offset between the bottom edge of the side surface and the top edge of the side surface according to the offset angle.

[0140] The above offset may specifically refer to the position offset between the base and the roof. Through the above offset, the roof outline can be transformed to the bottom outline.

[0141] Exemplarily, the above offset may be an offset vector, that is, the displacement of the roof and the base on the x-axis and the y-axis.

[0142] See also Figure 5 , Figure 5 A schematic diagram of the offset between a roof and a base shown in this application.

[0143] Figure 5 The coordinate system shown is a rectangular coordinate system constructed with the lower left corner of the target image as the coordinate circle center. Point P is a point on the base of the building. Point Q is a point on the roof of the building corresponding to point P. The offset (x2-x1, y2-y1) between the coordinates corresponding to point Q and the coordinates corresponding to point P is the above offset.

[0144] In this step, when determining the offset between the bottom edge of the side surface and the top edge of the side surface according to the offset angle, the device may input the offset angle, the bottom edge of the side surface and the top edge of the side surface into an offset determination unit for calculation to obtain the offset between the bottom edge of the side surface and the top edge of the side surface.

[0145] The offset determination unit includes an offset determination algorithm, which can determine the change in the position of the bottom edge of the side surface when the bottom edge of the side surface moves to the top edge of the side surface in the direction of the offset angle, and use the change in the position of the bottom edge of the side surface as the offset.

[0146] In the embodiment of the present application, the algorithm may also be to determine the change in the position of the top edge of the side surface when the top edge of the side surface moves to the bottom edge of the side surface in the direction of the offset angle, and use the change in the position of the top edge of the side surface as the offset. It is understandable that the offset can be determined by determining the top edge or the bottom edge. The two methods have the same principles and the steps can be referenced to each other. The following is an example of determining the offset by moving the bottom edge of the side surface as an example for explanation.

[0147] See also Figure 6 , Figure 6 A flow chart of a method for determining an offset shown in the present application.

[0148] like Figure 6 As shown, the device may first execute S602 to crop the side top edge probability map corresponding to the side top edge based on a preset border corresponding to the side top edge to obtain a first cropping result.

[0149] The preset frame may specifically be a preset frame including a side top edge. The image may be cropped by the frame to obtain the pixel points inside the frame. In some embodiments, after obtaining the side top edge by S104, the circumscribed frame corresponding to the side top edge may be determined as the preset frame.

[0150] In some embodiments, the top edge of the same building obtained in S104 may be discontinuous due to model reasons, so that the top edge cannot be well cut out according to the determined preset border, affecting the accuracy of the offset determination. Therefore, in order to obtain an accurate preset border, the preset border can be determined based on the roof outline and the side top edge.

[0151] Specifically, when determining the preset border, the circumscribed border corresponding to the combined edge obtained by combining a plurality of side top edges included in the roof outline may be determined as the preset border.

[0152] Since the preset border is determined based on the combined edge obtained by combining multiple side top edges included in the roof contour, the preset border can include a complete top edge, so that the first cropping result can include a complete side top edge, thereby improving the accuracy of offset determination.

[0153] After the preset border is determined, the side top edge probability map corresponding to the side top edge may be cropped using the preset border to obtain a first cropping result.

[0154] The side top edge probability map is specifically a top edge segmentation map output after image processing is performed on the target image in S104. The map includes the side top edge included in the target image. The first cropping result obtained by cropping the probability map according to the preset border actually obtains the area including the side top edge of the building.

[0155] After obtaining the first cropping result, S604 can be executed to move the above-mentioned side bottom edge multiple times in the direction of the above-mentioned offset angle according to the preset step size and the preset maximum offset, and after each movement, the side bottom edge probability map corresponding to the above-mentioned side bottom edge is cropped based on the above-mentioned preset border to obtain multiple second cropping results.

[0156] Before executing this step, specify the preset step size and the preset maximum offset.

[0157] Among them, the above-mentioned preset step length specifically refers to the coordinate value of the bottom edge moving along the x-axis direction. The preset step length can be set according to the actual situation. For example, if the target image resolution is large, a longer preset step length can be set; conversely, a smaller preset step length can be set. In the embodiment of the present application, the preset step length can also be a coordinate value moving along the y-axis direction, which is not particularly limited here. The following is an example of the preset step length moving m along the x-axis.

[0158] The above-mentioned preset maximum offset specifically refers to the maximum value of the bottom edge moving along the x-axis. The preset step size can be set according to the actual situation. For example, if the target image resolution is large, a longer maximum offset can be set; otherwise, a smaller maximum offset can be set. In the embodiment of the present application, the maximum offset can also be a coordinate value moving along the y-axis, which is not particularly limited here. The following is an example of the maximum offset being a movement of n (where n is greater than m) along the x-axis.

[0159] After determining the preset step size and the maximum offset, the above S604 can be executed to move the above-mentioned side bottom edge multiple times in the direction of the above-mentioned offset angle according to the preset step size and the preset maximum offset, and after each movement, the side bottom edge probability map corresponding to the above-mentioned side bottom edge is cropped based on the above-mentioned preset border to obtain multiple second cropping results.

[0160] The above-mentioned side bottom edge probability map is specifically a bottom edge segmentation map output after image processing is performed on the target image in S104. The map includes the side bottom edge included in the above-mentioned target image. The second cropping result obtained by cropping the probability map according to the preset border actually obtains the area corresponding to the top edge position of the side of the building in the side bottom edge probability map.

[0161] See also Figure 7 , Figure 7 This is a schematic diagram of the side bottom edge movement shown in the present application.

[0162] like Figure 7 As shown, Figure 7 The coordinate system shown is a rectangular coordinate system constructed with the lower left corner of the bottom edge probability map as the center of the coordinate circle. DE is the initial position of the bottom edge. FG is a certain position in the middle of the bottom edge when it moves. HI is the final position of the bottom edge when it moves (i.e., the maximum offset position). The dotted box is a predetermined preset border.

[0163] The bottom edge starts from the initial position DE, moves m steps in the x-axis direction and tan(α-90)*m steps in the y-axis direction each time until it moves to the HI position. After each movement, the area corresponding to the preset frame coordinate position on the bottom edge probability map can be cropped according to the preset frame to obtain multiple second cropping results.

[0164] After obtaining multiple second cropping results, S606 may be executed to determine a target cropping result matching the first cropping result from the multiple second cropping results, and determine the change in position of the bottom edge of the side surface when obtaining the target cropping result as the offset.

[0165] In this step, a distance determination method such as Euclidean distance and Mahalanobis distance can be used to determine the similarity between the first cropping result and the second cropping result, and the highest similarity is found from the determined similarities. After the highest similarity is determined, the cropping result corresponding to the highest similarity is determined as the target cropping result. After the target cropping result is determined, the change in the position of the bottom edge of the side when the target cropping result is obtained can be determined as the offset.

[0166] For example, assuming that the second cropping result obtained by cropping when the bottom edge moves to position FG is the target cropping result, then the change amount of the bottom edge from position DE to position FG on the x-axis and y-axis can be combined into the offset.

[0167] In the above scheme, since the second cropping result obtained according to the preset border cropping will be most similar to the first cropping result only when the bottom edge moves to the position of the top edge, the above scheme can be adopted to determine the change in the position of the bottom edge of the side surface when the bottom edge of the side surface is moved to the top edge of the side surface in the direction of the offset angle as the offset, thereby determining the accurate offset.

[0168] Please continue to see Figure 2After determining the offset, S108 may be continued to be executed, and the roof contour corresponding to the roof area may be transformed according to the offset to obtain the base contour.

[0169] In this step, the roof outline corresponding to the roof area may be determined first, for example, by determining the outline surrounded by the outermost pixel points corresponding to the roof area as the roof outline.

[0170] After the roof contour is determined, the roof contour corresponding to the roof area may be transformed according to the offset to obtain the base contour.

[0171] Specifically, the coordinates corresponding to each pixel point included in the roof outline can be translated in the x-axis and y-axis directions through the offset to obtain the coordinates corresponding to each pixel point corresponding to the base outline, thereby determining the base outline and completing the base prediction.

[0172] In the above scheme, the roof area corresponding to the building, the side bottom edge and the side top edge corresponding to the building, and the offset angle between the roof and the base corresponding to the building are determined from the acquired target image. Then, according to the offset angle, the offset between the side bottom edge and the side top edge is determined. Finally, according to the offset, the roof contour corresponding to the roof area is transformed to obtain the base contour, thereby achieving a translation transformation of the roof contour with obvious features, and obtaining the building base contour to complete the building base prediction.

[0173] In some embodiments, in order to improve the prediction accuracy of the building base, after obtaining the roof outline, the roof outline can be regularized.

[0174] See also Figure 8 , Figure 8 A schematic diagram of a base contour prediction process shown in this application.

[0175] like Figure 8 As shown, when executing S104 to process the target image, the edge directions corresponding to the pixel points included in the roof outline corresponding to the above-mentioned building can also be obtained.

[0176] The edge direction specifically refers to the direction of the normal vector of the edge. In practical applications, the edge direction is usually quantified by an edge direction angle. The edge direction angle may be the angle between the normal vector and the vertical direction (for example, the angle between the normal vector and the vertical direction in the clockwise direction).

[0177] For details, please see Fig. 9 , Fig. 9 FIG. 1 is a schematic diagram of an edge direction shown in this application. Fig. 9As shown, the direction corresponding to the normal vector LR of the edge JK is the edge direction of the edge JK. In some examples, a vertically downward direction vector LS can be constructed, and the angle of the edge direction can be indicated by the angle β between the normal vector LR and the direction vector LS. It can be understood that the edge direction corresponding to the pixel points included in a certain edge is usually consistent with the edge direction corresponding to the edge.

[0178] In some examples, the device may pre-divide the preset angle into N angle intervals, wherein N is a positive integer. In an embodiment of the present application, the preset angle may be an empirical angle. For example, 360 degrees or 720 degrees. In some embodiments, in order to narrow the value range of the edge direction type and expand the span range of the angle interval, N may be a positive integer less than or equal to a second preset threshold. The second preset threshold may be a more suitable number of angle intervals determined based on experience.

[0179] Since the value of N is less than or equal to the second preset threshold, the span range of the angle interval is expanded and the value range of the edge direction type is narrowed. Therefore, when quantifying the edge direction, there is no need to over-rely on the prediction accuracy of the above-mentioned image processing model, and the edge direction prediction accuracy can also be improved, thereby further improving the more accurate roof outline in the past and improving the prediction accuracy of the building base.

[0180] After determining N, identification values ​​may be assigned to the N angle intervals, respectively. The identification values ​​correspond to the angle intervals one by one. In some examples, the numbering sequence of the angle intervals may be used as the identification values. For example, when a certain angle interval is the third interval, 3 may be used as the identification value corresponding to the angle interval.

[0181] When determining the edge directions corresponding to the edges of the buildings included in the target image, the target image may be input into a pre-constructed edge direction prediction model to predict the edge direction.

[0182] See also Fig.10 , Fig.10 This is a schematic diagram of a process of performing image processing on a target image shown in the present application.

[0183] like Fig.10 As shown, the above image processing model includes, in addition to the above three branches, a branch 4. The branch 4 can be used to predict the edge direction corresponding to each pixel point included in the roof outline. The branch 4 shares the above backbone network with the other three branches.

[0184] The above-mentioned image processing model can be obtained by training based on a plurality of training samples annotated with annotation information. The above-mentioned annotation information also includes the edge direction corresponding to each pixel. When constructing the above-mentioned training samples, the angle interval identification value corresponding to the edge direction of each pixel of the original image can be annotated. For example, after obtaining the original image, each pixel of the original image can be traversed, and the angle interval identification value corresponding to the edge direction of each pixel can be annotated by annotation software.

[0185] Please continue to see Figure 8 After obtaining the edge direction, the roof outline can be regularized based on the edge direction to obtain the roof polygon corresponding to the building.

[0186] In practical applications, when the roof contour is regularized based on the edge direction to obtain the roof polygon corresponding to the building, any pixel point included in the roof contour can be used as the target pixel point to determine the difference between the edge direction corresponding to the target pixel point and the edge direction corresponding to the adjacent pixel point of the target pixel point.

[0187] When the difference between the edge direction corresponding to the target pixel point and the edge direction corresponding to the adjacent pixel point reaches a first preset threshold, the target pixel point is determined as a vertex of the roof polygon corresponding to the building.

[0188] The first preset threshold may be a threshold set based on experience. Reaching the threshold may indicate that two adjacent pixels do not belong to the same edge. Therefore, according to the above steps, the pixels included in the roof outline that belong to the same edge as the pixels included in the background may be deleted, and the pixels that do not belong to the background may be retained, thereby achieving the purpose of regularizing the roof outline.

[0189] It is understandable that the value of the first preset threshold is different when the edge direction is quantified in different ways. For example, when the edge direction is quantified by the direction angle, the first preset threshold may be 30. When the edge direction is quantified by the identification value of the angle interval corresponding to the direction angle, the first preset threshold may be 3.

[0190] In some examples, when the edge direction is quantified by the edge direction angle, when determining the difference between the edge direction corresponding to the target pixel point and the edge direction corresponding to the adjacent pixel point of the target pixel point, the first edge direction angle corresponding to the target pixel point can be determined, and the second edge direction angle corresponding to the adjacent pixel point of the target pixel point can be determined.

[0191] After determining the first edge direction angle and the second edge direction angle, the difference between the first edge direction angle and the second edge direction angle may be determined as the difference between the edge direction corresponding to the target pixel and the edge direction corresponding to the adjacent pixel of the target pixel.

[0192] In some examples, when determining the difference between the edge direction corresponding to the target pixel and the edge direction corresponding to the adjacent pixel of the target pixel, the first angle interval to which the edge direction corresponding to the target pixel belongs can be determined, and the second angle interval to which the edge direction corresponding to the adjacent pixel of the target pixel belongs can be determined.

[0193] After determining the first angle interval and the second angle interval, the difference between the identification value corresponding to the first angle interval and the identification value corresponding to the second angle interval can be determined as the difference between the edge direction corresponding to the target pixel point and the edge direction corresponding to the adjacent pixel point of the target pixel point.

[0194] After executing the above steps for all target pixel points included in the above roof outline, the roof polygon corresponding to the above building can be obtained based on the determined vertices of the roof polygon.

[0195] In some embodiments, after obtaining the roof polygon corresponding to the above-mentioned building based on the vertices of the determined roof polygon, the vertices of the above-mentioned roof polygon can be corrected based on a vertex correction model to obtain a corrected roof polygon; wherein the above-mentioned vertex correction model is a model determined based on a graph neural network.

[0196] It is understandable that the above vertex correction model can be two models independent of the image processing model used in this application, or it can be a sub-model (sub-module) of the image processing model, and there is no special limitation here. Of course, in order to control the amount of calculation, the above vertex correction model can be used as a sub-model (sub-module) of the image processing model to perform vertex correction.

[0197] Since this scheme uses a vertex correction model built based on a graph neural network to further correct the roof polygon, a more accurate roof contour can be obtained, thereby improving the prediction accuracy of the building base.

[0198] Please continue to see Figure 8 After obtaining the roof polygon, the roof polygon can be transformed according to the above offset to obtain the base outline.

[0199] Since the above-mentioned roof polygon is a regular shape obtained after regularizing the roof contour, this solution can predict a more accurate roof contour, thereby improving the prediction accuracy of the building base.

[0200] Since predicting building height requires relying on remote sensing images from multiple perspectives, LiDAR data, digital surface model (DSM) and other data, and these data are expensive and difficult to obtain, predicting building height is difficult and costly.

[0201] In some embodiments, in order to predict the building height through a single-view remote sensing image, after determining the offset, the building height prediction can also be performed based on the single-view target image. Specifically, after determining the offset, the building height corresponding to the building can be determined based on the offset and a predetermined scale between the building height and the offset.

[0202] In practical applications, the real heights of some buildings and the offsets corresponding to these buildings determined by the offset determination method described in this application can be obtained in advance, and then the scale between the building heights and the offsets can be determined based on the above data.

[0203] After determining the above scale, the corresponding building height can be determined according to the predicted offset.

[0204] In the above scheme, after determining the above offset, the building height corresponding to the above building can be obtained based on the above offset and the predetermined scale between the building height and the offset. Therefore, when predicting the building height, there is no need to rely on remote sensing images, laser radar (LIDAR) data, digital surface model (DSM) and other data from multiple perspectives, thereby simplifying the cost and difficulty of building height prediction.

[0205] The above is an introduction to the building base prediction and building height prediction solutions shown in this application. The following introduces the training method of the image processing model.

[0206] In the present application, the image processing model used in the building base prediction scheme may include a roof area prediction sub-model for outputting the above-mentioned roof area, a building edge prediction sub-model for outputting the above-mentioned side bottom edge and the above-mentioned side top edge, a building edge direction prediction sub-model for outputting the above-mentioned edge direction, and an offset angle prediction sub-model for outputting the above-mentioned offset angle.

[0207] In order to improve the prediction accuracy of the image processing model for the base area and the generalization ability of the model, a multi-task joint training method is adopted when training the image processing model.

[0208] In some embodiments, in order to increase the supervision information during the training of the image processing model, thereby improving the prediction accuracy of the image processing model, constraints on the roof and side areas of the building, the edges included in the building outline, the edge directions corresponding to each pixel point included in the building, and the offset angle between the roof and the base can be introduced.

[0209] See also Fig.11 , Fig.11 A method flow chart of an image processing model training method shown in this application.

[0210] like Fig.11 As shown, the method includes:

[0211] S1102, obtaining a plurality of training samples including annotation information; wherein the annotation information includes the roof area and the side area of ​​the building, the edges included in the building outline, the edge directions corresponding to the pixels included in the building, and the offset angle between the roof and the base.

[0212] When executing this step, the original image can be annotated with annotation information by manual annotation or machine-assisted annotation. For example, after acquiring the original image, on the one hand, the image annotation software can be used to annotate each pixel point included in the original image to indicate whether it belongs to the roof of the building, the side area or the background; mark which edge of the building outline it belongs to; mark the corresponding edge direction; on the other hand, the offset angle between the roof and the base of the building included in the image can be ensured. After completing the above-mentioned annotation operation on the original image, a training sample can be obtained. In an embodiment of the present application, one-hot encoding or other methods can be used for encoding when encoding the annotation information training sample, and the present application does not limit the specific encoding method.

[0213] S1104: Construct joint learning loss information based on the loss information corresponding to each sub-model included in the above-mentioned image processing model.

[0214] When executing this step, the loss information corresponding to each sub-model can be determined first. In order to improve the prediction accuracy of the sub-model, in this application, the loss information corresponding to each sub-model is cross entropy loss information.

[0215] After determining the loss information corresponding to each sub-model, the joint learning loss information can be constructed based on the loss information corresponding to each sub-model included in the above image processing model. For example, the loss information corresponding to each sub-model can be added to obtain the above joint learning loss information.

[0216] In the embodiment of the present application, a regularization term may be added to the above-mentioned joint learning loss information in the present application, which is not particularly limited here.

[0217] After determining the joint learning loss information and the training samples, S1106 may be executed to jointly train the sub-models included in the image processing model based on the joint learning loss information and the training samples until the sub-models converge.

[0218] When training the model, hyperparameters such as learning rate, number of training cycles, etc. may be specified first. After the hyperparameters are determined, supervised training may be performed on the image processing model based on the training samples annotated with annotation information.

[0219] In the supervised training process, after forward propagation is performed on the image processing model to obtain the calculation result, the error between the annotation information and the above calculation result can be evaluated based on the constructed joint learning loss information. After obtaining the error, the stochastic gradient descent method can be used to determine the descent gradient. After determining the descent gradient, the model parameters corresponding to the above image processing model can be updated based on back propagation. Repeat the above process until the above sub-models converge. In the embodiment of the present application, the present application does not specifically limit the conditions for model convergence.

[0220] When training the image processing model, due to the use of a supervised joint training method, the four sub-models included in the image processing can be trained simultaneously, so that the sub-models can both constrain and promote each other during the training process, thereby improving the convergence efficiency of the image processing model on the one hand; on the other hand, it promotes the backbone network shared by the sub-models to predict features that are more beneficial to the prediction of the base area, thereby improving the accuracy of the base prediction.

[0221] Corresponding to any of the above embodiments, the present application also proposes an image processing device.

[0222] See also Fig.12 , Fig.12 This is a schematic diagram of an image processing device shown in the present application.

[0223] like Fig.12 As shown, the above-mentioned device 1200 includes:

[0224] An acquisition module 1210 is used to acquire a target image containing a building;

[0225] The image processing module 1220 is used to perform image processing on the target image to determine the roof area of ​​the building, the side bottom edge and the side top edge of the building, and the offset angle between the roof and the base of the building;

[0226] An offset determination module 1230, configured to determine an offset between the bottom edge of the side surface and the top edge of the side surface according to the offset angle;

[0227] The transformation module 1240 is used to transform the roof contour corresponding to the roof area according to the offset to obtain a base contour.

[0228] In some examples shown, the apparatus 1200 further includes:

[0229] The building height determination module is used to determine the building height corresponding to the above-mentioned building based on the above-mentioned offset and a predetermined scale between the building height and the offset.

[0230] In some examples shown, the apparatus 1200 further includes:

[0231] An edge direction determination module is used to perform image processing on the target image to determine the edge directions corresponding to the pixel points included in the roof outline of the building;

[0232] The above-mentioned device 1200 further includes:

[0233] A regularization processing module performs regularization processing on the roof outline based on the edge direction to obtain a roof polygon corresponding to the building;

[0234] The transformation module 1240 is specifically used to transform the roof polygon according to the offset to obtain the base contour.

[0235] In some examples shown, the regularization processing module includes:

[0236] A first determination submodule is used to take any pixel point among the pixels included in the roof outline as a target pixel point, and determine the difference between the edge direction corresponding to the target pixel point and the edge direction corresponding to the adjacent pixel point of the target pixel point;

[0237] A second determination submodule is used to determine the target pixel as a vertex of a roof polygon corresponding to the building when the difference between the edge direction corresponding to the target pixel and the edge direction corresponding to the adjacent pixel reaches a first preset threshold;

[0238] The roof polygon determination submodule obtains the roof polygon corresponding to the above-mentioned building based on the vertices of the determined roof polygon.

[0239] In some examples shown, the apparatus 1200 further includes:

[0240] Division module: Divide the preset angle into N angle intervals; where N is a positive integer;

[0241] The first determination submodule is specifically used for:

[0242] Determine the first angle interval to which the edge direction corresponding to the target pixel point belongs;

[0243] Determine a second angle interval to which the edge direction corresponding to the adjacent pixel points of the target pixel point belongs;

[0244] The difference between the identification value corresponding to the first angle interval and the identification value corresponding to the second angle interval is determined as the difference between the edge direction corresponding to the target pixel point and the edge direction corresponding to the adjacent pixel point of the target pixel point.

[0245] In some examples shown, N is a positive integer less than or equal to the second preset threshold.

[0246] In some examples shown, the apparatus 1200 further includes:

[0247] The vertex correction module corrects the vertices of the above-mentioned roof polygon based on the vertex correction model to obtain a corrected roof polygon; wherein the above-mentioned vertex correction model is a model determined based on a graph neural network.

[0248] In some examples shown, the offset determination module 1230 includes:

[0249] The offset determination submodule is used to determine the change in the position of the bottom edge of the side surface when the bottom edge of the side surface moves to the top edge of the side surface in the direction of the offset angle, and use the change in the position of the bottom edge of the side surface as the offset.

[0250] In some examples shown, the above offset determination submodule is specifically used to:

[0251] Based on a preset border corresponding to the side top edge, cropping the side top edge probability map corresponding to the side top edge to obtain a first cropping result;

[0252] According to a preset step length and a preset maximum offset, the side bottom edge is moved multiple times in the direction of the offset angle, and after each movement, the side bottom edge probability map corresponding to the side bottom edge is cropped based on the preset border to obtain multiple second cropping results;

[0253] Among the plurality of second cropping results, a target cropping result matching the first cropping result is determined, and a change in the position of the bottom edge of the side surface when the target cropping result is obtained is determined as the offset.

[0254] In some examples shown, the apparatus 1200 further includes:

[0255] Determine the circumscribed border corresponding to the top edge of the side surface as the preset border; or,

[0256] The circumscribed frame corresponding to the combined edge obtained by combining the multiple side top edges included in the above roof outline is determined as the above preset frame.

[0257] In some examples shown, the image processing module 1220 is specifically used to:

[0258] Performing image processing on the target image using an image processing model to determine the roof area of ​​the building, the side bottom edge and the side top edge of the building, and the offset angle between the roof and the base of the building;

[0259] The above-mentioned image processing model includes a roof area prediction sub-model for outputting the above-mentioned roof area, a building edge prediction sub-model for outputting the above-mentioned side bottom edge and the above-mentioned side top edge, a building edge direction prediction sub-model for outputting the above-mentioned edge direction, and an offset angle prediction sub-model for outputting the above-mentioned offset angle.

[0260] In some examples shown, the training device 1300 corresponding to the training method of the above-mentioned image processing model includes:

[0261] The training sample acquisition module 1310 is used to acquire a plurality of training samples including annotation information; wherein the annotation information includes the roof area and the side area of ​​the building, the edges included in the building outline, the edge direction corresponding to each pixel point included in the building, and the offset angle between the roof and the base;

[0262] A loss information determination module 1320, configured to construct joint learning loss information based on the loss information corresponding to each sub-model included in the above-mentioned image processing model;

[0263] The joint training module 1330 is used to jointly train the sub-models included in the above-mentioned image processing model based on the above-mentioned joint learning loss information and the above-mentioned training samples until the above-mentioned sub-models converge.

[0264] The embodiment of the image processing device shown in the present application can be applied to an electronic device. Accordingly, the present application discloses an electronic device, which may include: a processor.

[0265] Memory used to store processor-executable instructions.

[0266] The processor is configured to call the executable instructions stored in the memory to implement the image processing method as shown in any of the above embodiments.

[0267] See also Fig.13 , Fig.13 This is a hardware structure diagram of an electronic device shown in this application.

[0268] like Fig.13 As shown, the electronic device may include a processor for executing instructions, a network interface for network connection, a memory for storing operating data for the processor, and a non-volatile memory for storing instructions corresponding to the image processing device.

[0269] The image processing device embodiment can be implemented by software, hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the electronic device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, in addition to Fig.13 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device in the embodiment is located may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0270] It is understandable that, in order to improve the processing speed, the corresponding instructions of the image processing device can also be directly stored in the memory, which is not limited here.

[0271] The present application proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute an image processing method as shown in any of the above embodiments.

[0272] Those skilled in the art will appreciate that one or more embodiments of the present application may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (which may include but are not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0273] The term "and / or" in the present application means at least one of the two. For example, "A and / or B" may include three options: A, B, and "A and B".

[0274] Each embodiment in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0275] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0276] The embodiments of the subject matter and functional operations described in this application can be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware that may include the structures disclosed in this application and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this application can be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0277] The processes and logic flows described in this application can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The above processes and logic flows can also be performed by special-purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special-purpose logic circuits.

[0278] A computer suitable for executing a computer program may include, for example, a general and / or special purpose microprocessor, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer may include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, a computer does not necessarily have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0279] Computer readable media suitable for storing computer program instructions and data may include all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0280] Although the application includes many specific implementation details, these should not be interpreted as limiting any disclosed scope or the scope of protection claimed, but are mainly used to describe the features of specific disclosed specific embodiments. Certain features described in multiple embodiments in the application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented with any suitable sub-combination. In addition, although the feature can work as above-mentioned in some combinations and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0281] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0282] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0283] The above is only a preferred embodiment of one or more embodiments of the present application and is not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of one or more embodiments of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Get a target image containing a building; Performing image processing on the target image to determine the roof area of ​​the building, the side bottom edge and the side top edge of the building, and the offset angle between the roof and the base of the building; Determine a change in the position of the bottom edge of the side surface when the bottom edge of the side surface moves to the top edge of the side surface in the direction of the offset angle, and use the change in the position of the bottom edge of the side surface as the offset between the bottom edge of the side surface and the top edge of the side surface; According to the offset, the roof contour corresponding to the roof area is transformed to obtain a base contour; The method further comprises: Performing image processing on the target image to determine edge directions corresponding to the pixel points included in the roof outline of the building; Based on the edge direction, regularizing the roof outline to obtain a roof polygon corresponding to the building; The step of transforming the roof contour corresponding to the roof area according to the offset to obtain the base contour includes: transforming the roof polygon according to the offset to obtain the base contour.

2. The method according to claim 1, characterized in that: The method further comprises: Based on the offset and a predetermined ratio between the building height and the offset, a building height corresponding to the building is determined.

3. The method according to claim 1, characterized in that The step of regularizing the roof contour based on the edge direction to obtain a roof polygon corresponding to the building includes: Taking any pixel point among the pixels included in the roof outline as a target pixel point, and determining the difference between an edge direction corresponding to the target pixel point and an edge direction corresponding to an adjacent pixel point of the target pixel point; When the difference between the edge direction corresponding to the target pixel point and the edge direction corresponding to the adjacent pixel point reaches a first preset threshold, the target pixel point is determined as the vertex of the roof polygon corresponding to the building; Based on the determined vertices of the roof polygon, a roof polygon corresponding to the building is obtained.

4. The method according to claim 3, characterized in that The method further comprises: Divide the preset angle into N angle intervals; where N is a positive integer; The determining the difference between the edge direction corresponding to the target pixel and the edge direction corresponding to the adjacent pixel of the target pixel includes: Determine a first angle interval to which the edge direction corresponding to the target pixel point belongs; Determine a second angle interval to which the edge direction corresponding to the adjacent pixel points of the target pixel point belongs; The difference between the identification value corresponding to the first angle interval and the identification value corresponding to the second angle interval is determined as the difference between the edge direction corresponding to the target pixel point and the edge direction corresponding to the adjacent pixel point of the target pixel point.

5. The method according to claim 4, characterized in that N is a positive integer less than or equal to the second preset threshold.

6. The method according to any one of claims 3 to 5, characterized in that: The method further comprises: Based on the vertex correction model, the vertices of the roof polygon are corrected to obtain a corrected roof polygon; wherein the vertex correction model is a model determined based on a graph neural network.

7. The method according to claim 1, characterized in that The step of determining a change in the position of the bottom edge of the side surface when the bottom edge of the side surface moves to the top edge of the side surface in the direction of the offset angle, and using the change in the position of the bottom edge of the side surface as an offset between the bottom edge of the side surface and the top edge of the side surface, comprises: Based on a preset border corresponding to the side top edge, cropping the side top edge probability map corresponding to the side top edge to obtain a first cropping result; According to a preset step size and a preset maximum offset, the side bottom edge is moved multiple times in the direction of the offset angle, and after each movement, the side bottom edge probability map corresponding to the side bottom edge is cropped based on the preset border to obtain multiple second cropping results; Among the multiple second cropping results, a target cropping result matching the first cropping result is determined, and the change amount of the bottom edge position of the side surface when the target cropping result is obtained is determined as the offset.

8. The method according to claim 7, characterized in that The method further comprises: Determine the circumscribed frame corresponding to the top edge of the side surface as the preset frame; or, The circumscribed frame corresponding to the combined edge obtained by combining multiple side top edges included in the roof outline is determined as the preset frame.

9. The method according to any one of claims 1 to 8, characterized in that: The step of performing image processing on the target image to determine the roof area of ​​the building, the side bottom edge and the side top edge of the building, and the offset angle between the roof and the base of the building includes: Performing image processing on the target image using an image processing model to determine the roof area of ​​the building, the side bottom edge and the side top edge of the building, and the offset angle between the roof and the base of the building; The image processing model includes a roof area prediction submodel for outputting the roof area, a building edge prediction submodel for outputting the side bottom edge and the side top edge, a building edge direction prediction submodel for outputting the edge direction, and an offset angle prediction submodel for outputting the offset angle.

10. The method according to claim 9, characterized in that The training method of the image processing model includes: Acquire a plurality of training samples including annotation information; wherein the annotation information includes the roof area and the side area of ​​the building, the edges included in the building outline, the edge direction corresponding to each pixel point included in the building, and the offset angle between the roof and the base; Constructing joint learning loss information based on the loss information corresponding to each sub-model included in the image processing model; Based on the joint learning loss information and the training samples, each sub-model included in the image processing model is jointly trained until each sub-model converges.

11. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a target image containing a building; An image processing module is used to perform image processing on the target image to determine the roof area of ​​the building, the side bottom edge and the side top edge of the building, and the offset angle between the roof and the base of the building; an offset determination module, used to determine a change in the position of the bottom edge of the side surface when the bottom edge of the side surface moves to the top edge of the side surface in the direction of the offset angle, and use the change in the position of the bottom edge of the side surface as an offset between the bottom edge of the side surface and the top edge of the side surface; A transformation module, used for transforming the roof contour corresponding to the roof area according to the offset to obtain a base contour; The device also includes: An edge direction determination module is used to perform image processing on the target image to determine the edge directions corresponding to the pixel points included in the roof outline of the building; A regularization processing module, which performs regularization processing on the roof outline based on the edge direction to obtain a roof polygon corresponding to the building; The transformation module is specifically used to transform the roof polygon according to the offset to obtain the base outline.

12. The device according to claim 11, characterized in that The device also includes: The building height determination module is used to determine the building height corresponding to the building based on the offset and a predetermined scale between the building height and the offset.

13. An electronic device, characterized in that: The device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to call the executable instructions stored in the memory to implement the image processing method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that: The storage medium stores a computer program for executing the image processing method according to any one of claims 1 to 10.

Images