A method and system for identifying ground object targets based on multi-modal image data

Abstract

The application provides a ground object recognition method and system based on multi-modal image data, relates to the technical field of image recognition, and has the technical solution points that according to cross-modal subject consistency and contour connection relationship, the building subject is retained, the weak response edge is supplemented, and adjacent interference is removed, then semantic fusion and graph spot generation are carried out in the modified area, the mixing of non-building areas such as roads and courtyards can be reduced, the building edge affected by the shadow is retained, and the recognition graph spot is more in line with the building range.

Description

Technical Field

[0001] This application relates to the field of image analysis technology, specifically to a method and system for identifying ground features based on multimodal image data. Background Technology

[0002] In road inspection and feature information collection, unmanned vehicles equipped with imaging devices such as visible light cameras and thermal infrared cameras can acquire multimodal images of surrounding features along the road. After the acquired images are correlated with time, vehicle position, and attitude information, they can be used to identify buildings on both sides of the road and record their category and visible range. Different modalities reflect the appearance texture and thermal radiation characteristics of features, providing complementary information for building identification.

[0003] When autonomous vehicles observe buildings from the ground, images often simultaneously show building facades, eaves, roads, courtyards, and vegetation. Building edges may be affected by trees, parked vehicles, or shadows; due to differences in imaging mechanisms and acquisition perspectives, the salient areas of the same building may also differ in different modalities. For example, some facade edges in visible light images may be unclear due to shadows, while adjacent road surfaces in thermal infrared images may have similar responses to building surfaces, causing non-building parts to be mixed into the candidate building area.

[0004] While geometric registration can reduce positional bias by directly using the registered candidate regions as a whole for fusion recognition, it struggles to further distinguish between weak-response building edges and adjacent surface interference. If both are retained, roads or courtyards may be misidentified as buildings; conversely, uniformly removing weaker-response areas can lead to missing building edges. Therefore, in multimodal images acquired by autonomous vehicles, preserving effective building edges while reducing adjacent surface interference is a crucial problem for ground object recognition. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for identifying ground features based on multimodal image data, so as to improve the identification area pollution and image object mismatch caused by the mixing of adjacent ground surfaces in the cross-modal corresponding areas of buildings.

[0006] This application provides a method for ground target recognition based on multimodal image data, including the following steps:

[0007] Obtain candidate building regions from at least two modal images within the same geographic area, establish cross-modal correspondences, and extract the main features, edge features, and transition features outside the boundary of each candidate building region;

[0008] Based on the aforementioned main characteristics, determine the cross-modal homogeneous region;

[0009] For the edge sub-blocks to be determined in the candidate building area, along the boundary normal direction between the edge sub-block and the adjacent homogeneous region, obtain the modal normalized pixel response mean sequence arranged from the inside of the main body to the outside, and fit the overall change trend of the sequence.

[0010] Based on the edge features and the transition features, check the angle between the edge sub-block and the adjacent contour segments of the same origin region and the distance between the endpoints. The edge sub-block with the angle lower than the preset angle tolerance and the distance between the endpoints within the preset gap range is determined to be able to continue the main contour; otherwise, it is determined to be unable to continue the main contour.

[0011] The edge sub-blocks whose overall change trend decays away from the main body and can continue the outline of the main body are identified as edge compensation regions. The edge sub-blocks whose overall change trend response change amplitude on the outside of the main body does not exceed a preset trend threshold and cannot continue the outline of the main body are identified as interference regions to be eliminated.

[0012] The original regions are retained, the edge compensation regions are filled in, and the confirmed interference regions to be removed are eliminated to obtain the correction regions for each mode.

[0013] The semantic fusion range of image features of each modality is defined by the modified region to identify building categories and generate building patches.

[0014] Furthermore, the determination of the homogeneous region, the edge compensation region, and the interference region to be eliminated includes:

[0015] For a pair of candidate building regions with the aforementioned cross-modal correspondence, the consistency of main body response, the consistency of contour direction, and the consistency of boundary transition mode are determined based on the main body features, the edge features, and the transition features, respectively. The three are then weighted according to preset weights to obtain a homology score.

[0016] The homology score is compared with a high confidence threshold and a low confidence threshold. Regions with scores higher than the high confidence threshold are used as the basis for determining the homology region. Regions with scores lower than the low confidence threshold are used as candidates for the interference region to be eliminated. Regions between the low confidence threshold and the high confidence threshold and containing the threshold endpoint are sent to the homology verification.

[0017] Based on the results of the homology verification and the connection relationship between the edge and the main outline of the homology region, the edge compensation region and the interference region to be removed are determined, and the high confidence threshold is greater than the low confidence threshold.

[0018] Furthermore, the step of determining the edge compensation region and the interference region to be removed by combining the results of the homology verification with the connection relationship between the edge and the main outline of the homology region includes:

[0019] The regions sent to the homology verification are divided into sub-blocks. The homology score of the sub-blocks is determined according to the main features and transition features of the corresponding spatial sub-blocks. Sub-blocks with a homology score in the middle of the region that is higher than the high confidence threshold are marked as homology sub-blocks to be retained. Sub-blocks with a score that decreases relative to the homology sub-blocks to be retained at the region edge are marked as edge sub-blocks to be verified.

[0020] Along the boundary normal direction between the edge sub-block to be verified and the adjacent homologous sub-block to be retained, obtain the modal normalized pixel response mean sequence arranged from the inside of the main body to the outside, fit the overall change trend of the pixel response mean sequence to characterize the response decay trend, and determine the connection relationship based on the included angle of the adjacent contour segments and the endpoint spacing.

[0021] The edge sub-blocks to be verified that decrease in the direction away from the main body and can continue the outline of the main body are identified as the edge compensation region. The edge sub-blocks to be verified that do not exceed the preset trend threshold and cannot continue the outline of the main body are identified as the interference region to be eliminated.

[0022] Furthermore, the step of removing the confirmed interference regions to be removed includes:

[0023] For the pixels in the interference area to be removed, a re-determination is made based on whether the pixel is located in the transition zone outside the boundary of the building candidate area, the boundary connection relationship with the same source area, and the gradient along the extension direction of the main body contour. Pixels located in the transition zone, connected to the boundary of the same source area, and whose gradient meets the boundary retention condition are temporarily stored as boundary candidate pixels, and the remaining pixels confirmed as interference by the re-determination are removed.

[0024] The candidate pixels of the boundary are included in the candidate boundary. The consistency of the contour direction between the modal correction regions is verified. Local erosion or regional source separation is performed on the abnormal boundary segments located on the interference side. The corrected regional boundary is then redefined to eliminate residual interference from the candidate boundary.

[0025] Furthermore, the method also includes:

[0026] Before a local region is identified as the edge compensation region or the interference region to be removed and the corresponding processing is performed, a homology score is obtained based on the main features, edge features and transition features of the local region, and the distance between the score and the high confidence threshold and low confidence threshold of the homology determination, as well as the edge compensation support and interference removal support of the local region are determined.

[0027] When the distance between the homology score and any of the thresholds is less than a preset buffer value, or when both the edge compensation support and the interference removal support are greater than a preset conflict threshold, the local region is placed in a frozen review state, and the completion or removal based on a single round of judgment is temporarily suspended.

[0028] Continue to acquire verification results for the local area in the frozen verification state, restrict the effective output range of the local area from expanding outward, and retain only the edge band that is directly connected to the same source area and whose width does not exceed the preset limit value; when the exit condition is met, the frozen verification state is released, and corresponding supplementation or elimination is performed according to the confirmed area affiliation.

[0029] Furthermore, the step of continuing to obtain verification results for the local area in the frozen verification state includes:

[0030] Evaluate whether deleting the local area breaks the homologous region, and whether adding the local area causes the boundary to expand beyond a preset width into the adjacent hardened surface;

[0031] When there is only a risk of disconnecting the same source region, the temporary processing direction of the local region is set to be compensated; when there is only a risk of the expansion exceeding the preset width, the temporary processing direction is set to be eliminated; when both risks exist, the frozen review state is maintained and the region boundary is not modified.

[0032] The homogeneous regions adjacent to the local region are divided into main sub-blocks and the homogeneous state of each main sub-block is determined. The processing direction of the same local region in continuous verification is recorded. Only when two consecutive verifications point to the same processing direction and the homogeneous state of adjacent main sub-blocks remains stable, it is determined that the exit condition is met, and the local region is determined as the edge compensation region or the interference region to be removed according to the confirmed processing direction.

[0033] Furthermore, the method also includes:

[0034] For the local area in the frozen review state, if the exit condition is still not met after two back checks following the initial judgment, the switching between completion and elimination of the local area is stopped, and the minimum connected edge band is retained according to the effective output range to form disputed edge segment markers and corresponding boundary credibility.

[0035] The disputed edge segment marker and the effective output range are passed to the semantic fusion and building patch generation process. For high building probability pixels located outside the disputed edge segment, they are allowed to be included in the building patch only if the pixel is continuous with the same source region and does not cross the restriction boundary of the effective output range.

[0036] Furthermore, the step of defining the semantic fusion range of each modal image feature by the modified region, identifying building categories, and generating building patches includes:

[0037] Based on the modified region, effective region masks for each modality are formed. The main building part in the modified region is taken as the main building region. Semantic fusion is performed on the image features within the effective region mask to obtain a building probability map.

[0038] Pixels within the main building area whose building probability exceeds a preset category probability threshold are designated as internal pixels of the building patch. For edge candidate pixels located outside the main building area, it is first verified whether they form a continuous extension with the boundary of the main building area. Edge candidate pixels that pass the verification are included in the effective area mask. The building probability of the edge candidate pixel is recalculated or supplemented within the updated effective area mask. Then, it is determined whether to include it in the building patch based on the building probability.

[0039] Furthermore, the method also includes:

[0040] Calculate the proportion of the area removed by each mode during the process of filling in the edge compensation area and removing the confirmed interference area to be removed to the area of ​​its original building candidate area. Compare the proportions of each mode with the cross-modal correspondence. Record the two different modes in each comparison as the first mode and the second mode, respectively.

[0041] When the proportion of the first mode is higher than the proportion of the second mode and the difference exceeds a preset difference threshold, the regional features corresponding to the first mode are checked back. If the check determines that there are hardened ground texture features in the region corresponding to the first mode that are inconsistent with the building body, the fusion weight of the building probability of the first mode in the corresponding region is reduced, and the building patch is regenerated based on the refused building probability map and the effective region mask.

[0042] This application also provides a ground feature recognition system based on multimodal image data, used to perform the aforementioned ground feature recognition method.

[0043] include:

[0044] The candidate region acquisition module is used to acquire building candidate regions in at least two modal images within the same geographical area, establish cross-modal correspondence, and extract the main features, edge features, and transition features outside the boundary of each building candidate region;

[0045] A homogeneous region determination module is used to determine cross-modal homogeneous regions based on the main features;

[0046] The response trend determination module is used to obtain the modal normalized pixel response mean sequence arranged from the inside of the main body to the outside along the boundary normal direction between the edge sub-block to be determined in the candidate building area, and to fit the overall change trend of the sequence.

[0047] The contour continuity determination module is used to check the included angle and endpoint spacing between the edge sub-block and the adjacent contour segments of the same origin region based on the edge features and the transition features. The edge sub-block with the included angle lower than the preset angle tolerance and the endpoint spacing within the preset gap range is determined to be able to continue the main contour; otherwise, it is determined to be unable to continue the main contour.

[0048] The region attribution determination module is used to determine the edge sub-blocks whose overall change trend decays away from the main body and can continue the outline of the main body as edge compensation regions, and to determine the edge sub-blocks whose overall change trend response change amplitude outside the main body does not exceed a preset trend threshold and cannot continue the outline of the main body as interference regions to be eliminated.

[0049] The corresponding region correction module is used to retain the homogeneous region, fill in the edge compensation region, and remove the confirmed interference region to be removed to obtain the correction region for each mode.

[0050] The target recognition module is used to define the semantic fusion range of each modality image feature by the correction region, identify building categories and generate building patches.

[0051] This application provides a method and system for identifying ground features based on multimodal image data. According to the consistency of cross-modal subjects and the contour connection relationship, the main body of the building is preserved, weak response edges are filled in and adjacent interference is removed. Then, semantic fusion and patch generation are performed in the corrected area. This can reduce the mixing of non-building areas such as roads and courtyards, and preserve the building edges affected by shadows, so that the identified patches are more in line with the building range. Attached Figure Description

[0052] Figure 1 This is a flowchart of a ground feature recognition method according to one embodiment of this application.

[0053] Figure 2 This is a schematic diagram illustrating regional homology verification and edge attribution determination in one embodiment of this application.

[0054] Figure 3 This is a flowchart illustrating the candidate boundary cleanup and verification process of one embodiment of this application.

[0055] Figure 4 This is a flowchart illustrating the freeze review and exit process of one embodiment of this application.

[0056] Figure 5 This is a schematic diagram of building patch generation constraints according to one embodiment of this application.

[0057] Figure 6 This is a block diagram of a ground feature recognition system according to one embodiment of this application. Detailed Implementation

[0058] The embodiments of this application are used to illustrate specific implementations of the technical solutions and are part of the embodiments of this application. The specific configurations in the embodiments are used to explain the corresponding technical relationships and are not an exhaustive enumeration of the protection scope of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without inventive effort are within the protection scope of this application.

[0059] It should be noted that similar reference numerals and letters in the accompanying drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in other drawings. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0060] Reference Figure 1 This application provides a method for identifying ground features based on multimodal image data, comprising the following steps:

[0061] Reference Figure 1 This application provides a method for identifying ground features based on multimodal image data, comprising the following steps:

[0062] Step S101: Obtain candidate building regions from at least two modal images within the same geographical area, establish cross-modal correspondence, and extract the main features, edge features, and transition features outside the boundary of each candidate building region;

[0063] Step S102: Determine the cross-modal homologous region based on the main characteristics;

[0064] Step S103: For the edge sub-blocks to be determined in the candidate building area, along the boundary normal direction between the edge sub-block and the adjacent homogeneous region, obtain the modal normalized pixel response mean sequence arranged from the inside of the main body to the outside, and fit the overall change trend of the sequence.

[0065] Step S104: Based on edge features and transition features, check the angle between the edge sub-block and the adjacent contour segments of the same origin region and the distance between the endpoints. Edge sub-blocks with an angle lower than the preset angle tolerance and an endpoint distance within the preset gap range are determined to be able to continue the main contour; otherwise, they are determined to be unable to continue the main contour.

[0066] Step S105: Define the edge sub-blocks whose overall change trend decays away from the main body and can continue the outline of the main body as edge compensation areas, and define the edge sub-blocks whose overall change trend response change amplitude outside the main body does not exceed the preset trend threshold and cannot continue the outline of the main body as interference areas to be removed.

[0067] Step S106: Retain the homogeneous regions, complete the edge compensation regions, and remove the confirmed interference regions to be removed to obtain the corrected regions for each mode.

[0068] Step S107: Define the semantic fusion range of image features of each modality by modifying the region, identify building categories and generate building patches.

[0069] When a building's roof and courtyard form a salient area in one modality, while another modality primarily displays the roof, directly merging the two candidate areas will allow the courtyard to participate in building identification as well. Geometric registration can only adjust the location of the area; further differentiation between the roof and courtyard is still needed to determine the portion of the building that actually belongs to it in each modality.

[0070] This implementation modifies candidate regions before semantic fusion. First, it identifies parts belonging to the same building entity in different modalities based on the main features. Then, it combines edge and transition features to determine whether the local contour can be continuous. For edges with weak responses or those connected to the main entity, it combines contour extension to determine whether they belong to a building.

[0071] For weak edges caused by shadows, verify the overall attenuation trend of the response, the angle between contour segments, and the distance between endpoints according to steps S103 to S105. If they meet the edge compensation area criteria, they are included as compensation objects. For roads or courtyards, even if their response intensity is similar to that of the roof, it is still necessary to determine their area affiliation by combining the consistency of the main body, response changes, and contour continuity. Based on the above judgment, retain, supplement, or remove the corresponding areas respectively, and then use the corrected range for semantic fusion and patch generation.

[0072] In this implementation, buildings are used as ground features. Building patches represent the identifiable building area in the current image, which can be a pixel mask or a closed polygon formed by it. When the corresponding geometric positioning conditions are available, this area can be further associated with geographic coordinates. The building facade area in the vehicle side view image, the roof area in the remote sensing image, and the building's ground footprint outline are different. During output, the surface or visible part they represent are recorded respectively.

[0073] Multimodal images can be acquired by visible light and thermal infrared cameras mounted on unmanned vehicles, or by at least two of optical remote sensing images, synthetic aperture radar images, and thermal infrared images. For georeferenced remote sensing images, after radiometric correction, coarse geometric registration, and resampling, candidate region correspondences are established under a common georeference. For vehicle-mounted images, correspondences are established within a common visible range based on the acquisition time, vehicle pose, and camera calibration parameters. Local offsets are allowed for significant pixels in different modalities, but the regions included in the comparison should correspond to the same building object.

[0074] Candidate regions can be generated by a feature segmentation network or proposed by combining shape and spatial location after regional saliency detection. Boundary deviations are allowed for candidate regions, and their categories are further determined in subsequent semantic fusion. Main features can be relative brightness, texture variation, or stable response distribution within the region; edge features can be contour direction and continuity; transition features can be response changes from near to far outside the contour, as well as the local response of adjacent hardened surfaces and shadows.

[0075] The original values ​​for different modalities do not necessarily have the same physical units. For example, optical brightness and radar scattering intensity are first converted into relative statistical descriptions within their respective modes, and then their spatial distribution, relative response, and shape relationships are compared. In one implementation, each feature is standardized according to the mean and standard deviation of the corresponding modal training samples; if the standard deviation is too small, a preset positive lower limit is used to prevent division by zero in the constant region. The components used for cross-modal comparisons have the same descriptive meaning and a fixed arrangement, and each component is compared item by item according to the same feature meaning.

[0076] Region attribution can be determined on connected regions or on local strip regions defined by contour segments. The former is suitable for buildings where the main body is clear and adjacent parts are easily separated, while the latter is suitable for buildings where only one side of the contour is occluded. Both use the corresponding main body as the attribution reference and apply the local determination to the pixel range of each modality itself, without requiring the two modalities to use the exact same contour.

[0077] For regular contours, completion can determine missing edges by extending the straight line of the observed contour; for gently curving contours, missing segments can be determined by the local direction of the existing curve. Completion is only performed within areas where the response weakens and there is a basis for connection. If there is a lack of main reference or sufficient edge evidence, adjacent road edges cannot be automatically used as substitute building contours.

[0078] The cleanup area can be saved as a binary mask, or as the original pixel set plus the added and deleted pixel sets. When using a mask, features are extracted for each modality according to its own effective range; when adding or deleting records, an equivalent effective range is first generated from the original region and the records, and then participates in the fusion. Both representations maintain the identifier of the same geographic object, and the removed region no longer participates in the semantic fusion of that building.

[0079] For example, a candidate area might include the main roof structure, a narrow, shadowed edge on one side of the roof, and adjacent courtyards. The main structure has corresponding textures and contours in both modalities, with the shadowed edge continuing along the main structure's contour, while the courtyard's transition response extends along the external hardened surface. The system preserves the main structure, completes the shadowed edge by continuity, and excludes the courtyards. If only the intersection of the two modal regions is taken, weak edges might be lost; if the union is taken, courtyards might be included. Treating the shadowed edge and courtyards separately preserves the building's weak edges and reduces courtyard intrusion.

[0080] After correction, the image features within the effective range of each modality are fed into the semantic model for fusion. Based on the fusion result, the building category is identified, and the image range of the building is determined. The semantic model selects the appropriate network structure according to the input modality. The correction results of the candidate regions are used to limit the pixels participating in the recognition, while the building category is still determined by the fused image features.

[0081] In one embodiment of unmanned vehicle data acquisition, the unmanned vehicle acquires images of buildings on both sides of a park road or urban / rural road. A visible light camera and a thermal infrared camera with overlapping fields of view are fixedly mounted on the vehicle body, along with a positioning device, an inertial measurement unit, and an onboard processor. Before data acquisition, the intrinsic parameters, distortion parameters, relative positions, and attitudes of the two cameras are calibrated, and the installation relationship between the cameras and the vehicle body is determined. The onboard processor saves the images, timestamps, vehicle pose, and calibration parameters for subsequent pairing and identification.

[0082] During data acquisition, the unmanned vehicle can briefly stop at a predetermined acquisition location, and a common trigger signal will activate two cameras to expose and acquire visible light and thermal infrared images of the visible portion of the same building. The actual exposure times of both cameras are recorded together. If the image is blurry, the interval between two frames exceeds a preset tolerance, or the target is obscured by a passing vehicle, the set of images is reacquired. When using continuous driving acquisition, the images are first paired according to timestamps, and then the vehicle's pose changes are used to check whether the two frames still have a sufficient common visible range; images that do not meet the conditions are not included in cross-modal candidate region pairs.

[0083] Distortion correction is performed on the paired images, and candidate building regions are extracted. For near-planar building facades, local homography transformations can be estimated using the intersections of corresponding window corners, wall corners, or eaves in the two images. The thermal infrared candidate regions are then mapped to visible light image coordinates for pairing. When there are insufficient corresponding points such as window corners, the markers to be paired are retained and additional samples are taken. Local correspondences are established separately for road surfaces, facades, and eaves at different depths; transformations on one facade are not directly applied to the entire image. Edges with significant parallax that cannot be reliably paired are temporarily excluded from the same-origin determination.

[0084] For example, when an autonomous vehicle collects data on a building adjacent to a road, the visible light image shows a weak response at the edge of the facade covered by trees, while the thermal infrared candidate region includes adjacent hardened pavement with a similar response to the wall. The system first compares the main features inside the corresponding facade to determine the common source region, and then distinguishes between the weak building edges and the adjacent pavement based on the contour direction and the transition from the facade outward. For weak response edges that can continue the contour of a wall corner or eaves and have passed the main consistency verification, they are filled in; for parts whose external fitted response changes do not exceed a preset trend threshold and cannot continue the building contour, they are marked as interference to be removed and then removed after further judgment. The original responses of visible light and thermal infrared are normalized to form corresponding descriptions, and the gray values ​​of the two are not directly compared.

[0085] Each modality stores a corrected effective region mask, and semantic fusion is performed only on features within the mask to obtain the building category and its patch in the reference image. The vehicle-mounted processor can perform recognition on-site or send the image, pose, and calibration data to ground processing equipment for recognition. When the same building appears repeatedly at multiple acquisition locations, it is associated and recorded by combining geographical location, common visible facades, and contour correspondence, and the visible range of each image is retained; missing markers are retained for parts completely obscured by trees or vehicles.

[0086] The vehicle-mounted image patch is first saved as image pixel coordinates, and associated with the acquisition location and camera pose. When the 3D position of the building surface needs to be output, a ranging device that completes the extrinsic parameter calibration with the camera can be added to correlate the measured depth with the image patch pixels, and then the spatial coordinates are calculated based on the camera model and vehicle pose. When only the acquisition location and pose are available but the target depth is missing, the image patch and pose association record are saved; for unobserved roofs or building backs, the complete building footprint outline is not inferred from this.

[0087] In the implementation of the unmanned vehicle using frozen verification, the same set of paired images can be reused for review; newly acquired images need to be re-paired and confirmed to still correspond to the same building section before being used for verification. Verification records include associated building identifiers, local locations, and acquisition times; judgments based on consistent processing directions for two consecutive images refer to the same local area. The training samples for the vehicle-mounted model include facades, eaves, and adjacent road surfaces; training and inference use the same camera calibration and modal feature processing methods.

[0088] In one implementation, the determination of the homogeneous region, the edge compensation region, and the interference region to be removed includes:

[0089] For candidate building regions with cross-modal correspondence, the consistency of main body response, contour direction, and boundary transition mode are determined based on main body features, edge features, and transition features, respectively. The three are weighted according to preset weights to obtain the homology score.

[0090] The homology score is compared with the high confidence threshold and the low confidence threshold. Regions above the high confidence threshold are used as the basis for determining homology regions, and regions below the low confidence threshold are used as candidates for interference regions to be eliminated. Regions between the low confidence threshold and the high confidence threshold and containing the threshold endpoints are sent to homology verification.

[0091] By combining the results of homology verification with the connection relationship between the main outlines of the edge and homology region, the edge compensation region and the interference region to be removed are determined, and the high confidence threshold is greater than the low confidence threshold.

[0092] Main body response consistency is used to compare the description within the region, contour orientation consistency is used to compare the external shape, and boundary transition consistency is used to compare how the region transitions to the adjacent ground surface. All three are transformed into quantities where larger values ​​indicate greater consistency, and then weighted to form a homogeneity score. Contour orientation difference is first converted into a consistency value before being included in the weighting.

[0093] One scoring method uses S = 0.40A + 0.35E + 0.25T. S represents the homology score, A represents the consistency of the main response, E represents the consistency of the contour direction, and T represents the consistency of the boundary transition method; each value ranges from 0 to 1. The weights are non-negative and sum to 1, and can be statistically selected from historical regions labeled with homology and non-homology. The above weights represent an initial configuration.

[0094] Subject response consistency can be obtained by distance normalization of standardized subject vectors, for example, by converting the mean absolute component difference into similarity using a monotonically decreasing function. Contour direction sequences are arranged according to common geographical directions and the same traversal direction, using a consistent starting reference. Direction differences are averaged after periodic angular reflection to the minimum included angle, and then converted into consistency. Transition mode consistency compares the response direction sequences outward from the corresponding boundary segment, using the ratio of matching positions to the number of valid comparison positions. When a component lacks valid samples, the consistency of that component is not calculated, and a verification mark is retained for the corresponding region.

[0095] Let H be the high-confidence threshold and L be the low-confidence threshold, satisfying 0 ≤ L < H ≤ 1. When S > H, it provides evidence of subject homology; when S < L, it forms interference candidates; and when L ≤ S ≤ H, it requires verification. Unifying the threshold endpoints into the verification interval avoids the lack of processing branches for boundary values. Regions with lower scores are first marked as interference candidates, and their edge pixels are further judged based on the subject and contour connection relationship.

[0096] The threshold can be adjusted in a bounded manner based on the confidence level of region segmentation. For example, if the average segmentation probability of at least one candidate region is less than 0.6, H should be moderately increased, and L should be increased by a small margin; if two regions have large areas and significant differences in their main features, L can be decreased to allow more regions to be considered for review rather than directly becoming interfering candidates. After each adjustment, L should be kept < H and the threshold should remain within the scoring range. The segmentation probability and intra-block variance are used to select the adjustment amount, which must be calibrated by the corresponding modality samples. A large area cannot be used to directly infer that regions are necessarily homologous.

[0097] In one implementation, the step of determining the edge compensation region and the interference region to be removed, by combining the results of homology verification with the connection relationship between the edge and the main outline of the homology region, includes:

[0098] The regions sent for homology verification are divided into sub-blocks. The homology score of the sub-blocks is determined based on the main and transitional features of the corresponding spatial sub-blocks. Sub-blocks with homology scores in the middle of the region that are higher than the high confidence threshold are marked as homology sub-blocks to be retained. Sub-blocks with scores at the edge of the region that are lower than the homology sub-blocks to be retained are marked as edge sub-blocks to be verified.

[0099] Along the boundary normal direction between the edge sub-block to be reviewed and the adjacent homologous sub-block to be retained, obtain the modal normalized pixel response mean sequence arranged from the inside of the main body to the outside, fit the overall change trend of the pixel response mean sequence to characterize the response decay trend, and determine the connection relationship based on the included angle of adjacent contour segments and the distance between endpoints.

[0100] Edge compensation regions are defined as edge sub-blocks whose overall trend of change decays away from the main body and can continue the outline of the main body. Interference regions to be eliminated are defined as edge sub-blocks whose overall trend of change on the outside of the main body does not exceed the preset trend threshold and cannot continue the outline of the main body.

[0101] Reference Figure 2 Sub-blocks with a homology score higher than the high confidence threshold in the central region are designated as potential homology sub-blocks to be retained, while sub-blocks with a lower edge score relative to these potential homology sub-blocks are entered into the verification process. The verification process first obtains the modal-normalized pixel response mean sequence and fits the overall trend. Then, it determines whether the contour can be continued based on the angle between adjacent contour segments and the distance between endpoints. When the overall trend decays away from the main body and can continue the main body contour, it is identified as an edge compensation region; when the overall trend shows a response change amplitude outside the main body that does not exceed a preset trend threshold and cannot continue the main body contour, it is identified as an interference region to be removed. If the above classification conditions are not met, the verification mark is retained; interference is not determined solely by a weak response or failure to meet compensation conditions.

[0102] The bounding rectangle of the region is divided into positionally corresponding sub-blocks. The nominal side length of each sub-block can be one-third of the shorter side of the region, and is limited to between 16 and 64 pixels. Small regions or sub-blocks whose outermost size is less than the nominal size are calculated based on actual effective pixels, and blank padding is not considered as architectural features. Spatially corresponding sub-blocks are paired by geographical location, and the consistency between the main body and transition features is calculated separately within each pair of sub-blocks.

[0103] When most sub-blocks in the central part of the region score highly, while the scores of continuous edge sub-blocks decrease, the sub-blocks of the same origin to be retained are used as the main reference. For each edge sub-block to be reviewed, adjacent main sub-blocks are used first; if multiple exist, the main sub-block with the closest spatial distance and connected to the contour segment is selected. Sampling is performed along the normal of the common boundary or the nearest boundary of the two, from the inside to the outside of the main block, and the average response within the strip at each sampling position is calculated to obtain the ordered attenuation curve.

[0104] The attenuation judgment uses the modal response normalized within the modality. If the overall response weakens from the main body outwards, and the contour direction of the weak response portion can continue with the main body, then it has the support of a weak building edge. This implementation uses the outer platform response and the inability of the contour to continue as a joint judgment of interference: the overall change trend is fitted to the aforementioned sampling sequence, and the difference between the maximum and minimum values ​​of the fitted response values ​​at each sampling position outside the main body is taken as the change amplitude of the outer response; when this amplitude does not exceed a preset trend threshold, it is determined that the outer response forms a platform. The edge sub-blocks that form the above platform and cannot continue the contour of the main body are identified as interference areas to be removed. The trend threshold used for platform judgment is calibrated according to ground resolution and modal response and saved in the current processing configuration. The platform is not a sufficient condition to negate the building alone; the condition that the contour cannot continue must still be met simultaneously.

[0105] Specifically, the overall trend of the sampling sequence can be fitted, while checking whether the angle between adjacent contour segments is lower than the set tolerance and whether the distance between endpoints is within the allowable gap range. When local noise exists, a small number of sampling points are allowed to be non-monotonic to avoid deleting entire segments of real weak edges due to single-point bounces. The trend, angle, and gap thresholds are calibrated together with the ground resolution and modal response, and saved in the current processing configuration.

[0106] Edge compensation is performed in the coordinates of the current mode itself. A connecting line can be fitted to the two endpoints of the main contour separated by the gap, and the slope, curvature, and straightness trends are checked. If the preset range of change is met, a straight line or smooth curve is used for connection, and the filled portion is added to the corresponding correction area. The contour of the other mode is used to verify the continuity relationship; the boundary after filling is still determined in the current mode.

[0107] In one implementation, the step of removing identified interference regions includes:

[0108] For pixels in the interference area, a re-determination is made based on whether the pixel is located in the transition zone outside the boundary of the building candidate area, its connection relationship with the boundary of the same source area, and the gradient along the extension direction of the main outline. Pixels located in the transition zone, connected to the boundary of the same source area, and whose gradients meet the boundary retention conditions are temporarily stored as boundary candidate pixels, and the remaining pixels that are confirmed as interference after re-determination are removed.

[0109] Boundary candidate pixels are included in the candidate boundary, and the consistency of the contour direction between the modal correction regions is verified. Local erosion or regional source separation is performed on the abnormal boundary segments located on the interference side, and the corrected regional boundary is redefined to eliminate residual interference from the candidate boundary.

[0110] Reference Figure 3 After the interference areas are re-evaluated by pixels, they are divided into two processing paths: confirmed interference and boundary candidates. Boundary candidates need to undergo cross-modal contour verification. Only pixels that meet the boundary retention conditions are retained as building boundaries, and candidate pixels that fail the verification are still removed.

[0111] Pixel re-evaluation uses the original candidate region and classification label as input to check whether the pixel is located within the transition zone, whether it is connected to the determined subject boundary, and the gradient along the subject contour extension direction. The gradient intensity threshold can be selected according to the effective boundary gradient distribution. Pixels separated from the subject boundary, or pixels with insufficient response in the subject extension direction and no contour support, are identified as interference that can be removed.

[0112] The contours of each modal correction region are organized into contour chains along the same traversal direction and sampled at arc length intervals. Correspondences of nearby contour points with compatible orientations are established under common geographic coordinates, and the paired orientation differences are calculated. The average orientation difference is used to check overall consistency; locations exceeding a preset angle are recorded separately as abnormal boundary segments, and these local abnormal segments are processed separately.

[0113] If the anomalous segment is located on the original interference side, and the correction area is still connected to the hardened surface, then a local erosion amplitude of one pixel, two pixels, or determined by the transition zone is applied at the anomalous location. The erosion target is the local boundary adhered to the interference, rather than the entire main body shrinking simultaneously. For local mis-adhesion, the separation boundary is defined by combining the edge-side response changes and the texture continuity of the main body side, separating the building roof from the hardened road surface, and then a new outline is generated.

[0114] The cleanup record saves the original area, confirmed removal area, compensation area, correction area, and corresponding edge segment number. The same pixel is not counted in both retention and removal simultaneously; when a second correction occurs, the area is calculated based on the final set of valid pixels to avoid distortion of area proportions caused by multiple rounds of cumulative deletion. The original area object remains unchanged; corrections are generated through marking and recording, facilitating the tracing of the correction source.

[0115] In one implementation, the method further includes:

[0116] Before a local region is identified as an edge compensation region or an interference region to be removed and the corresponding processing is performed, a homology score is obtained based on the main features, edge features and transition features of the local region, and the distance between the score and the high confidence threshold and low confidence threshold of homology determination, as well as the edge compensation support and interference removal support of the local region are determined.

[0117] When the distance between the homology score and any threshold is less than the preset buffer value, or when both the edge compensation support and the interference removal support are greater than the preset conflict threshold, the local area is placed in a frozen review state, and the completion or removal based on a single round of judgment is temporarily suspended.

[0118] For local areas under frozen review, continue to obtain review results, restrict the effective output range of the local area from expanding outward, and only retain the edge band that is directly connected to the same source area and whose width does not exceed the preset limit value; when the exit condition is met, the frozen review state is lifted, and the corresponding supplementation or elimination is performed according to the confirmed area affiliation.

[0119] After a cleanup, boundary verification might reveal that an edge, once filled in, expands into the road, prompting its removal. However, removal might then cause a break in the main structure, triggering filling again. Filling and removal alter the contour, transition zone, and area ratio used in the next calculation, causing the same edge to repeatedly change between two results. To address this, before reaching the maximum number of backtracking checks, unstable local areas are frozen for verification.

[0120] During the freezing period, local scores are continuously calculated and contour continuity is determined, without supplementing or eliminating based on single-round results. Each frozen area retains a fixed area identifier, local location, and boundary at the time of freezing for continuous review.

[0121] The critical distance can be calculated as C = min(|S - H|, |S - L|). C is the closest distance between the current homology score S and the high confidence threshold H and the low confidence threshold L. All quantities use the same scoring scale, and H > L. When C is less than a preset buffer value, the system enters a freeze state. The same set of multimodal images can be used for each round of review.

[0122] Conflict intensity can be calculated as D = min(B, R). B represents edge compensation support, formed by edge adhesion, contour continuity, and main body connection strength; R represents interference removal support, formed by hardened surface features of the transition zone, outer platform response, and adjacent non-building features. Both B and R are normalized to 0 to 1. A D greater than the conflict threshold indicates that both tendencies are simultaneously strong. Their respective components can be synthesized with non-negative weights that sum to 1, and the parameters are determined using the same annotation configuration.

[0123] Aside from scoring thresholds or conflicting evidence, a switch in local states between two rounds, or the assumption that deletion would break the main body while patching would expand towards the road, can also trigger a freeze. Freeze determination can be made after regional scoring but before sub-block review, or after edge attribution is determined but before actual correction. In the former case, regional scoring is used, and sub-block scoring is unnecessary.

[0124] During the freezing process, the original pixels and the boundaries at the time of freezing are preserved for subsequent verification. The range participating in semantic fusion is determined by a constrained mask; no new outward-expanding boundaries are added, and the entire local area is not deleted. Only the minimum edge band directly connected to the confirmed subject and whose width does not exceed the limit value is retained. The minimum edge band can be selected from the subject boundary within the area before freezing, and the narrow band required to maintain local connectivity is chosen. The width is configured based on the pixel distance of the common grid or the ground distance. The original region record is saved separately.

[0125] In one implementation, the step of continuing to acquire review results for a local area that is in a frozen review state includes:

[0126] Evaluate whether deleting a local area breaks up the source region, and whether adding a local area causes the boundary to expand beyond the preset width into the adjacent hardened surface.

[0127] When there is only a risk of disconnecting the same source region, the temporary processing direction of the local region is set to be compensated; when there is only a risk of expansion exceeding the preset width, the temporary processing direction is set to be eliminated; when both risks exist, the frozen review state is maintained and the region boundary is not modified.

[0128] The regions with the same origin adjacent to the local region are divided into main sub-blocks and the same origin state of each main sub-block is determined. The processing direction of the same local region in continuous verification is recorded. Only when two consecutive verifications point to the same processing direction and the same origin state of adjacent main sub-blocks remains stable, the exit condition is determined to be met, and the local region is determined as an edge compensation region or an interference region to be removed according to the confirmed processing direction.

[0129] Reference Figure 4 After a freeze, two types of risk assessments are conducted on the affected area before continuous review. The "pending compensation" or "pending removal" statuses are only provisional results during the review process; appropriate actions are only taken after the exit conditions are met. The freeze continues if both risks exist simultaneously.

[0130] Deletion risk can be simulated by removing the local area to be judged in a temporary pixel set, comparing whether the connected components connected to the main body are separated before and after removal. Expansion risk can be simulated by filling in the local area in a temporary contour, measuring the maximum width increase along the outer normal to the adjacent hardened surface relative to the boundary when entering freeze mode. The simulation results are only used for risk assessment and do not modify the region boundary at this time.

[0131] Deletion alone causing a disconnection will be prioritized for compensation; addition alone causing exceeding the width limit will be prioritized for removal. If both risks exist, the system will remain frozen; if neither risk exists, a temporary direction will be determined based on the subject's acceptance and evidence of interference. If the evidence is still insufficient, the system will remain frozen. After determining the temporary direction, it is still necessary to check whether the results of two consecutive reviews are consistent.

[0132] Two consecutive reviews refer to two adjacent valid review records for the same local area both pointing to compensation or both pointing to removal; the homogeneous state of adjacent subject sub-blocks remains unchanged in these two records and meets the corresponding subject determination conditions. If the direction is compensated once and removed once, or the subject reference also changes, the exit condition is not met. During the freeze period, the original image can still be reread, the score and transition features updated to obtain new verification basis.

[0133] Each local area stores the previous round's state, the current round's state, and the number of state changes. When the number of state changes reaches a preset limit, that local area is prohibited from alternating between compensation and removal. If the number of backchecks has not reached two, review can continue within the effective frozen range; if the number of backchecks reaches two, it will be handled according to the following termination rules, and will no longer wait for a new review round. The number of state changes and the number of backchecks are recorded separately; the former is used to limit reversals, and the latter is used to limit the total number of review rounds.

[0134] In one implementation, the method further includes:

[0135] For local areas under frozen review status, if the exit conditions are not met after two back checks following the initial judgment, the switching between completion and removal of the local area is stopped, and the minimum connected edge band is retained according to the effective output range to form disputed edge segment markers and corresponding boundary credibility.

[0136] The disputed edge segment markers and effective output ranges are passed to the semantic fusion and building patch generation process. For high building probability pixels located outside the disputed edge segment, they are allowed to be included in the building patch only if the pixel is continuous with the same source region and does not cross the limit boundary of the effective output range.

[0137] The number of backtracking iterations begins after the initial assessment and is limited to a maximum of two rounds. Each round uses the region data from the beginning of the round to complete local analyses, then updates boundaries and records uniformly to avoid using data from different rounds for edges processed sequentially. If the exit conditions are not met by the end of the second round, termination and conservative output are executed first, and a third round is not initiated.

[0138] The disputed boundary segment records its start and end positions, the boundary at which it enters the freeze zone, the effective range limitation boundary, and its credibility level. Credibility levels can be set at different levels based on stable homologous entities, confirmed compensation boundaries, and unresolved dispute boundaries, or they can be numerically derived from calibration data. The credibility level of a disputed segment is lower than that of a stable boundary, but this level does not indicate a measured statistical accuracy.

[0139] Dispute markers and region masks are processed together in semantics. Even if the model generates a high building probability outside the disputed edge, it must still be checked that the building is continuous with the main body and does not cross the frozen boundary. Pixels that do not meet the above location conditions are not included in the patch. The re-determination, completion, and boundary verification of pixels in the frozen area are also limited to conservative limits.

[0140] When the edge width required to maintain connectivity exceeds the configuration limit and both constraints cannot be satisfied simultaneously, the confirmed subject and the restricted edge are retained, dispute and review markers are output, and the original width limit is preserved. Subsequent spatial interpretation can review this part based on the dispute markers.

[0141] In one implementation, the steps of defining the semantic fusion range of image features of each modality by modifying the region, identifying building categories, and generating building patches include:

[0142] Based on the correction region, effective region masks for each modality are formed. The main building part in the correction region is taken as the main building region. Semantic fusion is performed on the image features within the effective region mask to obtain the building probability map.

[0143] Pixels within the main building area whose building probability exceeds a preset category probability threshold are designated as internal pixels of the building patch. For edge candidate pixels located outside the main building area, it is first verified whether they form a continuous extension with the boundary of the main building area. Edge candidate pixels that pass the verification are included in the effective area mask. The building probability of the edge candidate pixel is recalculated or supplemented within the updated effective area mask. Then, it is determined whether to include it in the building patch based on the building probability.

[0144] Reference Figure 5 The algorithm generates patches based on building probabilities and an effective area mask. Pixels within the main building area that meet the probability threshold are included in the patches. For edge candidates already recorded outside the main area, the contour continuity is first verified, and if approved, the effective mask is updated before calculating the building probability. Pixels outside other areas are not included in the patches.

[0145] The semantic model uses image content and source state markers within the effective regions of each modality. When using rectangular image blocks as input, invalid locations within the block are masked, and the effective mask is simultaneously passed during feature extraction or fusion to avoid reintroducing rectangular cropping into the culling region. For newly added edge candidates that pass verification, the effective range is updated first, and then semantic features are recalculated or supplemented within the updated range to obtain their building probability.

[0146] When the output is a network score, it can be transformed into a building probability of 0 to 1 through the corresponding probability normalization transformation; when the output is already a probability, its meaning is preserved, and it is not arbitrarily stretched again according to the maximum and minimum values ​​of a single image. Valid pixels with probabilities higher than the category probability threshold are used to form a building mask, and the contours are traced according to the connected regions to form closed patches, while preserving the object identification.

[0147] Image continuity checks focus on holes, break lines, and abnormal narrowing. When these occur in a confirmed edge compensation segment, the fill width is increased within the allowable compensation range, and the effective mask and probability calculation are updated. When these occur on a side to be removed and the true edge is still uncertain, the removal range in that area is reduced, and a review record is retained to avoid unjustifiably cutting off the main body. External pixels that fail the acceptance verification must not be directly added.

[0148] For georeferenced raster images, pixel locations are mapped to projected coordinates using the corresponding geographic transformation, resulting in vector polygons. For vehicle-mounted side-view images, localization is performed according to the aforementioned camera model and depth conditions, or the image coordinates and acquisition pose are preserved. The output includes category, closed contour, pixel mask, homology score, corrected boundary confidence, percentage of culled and compensated areas, and start and end positions of disputed edge segments. A manual verification mark is added when the boundary confidence is below the configured lower limit; reliable patches provide usable masks.

[0149] In one implementation, the method further includes:

[0150] Calculate the proportion of the area removed by each mode during the process of filling in the edge compensation area and removing the confirmed interference area to be removed to the area of ​​its original building candidate area. Compare the proportions of each mode with cross-modal correspondence. Record the two different modes in each comparison as the first mode and the second mode, respectively.

[0151] When the proportion of the first mode is higher than that of the second mode and the difference exceeds the preset difference threshold, the regional features corresponding to the first mode are checked back. If the back check determines that there are hardened ground texture features in the region corresponding to the first mode that are inconsistent with the main building, the fusion weight of the building probability of the first mode in the corresponding region is reduced, and the building patch is regenerated based on the refused building probability map and the effective region mask.

[0152] The proportion of areas removed is uniformly calculated using the effective pixel area of ​​the original candidate building region in the modality as the denominator and the currently confirmed effective area of ​​removal as the numerator; objects with a denominator of zero are treated as having no effective candidates and are not included in the proportion comparison. Both modalities use the same geographic object and the same round of correction records to avoid mixing and comparing the areas of different buildings or different versions.

[0153] The difference in proportion only triggers feature back-lookup; removing modalities with larger areas may still retain effective architectural features. For example, if a modality's original candidate is wide, but the main body is clear after purification and no residual hardened surface texture is found, the original fusion weight of that modality is still retained. Only when the back-lookup finds that the hardened surface texture is inconsistent with the main building will the probability contribution of that modality in the corresponding area be reduced.

[0154] One bimodal re-fusion method uses P = w1P1 + w2P2. P1 and P2 are the building probabilities of the first and second modalities at their corresponding locations within the effective area, respectively, and w1 and w2 are non-negative and sum to 1. When it is confirmed that the first modality is subject to significant adjacent interference, the weight of w1 is reduced relative to the pre-retrospective weight, and w2 is adjusted accordingly; for example, if w1 was greater than 0.3 before the retrospective weight, it can be adjusted to w1 = 0.3 and w2 = 0.7; another configuration can be obtained by regression analysis of measured labeled data to obtain the weights. When only one modality is valid at the current location, only the probability of that valid modality is used, and the insufficient cross-modal evidence marker for that location is retained; invalid locations are not treated as zero-probability locations for averaging.

[0155] During re-fusion, previously removed regions are not included in the calculation. The fusion probability is still combined with the effective mask from the same round to generate patches, and the correction results are written back to the object record. Multiple modalities can be matched pairwise and then organized according to the common object identifier. The weights of each modality are normalized within the set of effective modalities. When the first and second modalities are matched separately with the second and third modalities, the homology of the corresponding local parts of the first and third modalities still needs to be verified.

[0156] This application also provides a ground feature recognition system based on multimodal image data, used to perform the aforementioned ground feature recognition method, including:

[0157] The candidate region acquisition module is used to acquire building candidate regions in at least two modal images within the same geographical area, establish cross-modal correspondence, and extract the main features, edge features, and transition features outside the boundary of each building candidate region;

[0158] The homogeneous region determination module is used to determine cross-modal homogeneous regions based on the main characteristics;

[0159] The response trend determination module is used to obtain the modal normalized pixel response mean sequence arranged from the inside to the outside of the main body for the edge sub-block to be determined in the candidate area of ​​the building, along the boundary normal direction between the edge sub-block and the adjacent homogeneous area, and fit the overall change trend of the sequence.

[0160] The contour continuity determination module is used to check the angle between the edge sub-block and the adjacent contour segments of the same origin region and the distance between the endpoints based on the edge features and transition features. The edge sub-block with the angle lower than the preset angle tolerance and the distance between the endpoints within the preset gap range is determined to be able to continue the main contour; otherwise, it is determined to be unable to continue the main contour.

[0161] The region attribution determination module is used to determine the edge sub-blocks whose overall change trend decays away from the main body and can continue the outline of the main body as edge compensation regions, and to determine the edge sub-blocks whose overall change trend response change amplitude outside the main body does not exceed the preset trend threshold and cannot continue the outline of the main body as interference regions to be eliminated.

[0162] The corresponding region correction module is used to retain the homogeneous region, fill in the edge compensation region, and remove the confirmed interference region to be removed, so as to obtain the correction region of each mode.

[0163] The target recognition module is used to define the semantic fusion range of image features of each modality by modifying the region, identify building categories and generate building patches.

[0164] Reference Figure 6 The candidate region acquisition module, the homologous region determination module, the response trend determination module, the contour contiguity determination module, the region attribution determination module, the corresponding region correction module, and the target recognition module correspond to steps S101 to S107, respectively. Each module transmits candidate regions and features, homologous regions, response change trends, contour contiguity determination results, region attribution, and corrected regions in the above order, and outputs building categories and building patches. These modules can be implemented by different program units on the same processor, or they can be distributed across processing devices that exchange region data.

[0165] In embodiments using georeferenced remote sensing imagery, the imagery is resampled to the same ground resolution and divided into blocks of 256×256 pixels or 512×512 pixels, with adjacent blocks retaining 16 or 32 pixels of overlap. This overlap is used to preserve building edges across blocks; cross-block output is recorded sequentially according to geographic extent and object correspondence, with overlapping segments of the same building being merged in the output. Each block records modal identifiers and geographic extent.

[0166] The candidate segmentation network outputs probabilities for building candidates, non-building hard surface candidates, shadow candidates, vegetation candidates, and background candidates. The candidate label map retains the main category and the maximum probability. If the area of ​​a building candidate for a certain modality is less than the configured value, for example, less than 50 pixels, the block is marked as a block to be supplemented and is not directly used as reliable cross-modal evidence; candidate regions with smaller areas are further verified in the supplementary processing.

[0167] Subject features may include mean brightness, texture gradient intensity, local variance, and stable response ratio. The stable response ratio can be configured with tolerance within a modality to achieve the ratio of the number of pixels in the main distribution area of ​​the subject to the number of effective pixels in the subject. Edge features record intensity, direction, continuity, and corner response attenuation. The transition band extends outward from the candidate contour by 3, 5, or 7 pixels; in one configuration with a 0.5-meter ground resolution, it is taken as 5 pixels, corresponding to approximately 2.5 meters.

[0168] The region object records modality number, block number, region unique identifier, original pixel set, three types of feature vectors, and area. State, edge compensation, culling, and boundary correction are each stored using separate fields; objects before and after cleanup are associated through the same identifier. During pairing, the location, outline, and circumscribed polygons within a common geographic area are compared, not just those with the closest centroid. One configuration uses a condition that the intersection area of ​​two circumscribed polygons exceeds 30% of the area of ​​the smaller circumscribed polygon to generate co-occurring region pairs.

[0169] For objects with insufficient overlap but intersecting outward contours, a supplementary candidate verification branch can be entered to check the direction of the outward contours and the narrow bridging sections between them. Regular hardened surfaces are excluded as erroneous bridging candidates, while narrow extensions with continuous support from the building contour are considered edge compensation candidates. After being judged for origin and continuity, they are then entered into the corresponding area. The supplementary candidate areas still need to undergo origin verification.

[0170] The training samples for the semantic model are matched with the cleansing range, retaining architectural pixel supervision annotations from manual or reliable sources, and adding three types of region cleanliness labels: regions with complete homogeneity, regions with edge compensation, and regions with residual interference. The cleanliness labels are used to distinguish sample conditions and do not replace architectural category supervision. Complete homogeneous samples support subject recognition, edge-compensated samples retain weak edge training locations, and residual interference samples retain non-architectural supervision, preventing the model from uniformly learning hard surfaces as buildings.

[0171] Before training, the input is cleaned based on homogeneous states and purification records, removing large areas of roads or courtyards that may be mixed in. If the area of ​​the removed sample exceeds 15% of the effective area of ​​the original sample, that sample will not be used for training and will be reprocessed before a decision is made on whether to use it. The network is trained by building pixel annotation, and the same region and mask organization method is used during inference. The model, feature normalization statistics, and threshold configuration are kept consistent with the version.

[0172] In the complete combined processing, when a region-level score is abnormal, the original image patch is re-verified; when boundary consistency is abnormal, local erosion or region source separation is performed; and when the fusion result shows single-modal contamination, the corresponding probability weights are adjusted. The above back-check results are saved to the corresponding region record and used to update local boundaries. The number of back-checks for the same region in each processing stage is combined and calculated, with a maximum of two rounds; if the requirements are still not met after the second round, an unstable marker is output and the back-check is stopped.

[0173] For critical edges, a freeze verification can be enabled simultaneously, subjecting the original edge patching, pixel re-judgment, and boundary verification to the effective range of the freeze. Scoring, conflict degree, temporary direction, number of state changes, and credibility of disputed segments are all transmitted as object records.

[0174] Abnormal inputs can form additional outputs independently. When one modality lacks candidates while another modality has clear candidates, the modality with candidates is used as a unidirectional benchmark, and local candidate supplementation is performed again within the image patch expanded beyond the geographical range corresponding to the missing modality. After successful supplementation, homology and edge classification are re-executed; if a reliable correspondence still cannot be formed, only the unidirectional recognition result is retained, and the cross-modal confidence is reduced, explicitly marking it as insufficient cross-modal evidence.

[0175] If almost no main pixels remain after purification, stop further elimination and revert to the candidate records before the current homologous classification. Obtain a low-confidence recognition result under the still valid boundary constraints, and add a high-uncertainty and verification mark. This abnormal result is marked separately from the normally purified building patches, and the mark indicating incomplete purification is retained.

[0176] The output retains the original objects and the source of boundary corrections to support feature tracing in spatial interpretation.

[0177] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for identifying ground features based on multimodal image data, characterized in that, Includes the following steps: Obtain candidate building regions from at least two modal images within the same geographic area, establish cross-modal correspondences, and extract the main features, edge features, and transition features outside the boundary of each candidate building region; Based on the aforementioned main characteristics, determine the cross-modal homogeneous region; For the edge sub-blocks to be determined in the candidate building area, along the boundary normal direction between the edge sub-block and the adjacent homogeneous region, obtain the modal normalized pixel response mean sequence arranged from the inside of the main body to the outside, and fit the overall change trend of the sequence. Based on the edge features and the transition features, check the angle between the edge sub-block and the adjacent contour segments of the same origin region and the distance between the endpoints. The edge sub-block with the angle lower than the preset angle tolerance and the distance between the endpoints within the preset gap range is determined to be able to continue the main contour; otherwise, it is determined to be unable to continue the main contour. The edge sub-blocks whose overall change trend decays away from the main body and can continue the outline of the main body are identified as edge compensation regions. The edge sub-blocks whose overall change trend response change amplitude on the outside of the main body does not exceed a preset trend threshold and cannot continue the outline of the main body are identified as interference regions to be eliminated. The original regions are retained, the edge compensation regions are filled in, and the confirmed interference regions to be removed are eliminated to obtain the correction regions for each mode. The semantic fusion range of image features of each modality is defined by the modified region to identify building categories and generate building patches.

2. The method according to claim 1, characterized in that, The determination of the homogeneous region, the edge compensation region, and the interference region to be eliminated includes: For a pair of candidate building regions with the aforementioned cross-modal correspondence, the consistency of main body response, the consistency of contour direction, and the consistency of boundary transition mode are determined based on the main body features, the edge features, and the transition features, respectively. The three are then weighted according to preset weights to obtain a homology score. The homology score is compared with a high confidence threshold and a low confidence threshold. Regions with scores higher than the high confidence threshold are used as the basis for determining the homology region. Regions with scores lower than the low confidence threshold are used as candidates for the interference region to be eliminated. Regions between the low confidence threshold and the high confidence threshold and containing the threshold endpoint are sent to the homology verification. Based on the results of the homology verification and the connection relationship between the edge and the main outline of the homology region, the edge compensation region and the interference region to be removed are determined, and the high confidence threshold is greater than the low confidence threshold.

3. The method according to claim 2, characterized in that, The step of determining the edge compensation region and the interference region to be removed by combining the results of the homology verification with the connection relationship between the edge and the main outline of the homology region includes: The regions sent to the homology verification are divided into sub-blocks. The homology score of the sub-blocks is determined according to the main features and transition features of the corresponding spatial sub-blocks. Sub-blocks with a homology score in the middle of the region that is higher than the high confidence threshold are marked as homology sub-blocks to be retained. Sub-blocks with a score that decreases relative to the homology sub-blocks to be retained at the region edge are marked as edge sub-blocks to be verified. Along the boundary normal direction between the edge sub-block to be verified and the adjacent homologous sub-block to be retained, obtain the modal normalized pixel response mean sequence arranged from the inside of the main body to the outside, fit the overall change trend of the pixel response mean sequence to characterize the response decay trend, and determine the connection relationship based on the included angle of the adjacent contour segments and the endpoint spacing. The edge sub-blocks to be verified that decrease in the direction away from the main body and can continue the outline of the main body are identified as the edge compensation region. The edge sub-blocks to be verified that do not exceed the preset trend threshold and cannot continue the outline of the main body are identified as the interference region to be eliminated.

4. The method according to claim 1, characterized in that, The step of removing the confirmed interference regions to be removed includes: For the pixels in the interference area to be removed, a re-determination is made based on whether the pixel is located in the transition zone outside the boundary of the building candidate area, the boundary connection relationship with the same source area, and the gradient along the extension direction of the main body contour. Pixels located in the transition zone, connected to the boundary of the same source area, and whose gradient meets the boundary retention condition are temporarily stored as boundary candidate pixels, and the remaining pixels confirmed as interference by the re-determination are removed. The candidate pixels of the boundary are included in the candidate boundary. The consistency of the contour direction between the modal correction regions is verified. Local erosion or regional source separation is performed on the abnormal boundary segments located on the interference side. The corrected regional boundary is then redefined to eliminate residual interference from the candidate boundary.

5. The method according to claim 1, characterized in that, The method further includes: Before a local region is identified as the edge compensation region or the interference region to be removed and the corresponding processing is performed, a homology score is obtained based on the main features, edge features and transition features of the local region, and the distance between the score and the high confidence threshold and low confidence threshold of the homology determination, as well as the edge compensation support and interference removal support of the local region are determined. When the distance between the homology score and any of the thresholds is less than a preset buffer value, or when both the edge compensation support and the interference removal support are greater than a preset conflict threshold, the local region is placed in a frozen review state, and the completion or removal based on a single round of judgment is temporarily suspended. Continue to acquire verification results for the local area in the frozen verification state, restrict the effective output range of the local area from expanding outward, and retain only the edge band that is directly connected to the same source area and whose width does not exceed the preset limit value; when the exit condition is met, the frozen verification state is released, and corresponding supplementation or elimination is performed according to the confirmed area affiliation.

6. The method according to claim 5, characterized in that, The step of continuing to obtain verification results for the local area in the frozen verification state includes: Evaluate whether deleting the local area breaks the homologous region, and whether adding the local area causes the boundary to expand beyond a preset width into the adjacent hardened surface; When there is only a risk of disconnecting the same source region, the temporary processing direction of the local region is set to be compensated; when there is only a risk of the expansion exceeding the preset width, the temporary processing direction is set to be eliminated; when both risks exist, the frozen review state is maintained and the region boundary is not modified. The homogeneous regions adjacent to the local region are divided into main sub-blocks and the homogeneous state of each main sub-block is determined. The processing direction of the same local region in continuous verification is recorded. Only when two consecutive verifications point to the same processing direction and the homogeneous state of adjacent main sub-blocks remains stable, it is determined that the exit condition is met, and the local region is determined as the edge compensation region or the interference region to be removed according to the confirmed processing direction.

7. The method according to claim 6, characterized in that, The method further includes: For the local area in the frozen review state, if the exit condition is still not met after two back checks following the initial judgment, the switching between completion and elimination of the local area is stopped, and the minimum connected edge band is retained according to the effective output range to form disputed edge segment markers and corresponding boundary credibility. The disputed edge segment marker and the effective output range are passed to the semantic fusion and building patch generation process. For high building probability pixels located outside the disputed edge segment, they are allowed to be included in the building patch only if the pixel is continuous with the same source region and does not cross the restriction boundary of the effective output range.

8. The method according to claim 1, characterized in that, The step of defining the semantic fusion range of each modal image feature by the modified region, identifying building categories and generating building patches includes: Based on the modified region, effective region masks for each modality are formed. The main building part in the modified region is taken as the main building region. Semantic fusion is performed on the image features within the effective region mask to obtain a building probability map. Pixels within the main building area whose building probability exceeds a preset category probability threshold are designated as internal pixels of the building patch. For edge candidate pixels located outside the main building area, it is first verified whether they form a continuous extension with the boundary of the main building area. Edge candidate pixels that pass the verification are included in the effective area mask. The building probability of the edge candidate pixel is recalculated or supplemented within the updated effective area mask. Then, it is determined whether to include it in the building patch based on the building probability.

9. The method according to claim 8, characterized in that, The method further includes: Calculate the proportion of the area removed by each mode during the process of filling in the edge compensation area and removing the confirmed interference area to be removed to the area of ​​its original building candidate area. Compare the proportions of each mode with the cross-modal correspondence. Record the two different modes in each comparison as the first mode and the second mode, respectively. When the proportion of the first mode is higher than the proportion of the second mode and the difference exceeds a preset difference threshold, the regional features corresponding to the first mode are checked back. If the check determines that there are hardened ground texture features in the region corresponding to the first mode that are inconsistent with the building body, the fusion weight of the building probability of the first mode in the corresponding region is reduced, and the building patch is regenerated based on the refused building probability map and the effective region mask.

10. A ground feature recognition system based on multimodal image data, used to perform the method of claim 1, characterized in that, include: The candidate region acquisition module is used to acquire building candidate regions in at least two modal images within the same geographical area, establish cross-modal correspondence, and extract the main features, edge features, and transition features outside the boundary of each building candidate region; A homogeneous region determination module is used to determine cross-modal homogeneous regions based on the main features; The response trend determination module is used to obtain the modal normalized pixel response mean sequence arranged from the inside of the main body to the outside along the boundary normal direction between the edge sub-block to be determined in the candidate building area, and to fit the overall change trend of the sequence. The contour continuity determination module is used to check the included angle and endpoint spacing between the edge sub-block and the adjacent contour segments of the same origin region based on the edge features and the transition features. The edge sub-block with the included angle lower than the preset angle tolerance and the endpoint spacing within the preset gap range is determined to be able to continue the main contour; otherwise, it is determined to be unable to continue the main contour. The region attribution determination module is used to determine the edge sub-blocks whose overall change trend decays away from the main body and can continue the outline of the main body as edge compensation regions, and to determine the edge sub-blocks whose overall change trend response change amplitude outside the main body does not exceed a preset trend threshold and cannot continue the outline of the main body as interference regions to be eliminated. The corresponding region correction module is used to retain the homogeneous region, fill in the edge compensation region, and remove the confirmed interference region to be removed to obtain the correction region for each mode. The target recognition module is used to define the semantic fusion range of each modality image feature by the correction region, identify building categories and generate building patches.

Images