Urban update intelligent monitoring method, system and platform based on double-branch cross fusion and multi-view dynamic attitude

Abstract

The invention discloses a city update intelligent monitoring method, system and platform based on double-branch cross fusion and multi-view dynamic attitude, and the method comprises the steps: generating and obtaining first data corresponding to a target city region, and generating corresponding second data in combination with cube projection; creating a first model corresponding to the first data, and based on the first model, generating corresponding third data in combination with a Cross-MTF feature fusion operator; establishing a second model corresponding to the first data, and generating corresponding fourth data; constructing a third model corresponding to the city, and generating corresponding fifth data based on the third model; based on the third data, the fourth data and the fifth data, performing fusion processing on the multi-view dynamic attitude index, and generating corresponding sixth data; wherein the sixth data is urban update thermodynamic diagram data, and the system and the platform corresponding to the method can realize automatic and refined perception and analysis of urban facade changes.

Description

TECHNICAL FIELD

[0001] The application belongs to the cross field of computer vision and urban geographic information technology, and particularly relates to an urban renewal intelligent monitoring method, system and platform based on double-branch cross fusion and multi-view dynamic degree. BACKGROUND

[0002] At present, the existing urban renewal monitoring technology mainly relies on remote sensing images and artificial field investigation, and has many limitations.

[0003] Limitations of remote sensing images: Although it has wide observation capability, it is difficult to capture the fine changes of urban micro-built environment (such as building facade and street facilities), and is significantly affected by cloud cover.

[0004] Difficulties in change detection and semantic understanding: Traditional technology cannot efficiently detect street view changes and understand semantic information, cannot accurately locate the fine changes of street view facade, and cannot distinguish binary change results, static or flowing elements, resulting in inaccurate perception and analysis of urban renewal.

[0005] Defects in feature fusion strategy: In the street view image change detection method based on convolutional neural network, the pre-fusion and post-fusion strategy can only fuse part of the information, affecting the detection performance; although the multi-level feature fusion strategy has been improved, the single-branch structure has problems, and the exchange of image order will produce inconsistent detection results, which is not conducive to change splitting, and limits the development of weakly supervised semantic change detection.

[0006] Complex data set and training: The previous weakly supervised street view semantic change detection model is limited by the single-branch feature fusion method, and needs a complex special data set generation and training process for change splitting, increasing the research cost and implementation difficulty.

[0007] Lack of dynamic indicators: The existing land use dynamic degree model ignores the internal changes of the same category (such as building renovation), and does not combine with the street view perspective, making it difficult to reflect the details of urban renewal.

[0008] Therefore, in view of the above technical problems and defects, it is urgent to design and develop an urban renewal intelligent monitoring method, system and platform based on double-branch cross fusion and multi-view dynamic degree. SUMMARY

[0009] In order to overcome the deficiencies and difficulties existing in the prior art, the purpose of the present application is to realize end-to-end weakly supervised change segmentation through a double-branch cross-fusion model (Cross-C2PO), combine a differentiable union operator and a multi-view dynamic degree index, improve the detection accuracy and efficiency of urban facade changes, conduct statistical analysis and visualization of multi-functional area update change detection, and intuitively display the distribution of urban physical changes to help developers identify hot areas and provide a reference for the formulation of update strategies, and provide an important method and case study for the intelligent combination of street view and computer vision applications.

[0010] The first purpose of the present application is to provide an urban renewal intelligent monitoring method based on double-branch cross-fusion and multi-view dynamic degree; the second purpose of the present application is to provide an urban renewal intelligent monitoring system based on double-branch cross-fusion and multi-view dynamic degree; and the third purpose of the present application is to provide an urban renewal intelligent monitoring platform based on double-branch cross-fusion and multi-view dynamic degree.

[0011] The first purpose of the present application is achieved by the method comprising the following steps:

[0012] First data corresponding to a target urban area is generated and obtained, and based on the first data, corresponding second data is generated in combination with a cube projection; wherein the first data is a double-time phase street view image pair data; and the second data is multi-view perspective view data;

[0013] A first model corresponding to the first data is created, and based on the first model, corresponding third data is generated in combination with a Cross-MTF feature fusion operator; wherein the first model is a double-branch cross-fusion model Cross-C2PO; and the third data is change mask data;

[0014] A second model corresponding to the first data is established, and based on the second model, corresponding fourth data is generated; wherein the second model is a pre-trained semantic segmentation model; and the fourth data is a semantic change area;

[0015] A third model corresponding to the city is constructed, and based on the third model, corresponding fifth data is generated; wherein the third model is an urban renewal dynamic degree model; and the fifth model is an update intensity index data;

[0016] Based on the third data, the fourth data and the fifth data, a multi-view dynamic degree index is fused and processed, and corresponding sixth data is generated; wherein the sixth data is urban renewal heat map data.

[0017] Further, the first data corresponding to the target urban area is generated and acquired, and based on the first data, corresponding second data is generated by combining cube projection, and the second data further comprises:

[0018] Based on the first data, front, rear, left and right multi-view perspective view data are respectively generated by cube projection.

[0019] Further, the first model corresponding to the first data is created, and based on the first model, corresponding third data is generated by combining a Cross-MTF feature fusion operator, and the third data further comprises:

[0020] Based on the third data, corresponding seventh data is generated by combining a differentiable union set operator; wherein the seventh data is joint change mask data;

[0021] The change mask data output by the double-branch is subjected to channel-level minimum or maximum screening processing, and end-to-end weak supervision training is realized through a gradient reservation mechanism.

[0022] Further, the double-branch cross-fusion model Cross-C2PO comprises:

[0023] A symmetrical double-branch architecture processes t0→t1 and t1→t0 contrast sequences respectively, an improved Cross-MTF feature fusion operator, and a feature pyramid decoder for fusing multi-scale feature maps.

[0024] Further, the second model corresponding to the first data is established, and based on the second model, corresponding fourth data is generated, and the fourth data further comprises:

[0025] The panoramic semantic change detection result is converted into four-direction perspective view data;

[0026] The eighth data corresponding to the dynamic degree of each view angle is respectively calculated and generated; wherein the eighth data is index data;

[0027] Based on the eighth data, corresponding ninth data is generated by weighted fusion processing; wherein the ninth data is comprehensive dynamic degree index data.

[0028] The second object of the application is achieved in that: the system is used to realize the urban renewal intelligent monitoring method based on double-branch cross-fusion and multi-view dynamic degree; the system comprises:

[0029] A first data generation unit is configured to generate and acquire first data corresponding to a target urban area, and based on the first data, generate corresponding second data by combining cube projection; wherein the first data is a double-time street view image pair data; and the second data is multi-view perspective view data.

[0030] The first model construction unit is configured to create a first model corresponding to the first data, and generate third data corresponding to the first model based on the first model and in combination with a Cross-MTF feature fusion operator; wherein the first model is a double-branch cross-fusion model Cross-C2PO; and the third data is change mask data.

[0031] The second model construction unit is configured to establish a second model corresponding to the first data, and generate fourth data corresponding to the second model based on the second model; wherein the second model is a pre-trained semantic segmentation model; and the fourth data is a semantic change region.

[0032] The third model construction unit is configured to construct a third model corresponding to the city, and generate fifth data corresponding to the third model based on the third model; wherein the third model is a city update dynamic degree model; and the fifth model is update intensity index data.

[0033] The second data generation unit is configured to fuse and process multi-view dynamic degree indexes based on the third data, the fourth data, and the fifth data, and generate sixth data corresponding thereto; wherein the sixth data is city update heat map data.

[0034] Further, the first data generation unit further comprises:

[0035] The first generation module is configured to generate front, rear, left, and right multi-view perspective data based on the first data through cubic projection.

[0036] And / or, the first model construction unit further comprises:

[0037] The second generation module is configured to generate seventh data corresponding to the third data in combination with a differentiable union operator; wherein the seventh data is joint change mask data.

[0038] The first processing module is configured to perform channel-level minimum or maximum filtering processing on the change mask data output by the double-branch, and implement end-to-end weak supervision training through a gradient reservation mechanism.

[0039] And / or, the second model construction unit further comprises:

[0040] The second processing module is configured to convert panoramic semantic change detection results into four-direction perspective data.

[0041] The third generation module is configured to calculate and generate eighth data corresponding to the dynamic degree of each view; wherein the eighth data is index data.

[0042] The third processing module is configured to generate corresponding ninth data through weighted fusion processing based on the eighth data; wherein the ninth data is a comprehensive dynamic index data.

[0043] Further, the double-branch cross-fusion model Cross-C2PO comprises: a symmetrical double-branch architecture for processing contrast sequences of t0→t1 and t1→t0 respectively; an improved Cross-MTF feature fusion operator; and a feature pyramid decoder for fusing multi-scale feature maps.

[0044] A third object of the present application is achieved by comprising a processor, a memory, and a double-branch cross-fusion and multi-view dynamic urban renewal intelligent monitoring platform control program; wherein the processor executes the double-branch cross-fusion and multi-view dynamic urban renewal intelligent monitoring platform control program, the double-branch cross-fusion and multi-view dynamic urban renewal intelligent monitoring platform control program is stored in the memory, and the double-branch cross-fusion and multi-view dynamic urban renewal intelligent monitoring platform control program implements the double-branch cross-fusion and multi-view dynamic urban renewal intelligent monitoring method.

[0045] The present application generates and obtains first data corresponding to a target urban area by a method, and generates corresponding second data based on the first data in combination with a cube projection; wherein the first data is a double-time phase street view image pair data; the second data is multi-view perspective view data; a first model corresponding to the first data is created, and corresponding third data is generated based on the first model in combination with a Cross-MTF feature fusion operator; wherein the first model is a double-branch cross-fusion model Cross-C2PO; the third data is change mask data; a second model corresponding to the first data is established, and corresponding fourth data is generated based on the second model; wherein the second model is a pre-trained semantic segmentation model; the fourth data is a semantic change area; a third model corresponding to the city is constructed, and corresponding fifth data is generated based on the third model; wherein the third model is an urban renewal dynamic model; the fifth model is an update intensity index data; multi-view dynamic indicators are fused based on the third data, the fourth data, and the fifth data, and corresponding sixth data is generated; wherein the sixth data is urban renewal heat map data, and a system and a platform corresponding to the method can realize automatic and fine perception and analysis of urban facade changes.

[0046] That is, the present application provides significant technical support and supplement to existing urban renewal monitoring technology by integrating the street view image classification and change detection model Cross-C2PO, including:

[0047] Dual-branch cross-fusion model: a novel dual-branch architecture is proposed, which realizes end-to-end weakly supervised change split by cross-comparing time image pairs (t0→t1 and t1→t0) and combining an improved Cross-MTF feature fusion operator. The model solves the input order ambiguity problem caused by the feature fusion operator not satisfying the commutative law in traditional single-branch structure, simplifies the semantic change detection process, and does not need to rely on synthetic data set training, significantly improving the detection efficiency and robustness.

[0048] Differentiable union operator: a differentiable union operator is designed to ensure that the joint change mask covers all predicted change areas while maintaining gradient propagation consistency. This technique breakthrough solves the inconsistency problem of change masks in weakly supervised training, providing mathematical guarantee for model optimization.

[0049] Multi-view urban renewal dynamic degree model: existing land use dynamic degree models ignore intra-class changes and do not combine with street view perspectives, making it difficult to reflect urban renewal details. The invention constructs a street view data-oriented urban renewal dynamic degree index, integrates street view change points and dynamic degree to form a multi-dimensional representation of urban facade change characteristics, conducts panoramic map change pattern and front, rear, left and right view local analysis of urban function renewal intensity, and clarifies the connotation and denotation of urban street view change and index calculation. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0051] Figure 1 The weakly supervised street view semantic change detection process schematic diagram of the present application scheme;

[0052] Figure 2 (a) Previous binary change detection model architecture; (b) Cross-C2PO model architecture; (c) Cross-C2PO feature fusion operator; (d) Cross-C2PO differentiable union operator;

[0053] Figure 3 The semantic segmentation result instance schematic diagram of the present application scheme;

[0054] Figure 4 The overall process schematic diagram of urban renewal detection and analysis of the present application scheme;

[0055] Figure 5(left) City renewal dynamic degree map based on panoramic street view perception; (right) City renewal four-direction view (front (a), back (b), left (c), right (d)) based on panoramic street view perception;

[0056] Figure 6 For the present application, a flow chart of an intelligent monitoring method for city renewal based on double-branch cross fusion and multi-view dynamic degree is shown.

[0057] Figure 7 For the present application, a schematic diagram of the architecture of an intelligent monitoring system for city renewal based on double-branch cross fusion and multi-view dynamic degree is shown.

[0058] Figure 8 For the present application, a schematic diagram of the architecture of an intelligent monitoring platform for city renewal based on double-branch cross fusion and multi-view dynamic degree is shown. DETAILED DESCRIPTION

[0059] In order to better understand the purpose, technical solutions and advantages of the present application, the present application will be further described below in combination with the drawings and specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the present description.

[0060] The present application can also be implemented or applied through other different specific examples, and various modifications and changes can be made to the details in the present description based on different views and applications without departing from the spirit of the present application.

[0061] It should be noted that if the present application embodiments involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative positional relationship, motion condition, etc. between the components in a certain specific posture (as shown in the drawings), and if the specific posture changes, the directional indications also change accordingly.

[0062] In addition, if the present application embodiments involve descriptions of "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. Secondly, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of a person skilled in the art, and when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0063] Preferably, the urban renewal intelligent monitoring method based on double-branch cross fusion and multi-view dynamic degree of the present application is applied in one or more terminals or servers. The terminal is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and the hardware thereof includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0064] The terminal can be a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The terminal can interact with a user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device and the like.

[0065] The present application is to realize an urban renewal intelligent monitoring method, system and platform based on double-branch cross fusion and multi-view dynamic degree.

[0066] As shown in Figure 6 FIG. 1 is a flowchart of the urban renewal intelligent monitoring method based on double-branch cross fusion and multi-view dynamic degree provided by the embodiment of the present application.

[0067] In the embodiment, the urban renewal intelligent monitoring method based on double-branch cross fusion and multi-view dynamic degree can be applied in a terminal with display function or a fixed terminal, and the terminal is not limited to a personal computer, a smart phone, a tablet computer, a desktop computer or an all-in-one computer with a camera, etc.

[0068] The urban renewal intelligent monitoring method based on double-branch cross fusion and multi-view dynamic degree can also be applied in a hardware environment composed of a terminal and a server connected to the terminal through a network. The network includes but is not limited to a wide area network, a metropolitan area network or a local area network. The urban renewal intelligent monitoring method based on double-branch cross fusion and multi-view dynamic degree of the embodiment of the present application can be executed by the server, or can be executed by the terminal, or can be executed by the server and the terminal together.

[0069] For example, for the urban renewal intelligent monitoring terminal based on the double-branch cross fusion and multi-view dynamic degree, the method provided by the present application can be directly integrated on the terminal to provide the urban renewal intelligent monitoring function based on the double-branch cross fusion and multi-view dynamic degree, or a client for implementing the method of the present application can be installed. For another example, the method provided by the present application can also run on a server or the like in the form of a software development kit (SDK), and the SDK can provide an interface for the urban renewal intelligent monitoring function based on the double-branch cross fusion and multi-view dynamic degree, so that a terminal or other device can implement the urban renewal intelligent monitoring function based on the double-branch cross fusion and multi-view dynamic degree through the interface. The present application will be further described below with reference to the accompanying drawings.

[0070] As shown in Figures 1-6 , the present application provides an urban renewal intelligent monitoring method based on double-branch cross fusion and multi-view dynamic degree, which comprises the following steps:

[0071] S1, generating and acquiring first data corresponding to a target urban area, and generating corresponding second data based on the first data and in combination with a cube projection; wherein the first data is a double-time-phase street view image pair data; and the second data is multi-view perspective view data;

[0072] S2, creating a first model corresponding to the first data, and generating corresponding third data based on the first model and in combination with a Cross-MTF feature fusion operator; wherein the first model is a double-branch cross fusion model Cross-C2PO; and the third data is change mask data;

[0073] S3, establishing a second model corresponding to the first data, and generating corresponding fourth data based on the second model; wherein the second model is a pre-trained semantic segmentation model; and the fourth data is a semantic change area;

[0074] S4, constructing a third model corresponding to the city, and generating corresponding fifth data based on the third model; wherein the third model is an urban renewal dynamic degree model; and the fifth model is an update intensity index data;

[0075] S5, fusing and processing multi-view dynamic degree indexes based on the third data, the fourth data and the fifth data, and generating corresponding sixth data; wherein the sixth data is an urban renewal heat map data.

[0076] The generating and acquiring first data corresponding to a target urban area, and generating corresponding second data based on the first data and in combination with a cube projection further comprises:

[0077] S11, respectively generate front, rear, left and right multi-view perspective data based on the first data through cubic projection.

[0078] The first model corresponding to the first data is created, and the third data corresponding to the first model is generated based on the first model and in combination with a Cross-MTF feature fusion operator.

[0079] S21, generate corresponding seventh data in combination with a differentiable union operator based on the third data; wherein the seventh data is joint change mask data.

[0080] S22, perform channel-level minimum or maximum filtering processing on the change mask data output by the double-branch, and realize end-to-end weak supervision training through a gradient reservation mechanism.

[0081] The double-branch cross-fusion model Cross-C2PO includes: a symmetrical double-branch architecture for processing contrast sequences t0→t1 and t1→t0 respectively; an improved Cross-MTF feature fusion operator; and a feature pyramid decoder for fusing multi-scale feature maps.

[0082] The second model corresponding to the first data is established, and the fourth data corresponding to the second model is generated based on the second model.

[0083] S31, convert the panoramic semantic change detection result into four-direction perspective data;

[0084] S32, respectively calculate the eighth data corresponding to the dynamic degree of each view; wherein the eighth data is index data.

[0085] S33, generate corresponding ninth data through weighted fusion processing based on the eighth data; wherein the ninth data is comprehensive dynamic degree index data.

[0086] Specifically, in the embodiment of the present application, an urban renewal monitoring technology integrating street view image classification and change detection is provided, including the following steps:

[0087] Data acquisition and preprocessing: the present application uses 360° panoramic street view images collected at fixed intervals along the urban road network for multiple periods. In order to obtain the urban renewal dynamic degree of the four directions of the street, the change detection result of the panoramic image is converted into cubic projection to obtain the perspective view of the six views of front, rear, left, right, up and down. This conversion effectively reduces the image distortion of the street view, making the image closer to the real view. Data enhancement processing is performed, i.e. the image is rotated, windowed and normalized to construct the training and test set.

[0088] Street view weakly supervised semantic change detection: Since the existing binary change detection method cannot complete the change splitting step, the previous weakly supervised semantic change detection method needs to couple the change splitting and semantic segmentation two sub-problems through a carefully designed network Figure 1 b), however, this coupling makes the network design difficult to solve the general segmentation problem, and at the same time needs to create a synthetic dataset containing the labels required by the two sub-problems for training from scratch, which further increases the complexity of this step, including class sampling and four morphological operations (erosion, dilation, opening, closing).

[0089] The present application proposes a new weakly supervised semantic change detection process. Specifically, as shown in Figure 1 c, the present application couples binary change detection and change splitting, and proposes Cross-C2PO model to simultaneously process the two sub-problems, Cross-C2PO is an easy-to-migrate architecture, and the current binary change detection method can be easily migrated to this architecture to realize end-to-end weakly supervised change splitting without introducing M0, M1 label information. This allows us to make full use of the most advanced semantic segmentation models and their pre-trained weights, such as DeepLab, SegFormer, SAM to get the semantic segmentation results of the image pair, and finally only keep the change area position indicated by the change splitting mask M0, M1 obtained by Cross-C2PO to get the final semantic change detection result S0, S1.

[0090] Cross-C2PO model structure: the core of the model is to realize end-to-end weakly supervised change splitting through a double-branch architecture, and to migrate C-3PO to this architecture to finally form the proposed Cross-C2PO Figure 2 b). Specifically, the present application defines the change detection task as a process of finding relative changes through cross comparison: the comparison between t0 and t1 images represents the change of t0 relative to t1, to represent the changed objects in the t0 image, and vice versa, the comparison between t1 and t0 images represents the change of t1 relative to t0, to represent the changed objects in the t1 image, which means that two different comparison orders are regarded as different detection processes. Cross-C2PO uses VGG-16 as an encoder to extract features of the input image pair, and generates multi-scale feature maps F0, F1 from 1 / 4 to 1 / 32 resolution. Subsequently, the present application sets up two branches to detect relative changes, in the upper branch, fuse(F0, F1) is used for feature fusion to represent the feature difference of F0 relative to F1, and in the lower branch, fuse(F1, F0) is used to represent the feature difference of F1 relative to F0, obviously, at this time, it is necessary to explicitly specify that the fuse function does not satisfy the commutative law such that fuse(X, Y)≠ fuse(Y, X), otherwise it would result in two branches having the same output. Since Cross-C2PO is an architectural migration version of C-3PO, its decoder, FPN, is directly utilized: FPN fuses features of different spatial scales, effectively improving the detection capability of objects of different sizes, and has been widely applied in tasks such as object detection and semantic segmentation. The fused feature maps are respectively passed through the decoder to obtain change maps m0, m1∈R 2×H×W .

[0091] Binary change detection divides each pixel into two classes, class 0 is unchanged (background), class 1 is changed, so m0, m1 contains two channels respectively representing the response values of the unchanged class and the changed class. The change masks of the image pair (ChangeMask) M0, M1∈{0, 1} H×W are obtained by the following formula:

[0092] M 0,1 = maxIndex(m 0,1 ) #(1)

[0093] where maxIndex represents the maximum value index along the channel dimension, that is, the class index of the maximum value, which is also called the binary operation. The invention expects to maintain the correct consistency relationship between the change masks: the changed area in M is equal to the union set of the changed areas in M0 and M1, and a differentiable union operator is designed to generate a joint change map m to maintain this consistency relationship to make the gradient correctly propagate to realize weakly supervised training Figure 2 d) :

[0094]

[0095] where [·] represents concatenation along the channel direction, represents the i-th channel of m {0,1} . Equation (2) takes the minimum unchanged response value of m0, m1 at each position as the unchanged response value of m, and the maximum changed response value as the changed response value of m, which will ensure that the changed area in m can cover all areas predicted as changed in m0 and m1.

[0096] The MTF module proposed by C-3PO divides the change into "exchange", "disappear", "appear" Figure 5 ) and uses three different functions to detect Figure 2 c) :

[0097] Exchange = ReLU(max(F0, F1) - min(F0, F1)) #(3)

[0098] Disappear = ReLU(F0 - F1) # (4)

[0099] Appear = ReLU(F1 - F0) # (5)

[0100] MTF can be expressed as:

[0101] MTF(F0, F1) = Conv(Exchange) + Conv(Disappear + Appear) + Conv(t0Info + t1Info) # (6)

[0102] where Conv represents a convolution operation with a window size of 3x3, t0Info = ReLU(F0), t1Info = ReLU(F1) are used to provide auxiliary information. It can be observed that MTF does not satisfy our requirement for the fuse function, because MTF satisfies the commutative law, i.e., MTF(F0, F1) = MTF(F1, F0). Therefore, we propose a modified version of MTF, Cross-MTF, as the fuse function of the network Figure 4 c).

[0103] Specifically, the reason why equation (3) satisfies the commutative law lies in the symmetry of Disappear and Appear, t0Info and t1Info, and our proposed architecture also has two symmetric branches, so they can be merged into two functions:

[0104] Disappear(X, Y) = ReLU(X - Y) # (7)

[0105] Info(X, Y) = ReLU(X) # (8)

[0106] Finally, Cross-MTF is expressed as:

[0107] Cross-MTF(X, Y) = Conv(Exchange) + Conv(Disappear(X, Y)) + Conv(Info(X, Y)) # (9)

[0108] The sub-functions contained in the original MTF will be scattered into the upper and lower branches with different input orders, and Cross-MTF does not satisfy the commutative law. This modification has an interpretable meaning: under the cross-contrast view, t0, t1 images will be the reference object respectively, at this time, "appearance" and "disappearance" belong to the same kind of changes, such as Figure 2 c the red box shows that the object disappears at t0 time is equivalent to the object appears at t1 time.

[0109] Semantic segmentation and classification: Pre-trained model transfer: DeepLabV3+(Cityscapes dataset pre-training) is used for semantic segmentation of street view images to extract building, road, vegetation and other land object categories. Semantic change detection: Combine the change mask (M0, M1) output by Cross-C2PO with the semantic segmentation result to filter out moving elements (vehicles, pedestrians) and generate semantic change areas (S0, S1).

[0110] City dynamic model: Land use dynamic is used to represent the rate of land resource change to describe the degree of land use / land cover change, which is mainly divided into single land use type dynamic and comprehensive land use dynamic. Comprehensive land use dynamic represents the overall change rate of land use in a study area. In this paper, the comprehensive land use dynamic model is introduced into the street-level urban renewal perception, and a city dynamic model is proposed to finely perceive the degree of urban change.

[0111] The formula of the comprehensive land use dynamic model is as follows:

[0112]

[0113] Among them:

[0114]

[0115] U ij represents the total area of the i-th land use type converted to the j-th land use type in the monitoring period, U i represents the area of the i-th land type at the beginning of the study period, T represents the duration of the study, and if T is in years, LC represents the annual change rate of comprehensive land use in the study area. As can be seen from equation (11), the land use dynamic model does not study the changes within the category, but in urban renewal research, there are many changes between the same categories, such as changes in advertising content, building renovation, etc. Therefore, the restriction should be removed in the city dynamic model.

[0116] The formula of the city dynamic model proposed in the present application is as follows:

[0117]

[0118] In order to remove the interference of urban mobile elements, the t0, t1 semantic change detection results are used to remove vehicles and pedestrian categories, then converted into binary change masks and the union set is obtained to obtain the combined change mask after removing the mobile elements, C represents the pixel area of the change part, K0 represents the pixel area after removing the sky category in the t0 image semantic segmentation result, T represents the research duration, if T is in years, CD represents the annual change rate of urban features in the study area. The annual dynamic degree of urban renewal in the main urban area of Guangzhou in 2013 and 2019 is studied, so T is 7. The corresponding relationship between each variable and formula (10) is as follows:

[0119]

[0120] Urban renewal detection and analysis overall process

[0121] The overall framework of the application mainly includes three parts: street view image pair acquisition, street view weakly supervised semantic change detection and index calculation and mapping. Figure 4 ) can be further divided into four steps: 1. Generate sampling points along the road network, and obtain panoramic images taken by vehicle-mounted cameras in 2013 and 2019 at each sampling point. 2. Use the semantic change detection process proposed by us to process the panoramic images to obtain the change object and its category information in the image pair. 3. Use cubic projection to convert the semantic change detection results in the panoramic view into four perspective views in front, back, left and right to obtain the change perception conforming to the human eye observation rule. 4. Use the urban renewal dynamic degree index constructed by us to evaluate the renewal degree of the sampling point using the panoramic and four perspective view detection results, and perform spatio-temporal change pattern analysis and mapping.

[0122] Precision evaluation index

[0123] The application uses F1 score to evaluate the performance of the change detection algorithm. F1 score is the harmonic mean of precision and recall, which can comprehensively measure the performance of the model in accuracy and comprehensiveness, and its value range is 0 to 1, and higher F1 value represents higher precision and recall. The formula of F1 is as follows:

[0124]

[0125] In the binary change detection task, the "change" category represents positive, and the "unchanged" category represents negative. Given true TP (True Positive), true negative TN (True Negative), false positive FP (False Positive) and false negative FN (False Negative), there are

[0126] Specific implementation content: data acquisition and processing

[0127] The research area is determined, such as the main urban area of Guangzhou in this study, covering Yuexiu, Liwan and other regions, and 11431 pairs of panoramic street view images in 2013 and 2019 are collected respectively. These data provide double-time urban facade changes and high-resolution urban renewal dynamics.

[0128] Cross-C2PO is evaluated on two commonly used public change detection datasets, VL-CMU-CD and PSCD.

[0129] VL-CMU-CD contains 1362 registered perspective image pairs. Following previous methods, the images are resized to 512x512, and 933 pairs are divided into the training set and 429 pairs are divided into the test set. Following previous methods, the training set is expanded to 3732 pairs by rotation augmentation. VL-CMU-CD only contains joint change labels and does not provide change split labels.

[0130] PSCD contains 770 pairs of panoramic street view images. Following the official approach, the images are resized to 1024x224, then sliding cropping is performed on the width with a step size of 56 and a window size of 224x224, and the generated image blocks are resized to 256x256 resolution, and finally each image block is rotated for augmentation. 5-fold cross-validation is used for training and testing, and each training set will contain 36960 pairs of 256x256 image blocks, and each test set will contain 154 pairs of 1024x256 image blocks. PSCD provides joint change and change split labels.

[0131] Model training and verification: In the training stage, VGG-16 is initialized using ImageNet pre-trained weights. The model is optimized using the Adam optimizer with an initial learning rate of 〖10〗^(-4), and the cosine annealing schedule is used to dynamically adjust the learning rate. To facilitate reproducibility, the batch size is 4 for the VL-CMU-CD dataset and 16 for the PSCD dataset, and all comparative models are trained for 100 epochs. The present invention uses a NVIDIA GeForce RTX 4090 GPU for all experiments.

[0132] The binary change detection task divides each pixel position into one of two categories, "changed" or "unchanged", which can actually be regarded as a binary classification semantic segmentation task. Therefore, the Cross-C2PO proposed in the present application is trained using cross-entropy as the loss function. It should be noted that in a typical street view change detection task, the changed area usually only accounts for a small part. In order to alleviate the influence of this class imbalance, the present application uses a weighted cross-entropy loss function. The weighted cross-entropy loss formula at each pixel position is as follows:

[0133]

[0134] N represents the number of categories, N = 2 in the binary change detection task, where category 0 represents the background, i.e. unchanged, and category 1 represents the change. y i i respectively represent the i-th category of the joint change mask ground truth and the model prediction m. w i represents the weight of the i-th category, and the weight value is obtained by counting the proportion of each category in the training set:

[0135]

[0136] n c represents the number of pixels in the changed area in the training set, n a represents the total number of pixels.

[0137] Accuracy evaluation index

[0138] The present application uses F1 score to evaluate the performance of the change detection algorithm. The F1 score is the harmonic mean of precision and recall, which can comprehensively measure the performance of the model in terms of accuracy and comprehensiveness, and its value ranges from 0 to 1. A higher F1 value represents higher precision and recall. The formula of F1 is as follows:

[0139]

[0140] In the binary change detection task, the "changed" category represents positive, and the "unchanged" category represents negative. Given True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN), there are

[0141] ​Table 1, Table 2 demonstrates the accuracy of Cross-C2PO compared with previous methods. In order to make a fair comparison, the backbone network of all methods is pre-trained using the ImageNet dataset. As shown in Table 1, on the VL-CMU-CD dataset, Cross-C2PO improves the accuracy of C-3PO by 1.6%, and improves the accuracy of CSCDNet and DR-TANet by 5% and 6.5%, respectively, which is a significant improvement over previous models. As shown in Table 2, on the PSCD dataset, the joint change F1 score of Cross-C2PO is better than previous methods, which improves the accuracy of C-3PO by 1.1%, and improves the accuracy of CSCDNet and DR-TANet by 6.7% and 5.6%, respectively. In addition, the F1 scores of m0 and m1 are 71.1% and 74.1%, respectively, which shows that the model has good weakly supervised change splitting capability.

[0142] Table 1 Cross-C2PO and previous models F1 score comparison on VL-CMU-CD dataset

[0143] Method VL-CMU-CD Literature reference FC-EF 44.6 Daudt et al., 2018 FC-Siam-diff 65.3 Daudt et al., 2018 FC-Siam-conc 65.6 Daudt et al., 2018 DR-TANet 75.1 Chen et al., 2021 CSCDNet 76.6 Sakurada et al., 2018 C-3PO 80.0 Wang et al., 2023 Cross-C2PO 81.6 Model herein

[0144] Table 2 Cross-C2PO and previous models F1 score comparison on PSCD dataset

[0145]

[0146] Dynamic degree analysis and visualization

[0147] Results show: The high-intensity renewal area in Guangzhou is concentrated in the industrial land of Liwan District and the main road of Yuexiu District. The dynamic degree histogram shows that about 20% of the area is at a high / medium level, which conforms to the "80 / 20 rule".

[0148] Multi-view comparison: The right view detects more changes (the central axis of Yuexiu District), and the left view is mainly micro-renewal (advertising board renovation).

[0149] Technical key points of the present application scheme: Cross-C2PO dual-branch cross-fusion model: realizes end-to-end weakly supervised change splitting and eliminates input order ambiguity. Differentiable union operator: ensures the integrity of the joint change mask and supports gradient backpropagation. Urban renewal dynamic degree model: fusion of multi-view street view data, quantification of renewal intensity and correlation of urban functional areas. Semantic segmentation transfer technology: use pre-trained models (such as DeepLabV3+) to improve classification accuracy and reduce labeling cost.

[0150] Specific implementation of Cross-C2PO model, including double-branch architecture, Cross-MTF operator and differentiable union algorithm. Cubic projection-based multi-view street view data processing method and dynamic degree mapping process. Urban renewal dynamic degree calculation formula and multi-level classification standard. Integrated street view classification, change detection and dynamic degree analysis integrated system architecture.

[0151] To achieve the above object, the application also provides a kind of urban renewal intelligent monitoring system based on double-branch cross fusion and multi-view dynamic degree, as shown in Figure 7 The system is used to realize the urban renewal intelligent monitoring method based on double-branch cross fusion and multi-view dynamic degree;The system specifically includes:

[0152] First data generation unit, for generating and acquiring the first data corresponding to the target urban area, and based on the first data, combined with cubic projection to generate corresponding second data;Wherein the first data is double temporal street view image pair data;The second data is multi-view perspective data;

[0153] First model construction unit, for creating the first model corresponding to the first data, based on the first model, and combined with Cross-MTF feature fusion operator to generate corresponding third data;Wherein the first model is double-branch cross fusion model Cross-C2PO;The third data is change mask data;

[0154] Second model construction unit, for establishing the second model corresponding to the first data, and based on the second model, generating corresponding fourth data;Wherein the second model is a pre-trained semantic segmentation model;The fourth data is semantic change area;

[0155] Third model construction unit, for constructing the third model corresponding to the city, based on the third model, generating corresponding fifth data;Wherein the third model is an urban renewal dynamic degree model;The fifth model is an update intensity index data;

[0156] Second data generation unit, for fusing multi-view dynamic degree index based on the third data, the fourth data and the fifth data, and generating corresponding sixth data;Wherein the sixth data is urban renewal heat map data.

[0157] The first data generation unit further includes:

[0158] First generation module, for generating front, rear, left and right multi-view perspective data based on the first data through cubic projection;

[0159] And / or, the first model construction unit further includes:

[0160] The second generation module is configured to generate corresponding seventh data based on the third data by combining a differentiable union operator; wherein the seventh data is joint change mask data.

[0161] The first processing module is configured to perform channel-level minimum or maximum filtering processing on the change mask data output by the double-branch, and to realize end-to-end weakly supervised training through a gradient reservation mechanism.

[0162] And / or, the second model construction unit further comprises:

[0163] The second processing module is configured to convert the panoramic semantic change detection result into four-direction perspective view data.

[0164] The third generation module is configured to respectively calculate and generate eighth data corresponding to the dynamic degree of each view angle; wherein the eighth data is index data.

[0165] The third processing module is configured to generate corresponding ninth data by weighted fusion processing based on the eighth data; wherein the ninth data is comprehensive dynamic degree index data.

[0166] The double-branch cross-fusion model Cross-C2PO comprises:

[0167] The symmetric double-branch architecture processes the contrast order of t0-t1 and t1-t0 respectively; the improved Cross-MTF feature fusion operator; and the feature pyramid decoder for fusing multi-scale feature maps.

[0168] In the system scheme embodiment of the present application, the method steps involved in the urban renewal intelligent monitoring based on double-branch cross-fusion and multi-view dynamic degree have been described in detail above, that is, the function modules in the system are used to realize the steps or sub-steps in the above method embodiment, which will not be described here.

[0169] To achieve the above-mentioned purpose, the present application also provides an urban renewal intelligent monitoring platform based on double-branch cross-fusion and multi-view dynamic degree, as shown in Figure 8 The processor executes the urban renewal intelligent monitoring platform control program based on double-branch cross-fusion and multi-view dynamic degree, the urban renewal intelligent monitoring platform control program based on double-branch cross-fusion and multi-view dynamic degree is stored in the memory, and the urban renewal intelligent monitoring platform control program based on double-branch cross-fusion and multi-view dynamic degree realizes the urban renewal intelligent monitoring method steps. For example:

[0170] S1, generate and acquire first data corresponding to a target urban area, and generate corresponding second data based on the first data in combination with cube projection; Wherein the first data is a double temporal street view image pair data; The second data is multi-view perspective view data;

[0171] S2, create a first model corresponding to the first data, generate corresponding third data based on the first model in combination with a Cross-MTF feature fusion operator; Wherein the first model is a double-branch cross-fusion model Cross-C2PO; The third data is change mask data;

[0172] S3, establish a second model corresponding to the first data, and generate corresponding fourth data based on the second model; Wherein the second model is a pre-trained semantic segmentation model; The fourth data is a semantic change area;

[0173] S4, construct a third model corresponding to the city, generate corresponding fifth data based on the third model; Wherein the third model is a city update dynamic degree model; The fifth model is an update intensity index data;

[0174] S5, based on the third data, the fourth data and the fifth data, multi-view dynamic degree index fusion processing is carried out, and corresponding sixth data is generated; Wherein the sixth data is city update heat map data.

[0175] The specific details of the steps have been described above, and will not be repeated here.

[0176] In the embodiment of the application, the built-in processor of the city update intelligent monitoring platform based on double-branch cross-fusion and multi-view dynamic degree can be composed of integrated circuits, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple packaged integrated circuits with the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors and combinations of various control chips, etc. The processor connects various components through various interfaces and lines, and executes programs or units stored in the memory and calls data stored in the memory to perform various functions and process data of the city update intelligent monitoring platform based on double-branch cross-fusion and multi-view dynamic degree;

[0177] The memory is used to store program codes and various data, and is installed in the city update intelligent monitoring platform based on double-branch cross-fusion and multi-view dynamic degree, and realizes high-speed and automatic access to programs or data during running.

[0178] The memory includes Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk memory, magnetic disk memory, magnetic tape memory, or any other medium that can be used to carry or store data which is readable by a computer.

[0179] The present application generates and acquires first data corresponding to a target urban area by a method, and generates corresponding second data based on the first data in combination with cube projection; wherein the first data is a double-time phase street view image pair data; the second data is multi-view perspective view data; a first model corresponding to the first data is created, and third data corresponding thereto is generated based on the first model in combination with a Cross-MTF feature fusion operator; wherein the first model is a double-branch cross-fusion model Cross-C2PO; the third data is change mask data; a second model corresponding to the first data is established, and fourth data corresponding thereto is generated based on the second model; wherein the second model is a pre-trained semantic segmentation model; the fourth data is a semantic change area; a third model corresponding to the city is constructed, and fifth data corresponding thereto is generated based on the third model; wherein the third model is a city update dynamic degree model; the fifth model is update intensity index data; multi-view dynamic degree indexes are fused and processed based on the third data, the fourth data and the fifth data, and sixth data corresponding thereto is generated; wherein the sixth data is city update heat map data, and a system and a platform corresponding to the method can realize automatic and refined perception and analysis of urban facade changes.

[0180] That is, the present application scheme brings significant technical support and supplement to the existing urban renewal monitoring technology by integrating the street view image classification and change detection model Cross-C2PO, including:

[0181] Dual-branch cross-fusion model: a novel dual-branch architecture is proposed, which realizes end-to-end weakly supervised change split by cross-comparing time image pairs (t0→t1 and t1→t0) and combining an improved Cross-MTF feature fusion operator. The model solves the input order ambiguity problem caused by the feature fusion operator not satisfying the commutative law in traditional single-branch structure, simplifies the semantic change detection process, and does not need to rely on synthetic data set training, significantly improving the detection efficiency and robustness.

[0182] Differentiable union operator: a differentiable union operator is designed to ensure that the joint change mask covers all predicted change areas while maintaining gradient propagation consistency. This technique solves the inconsistency problem of change masks in weakly supervised training and provides mathematical guarantees for model optimization.

[0183] Multi-view urban renewal dynamic degree model: existing land use dynamic degree models ignore intra-class changes and are not combined with street view perspectives, making it difficult to reflect urban renewal details. The invention constructs a street view data-oriented urban renewal dynamic degree index, integrates street view change points and dynamic degree to form a multi-dimensional representation of urban facade change characteristics, conducts panoramic map change pattern and local analysis of front, rear, left and right views of urban function renewal intensity, and clarifies the connotation and denotation of urban street view change and index calculation.

[0184] The above-described embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as limiting the scope of the present patent. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are all within the scope of the present application. Therefore, the protection scope of the present patent should be subject to the appended claims.

Claims

1. An urban renewal intelligent monitoring method based on double-branch cross fusion and multi-view dynamic degree, characterized in that, The method comprises the steps of: generating and obtaining first data corresponding to a target urban area, and generating corresponding second data based on the first data in combination with a cube projection; wherein the first data is a double-time-phase street view image pair data; and the second data is multi-view perspective view data; creating a first model corresponding to the first data, and generating corresponding third data based on the first model in combination with a Cross-MTF feature fusion operator; wherein the first model is a double-branch cross-fusion model Cross-C2PO; and the third data is change mask data; establishing a second model corresponding to the first data, and generating corresponding fourth data based on the second model; wherein the second model is a pre-trained semantic segmentation model; and the fourth data is a semantic change area; constructing a third model corresponding to the city, and generating corresponding fifth data based on the third model; wherein the third model is a city update dynamic degree model; and the fifth model is update intensity index data; based on the third data, the fourth data and the fifth data, fusing and processing multi-view dynamic degree indicators, and generating corresponding sixth data; wherein the sixth data is city update heat map data.

2. The urban renewal intelligent monitoring method based on the double-branch cross fusion and multi-view dynamic degree according to claim 1, characterized in that, The generating and obtaining first data corresponding to a target urban area, and generating corresponding second data based on the first data in combination with a cube projection further comprises: based on the first data, generating front, rear, left and right multi-view perspective view data respectively through a cube projection. 3.The urban renewal intelligent monitoring method based on the dual-branch cross fusion and multi-view dynamic degree according to claim 1, characterized in that, The creating a first model corresponding to the first data, and generating corresponding third data based on the first model in combination with a Cross-MTF feature fusion operator further comprises: based on the third data, generating corresponding seventh data in combination with a differentiable union operator; wherein the seventh data is joint change mask data; performing channel-level minimum or maximum filtering processing on the change mask data output by the double-branch, and realizing end-to-end weak supervision training through a gradient reservation mechanism. 4.The urban renewal intelligent monitoring method based on the dual-branch cross fusion and multi-view dynamic degree according to claim 1 or 3, characterized in that, The double-branch cross-fusion model Cross-C2PO comprises: a symmetrical double-branch architecture for processing t0→t1 and t1→t0 contrast sequences respectively; an improved Cross-MTF feature fusion operator; and a feature pyramid decoder for fusing multi-scale feature maps.

5. The urban renewal intelligent monitoring method based on double-branch cross fusion and multi-view dynamic degree according to claim 1, characterized in that, The establishing a second model corresponding to the first data, and generating corresponding fourth data based on the second model further comprises: converting panoramic semantic change detection results into four-direction perspective view data; respectively calculating eighth data corresponding to the dynamic degree of each view; wherein the eighth data is index data; based on the eighth data, generating corresponding ninth data through weighted fusion processing; wherein the ninth data is comprehensive dynamic degree index data.

6. An urban renewal intelligent monitoring system based on double-branch cross fusion and multi-view dynamic degree, characterized in that, The system is used to realize the city update intelligent monitoring method based on double-branch cross-fusion and multi-view dynamic degree according to any one of claims 1 to 5; and the system comprises: The first data generation unit is configured to generate and acquire first data corresponding to a target urban area, and generate corresponding second data based on the first data and in combination with a cube projection; the first data is a double-time-phase street view image pair data; and the second data is multi-view perspective view data. The first model construction unit is configured to create a first model corresponding to the first data, and generate corresponding third data based on the first model and in combination with a Cross-MTF feature fusion operator; the first model is a double-branch cross-fusion model Cross-C2PO; and the third data is change mask data. The second model construction unit is configured to establish a second model corresponding to the first data, and generate corresponding fourth data based on the second model; the second model is a pre-trained semantic segmentation model; and the fourth data is a semantic change area. The third model construction unit is configured to construct a third model corresponding to the city, and generate corresponding fifth data based on the third model; the third model is a city update dynamic degree model; and the fifth model is update intensity index data. The second data generation unit is configured to fuse and process multi-view dynamic degree indexes based on the third data, the fourth data, and the fifth data, and generate corresponding sixth data; the sixth data is city update heat map data.

7. The urban renewal intelligent monitoring system based on the double-branch cross fusion and multi-view dynamic degree according to claim 6, characterized in that, The first data generation unit further includes: The first generation module is configured to generate front, rear, left, and right multi-view perspective view data respectively based on the first data and through a cube projection. The first model construction unit further includes: The second generation module is configured to generate corresponding seventh data based on the third data and in combination with a differentiable union operator; the seventh data is joint change mask data. The first processing module is configured to perform channel-level minimum or maximum filtering processing on the change mask data output by the double-branch, and implement end-to-end weak supervision training through a gradient reservation mechanism. The second model construction unit further includes: The second processing module is configured to convert panoramic semantic change detection results into four-direction perspective view data. The third generation module is configured to calculate and generate eighth data corresponding to the dynamic degree of each view respectively; the eighth data is index data. The third processing module is configured to generate corresponding ninth data through weighted fusion processing based on the eighth data; the ninth data is comprehensive dynamic degree index data. 8.The urban renewal intelligent monitoring system based on the dual-branch cross fusion and multi-view dynamic degree according to claim 6 or 7, characterized in that, The double-branch cross-fusion model Cross-C2PO includes: a symmetrical double-branch architecture for processing t0→t1 and t1→t0 comparison sequences respectively, an improved Cross-MTF feature fusion operator, and a feature pyramid decoder for fusing multi-scale feature maps.

9. An urban renewal intelligent monitoring platform based on double-branch cross fusion and multi-view dynamic degree, characterized in that, The application relates to a double-branch cross fusion and multi-view dynamic urban renewal intelligent monitoring platform control program comprising a processor, a memory and a double-branch cross fusion and multi-view dynamic urban renewal intelligent monitoring platform control program; wherein the double-branch cross fusion and multi-view dynamic urban renewal intelligent monitoring platform control program is stored in the memory and is executed by the processor; and the double-branch cross fusion and multi-view dynamic urban renewal intelligent monitoring platform control program realizes the double-branch cross fusion and multi-view dynamic urban renewal intelligent monitoring method as claimed in any one of claims 1 to 5.

Images