Method and system for detecting illegal building with open vocabularies and readable medium

Abstract

The invention belongs to the technical field of computer vision and artificial intelligence, and relates to an illegal building detection method and system based on open vocabularies and a readable medium, and the method comprises the steps: obtaining a building image and illegal building description of a natural language; inputting the building image and the illegal building description into a multi-modal feature fusion model, framing out a position corresponding to the illegal building description in the building image, taking the position as a suspected illegal area, and describing the suspected illegal area; converting the illegal building description into a semantic vector, and comparing the semantic vector with contents in a policy knowledge base to obtain a plurality of policy contents with the highest similarity; and according to policy content and description of the suspected violation area, comprehensively judging whether violation exists, and finally determining whether the building violates violation. According to the method, the limitation of closed set training is broken through, open vocabulary illegal building type detection is realized, multi-modal information is fused, environmental factor interference is reduced, compliance transformation and illegal building are accurately distinguished, and illegal building judgment accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to an open vocabulary illegal construction detection method, system and readable medium, belonging to the technical field of computer vision and artificial intelligence. BACKGROUND

[0002] Illegal construction refers to buildings that are newly built, expanded or rebuilt in the urban planning area without going through legal approval procedures, such as buildings that have not applied for or obtained approval, buildings that have changed the planning without permission, and temporary buildings that have exceeded the period of use. During the process of urbanization, the disordered growth of illegal construction has a serious impact on urban space planning, public safety and ecological environment. Therefore, efficient and accurate detection and positioning of illegal construction has become a key requirement for urban governance.

[0003] Existing illegal construction detection techniques based on deep learning detection methods rely on closed-set training data and can only identify building types defined in the training set. For open vocabulary illegal construction types such as unlicensed steel frame houses and temporary board houses occupying fire access, which are not included in the training set, they cannot be effectively identified, have poor generalization ability, and are difficult to cope with dynamic changes in illegal construction scenarios. Illegal construction detection methods based on time series image comparison are easily disturbed by environmental factors such as weather, light, and vegetation obstruction, leading to misjudgment of the changing area. They can only identify whether there is a building change and cannot associate key features in the illegal construction description with local policy provisions, making it difficult to distinguish between compliant modifications (such as balcony encapsulation) and illegal construction (such as unlicensed additional floors), resulting in low accuracy.

[0004] In addition, the detection results of existing technologies only output whether it is illegal construction, lack of matching illegal construction features, policy basis and other key information, law enforcement personnel cannot trace the judgment logic, and it is difficult to serve as evidence to support law enforcement. SUMMARY

[0005] To solve the above problems, the present application provides an open vocabulary illegal construction detection method, system and readable medium, which breaks through the closed-set training limitation and realizes open vocabulary illegal construction type detection, improving the accuracy of illegal construction judgment.

[0006] To achieve the above object, the present application provides the following technical scheme: an illegal building detection method of open vocabulary, comprising the following steps: obtaining a building image and a natural language illegal building description; inputting the building image and the illegal building description into a multi-modal feature fusion model, framing the corresponding position of the illegal building description in the building image as a suspected illegal area, and describing the suspected illegal area; converting the illegal building description into a semantic vector, comparing the semantic vector with the content in the policy knowledge base, and obtaining several policy contents with the highest similarity; comprehensively judging whether there is illegal construction according to the policy content and the description of the suspected illegal area, and finally determining whether the building is illegal.

[0007] Further, the building image is obtained by a UAV equipped with a high-definition optical camera, and the building image includes a building orthographic image, a building side view image, a building material, a building layer, and the relative positions of the building and green land, roads, and fire passages, and a building surrounding environment.

[0008] Further, the multi-modal feature fusion model includes a visual encoder, a visual-language adapter, and a large language model, the visual encoder is used for feature extraction of the building image, the visual-language adapter is used for fusing the building image features and the illegal building description to generate a mixed feature vector, and the large language model outputs the suspected illegal area and generates an output result including images and texts.

[0009] Further, the method for feature extraction of the building image is: pre-processing the building image, the pre-processing method includes denoising, resolution unification, associated geographic information, and obtaining building latitude and longitude coordinates; through 2D position coding, the spatial position information in the pre-processed building image is retained, the building image including the spatial position information is input into the multi-layer self-attention mechanism of the Vit-bigG encoder, and the building image features are obtained.

[0010] Further, the method for fusing the building image features and the illegal building description is: performing cross-attention operation on the query vector and the building image features, and compressing the building image features into a fixed-length sequence; converting the illegal building description into a token sequence; fusing the fixed-length sequence and the token sequence, making the building image features and the illegal building description interact through the cross-attention mechanism, finding the corresponding area of the illegal building description in the building image; and outputting the corresponding area in the form of a bounding box coordinate.

[0011] Further, the method of generating the output result of the picture-text includes: converting the illegal building description into a token sequence; inputting the token sequence into a Qwen Decoder layer to generate natural language through autoregressive decoding; and outputting a picture-text output result according to the natural language. The intermediate decoding result obtained by the Qwen Decoder layer through the multi-head attention mechanism is an intermediate process, and the final purpose is to generate natural language.

[0012] Further, the method of constructing the policy knowledge base includes: extracting text and structural information of a policy document by using a multi-modal analysis function of Qwen-VL; intelligently segmenting policy clauses in the policy document by a language model and automatically labeling key metadata; converting the text of the policy document into high-quality vectors by a semantic embedding technology of Qwen-VL; and storing the high-quality vectors in a vector database to form a searchable policy knowledge base.

[0013] Further, the policy document includes a structured or unstructured planning policy document; the policy clauses include building height restrictions, land use property requirements, and approval process violation identification standard clauses; and the key metadata includes validity period, issuing agency, and applicable topics.

[0014] The application further discloses an open-vocabulary illegal building detection system, which comprises: an image-text input module configured to obtain building images and natural language illegal building descriptions; a multi-modal feature fusion module configured to input the building images and the illegal building descriptions into a multi-modal feature fusion model, frame positions corresponding to the illegal building descriptions in the building images as suspected illegal areas, and describe the suspected illegal areas; a policy comparison module configured to convert the illegal building descriptions into semantic vectors, compare the semantic vectors with contents in a policy knowledge base, and obtain a plurality of policy contents with the highest similarity; and a selection and positioning module configured to comprehensively judge whether there is illegal construction according to the policy contents and descriptions of the suspected illegal areas, and finally determine whether the building is illegal.

[0015] The application further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the open-vocabulary illegal building detection method.

[0016] The technical scheme of the application has at least the following technical effects or advantages: (1) The application breaks through the limitation of closed set training, realizes open-vocabulary illegal building type detection, and can identify new illegal buildings that are not defined.

[0017] (2) The application fuses building images, related policies, building descriptions and other multi-modal information, multi-modal information fusion reduces environmental factor interference, accurately distinguishes legal reconstruction from illegal construction, improves the accuracy of illegal building judgment, and can accurately distinguish legal reconstruction from illegal construction.

[0018] (3) The key basis of the judgment detection result in the output result in the application makes law enforcement personnel can trace the judgment logic, can quickly judge the correctness of the detection result, and timely checks and corrects the detection result, guarantees the result interpretability and law enforcement availability.

[0019] (4) The method in the application has strong generalization ability: open vocabulary recognition supports new illegal building detection, adapts to different cities, different scene illegal building types, and does not need to frequently update training data. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a flowchart of the open vocabulary illegal building detection method in an embodiment of the application; Figure 2 is a structural schematic diagram of the multi-modal feature fusion model in an embodiment of the application; Figure 3 is an output result schematic diagram of the multi-modal feature fusion model in an embodiment of the application. DETAILED DESCRIPTION

[0021] In order for those skilled in the art to better understand the technical solutions of the application, the application is described in detail through specific embodiments. However, it should be understood that the provision of the specific embodiments is only for better understanding of the application, and they should not be understood as a limitation on the application. In the description of the application, it should be understood that the terms used are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0022] To address the limitations of existing technologies in effectively identifying illegal building types using open-vocabulary methods and in associating key features in illegal building descriptions with local policy provisions, this invention proposes an open-vocabulary method, system, and readable medium for illegal building detection. The method involves acquiring building images and natural language descriptions of illegal buildings; inputting these images and descriptions into a multimodal feature fusion model; outlining the locations in the building image corresponding to the illegal building descriptions as suspected illegal areas; and describing these suspected illegal areas; converting the illegal building descriptions into semantic vectors; comparing these semantic vectors with content in a policy knowledge base; and obtaining the policy content with the highest similarity; based on the policy content and the generated descriptions of suspected illegal areas, a comprehensive judgment is made regarding the existence of a violation, ultimately determining whether the building is illegal. This invention overcomes the limitations of closed-set training, enabling the detection of illegal building types using open-vocabulary methods, and can identify undefined new types of illegal buildings. By integrating multimodal information such as building images, relevant policies, and building descriptions, it improves the accuracy of illegal building determination and can accurately distinguish between compliant renovations and illegal buildings.

[0023] Example 1 This embodiment discloses an open-vocabulary method for detecting illegal buildings, such as... Figure 1 As shown, it includes the following steps: S1 acquires images of buildings and natural language descriptions of illegal structures.

[0024] Architectural images were acquired using drones equipped with high-definition optical cameras. These images include orthophotos, side views, building materials, number of floors, relative positions of the building to green spaces, roads, and fire lanes, and the surrounding environment. The resolution of the architectural images is no less than 4K. Feature extraction from the architectural images involves preprocessing, including noise reduction, resolution standardization, associating with geographic information, and obtaining the building's latitude and longitude coordinates.

[0025] The description of illegal buildings in natural language is a description of the illegal buildings entered by the user, including open vocabulary features of illegal buildings, such as a four-story steel-framed factory building in an industrial park that exceeds the approved height by 3 meters, and a two-story brick-concrete temporary house in a residential green space that has not obtained a permit.

[0026] S2 inputs the building image and the description of the illegal building into the multimodal feature fusion model, outlines the position in the building image corresponding to the description of the illegal building as the suspected illegal area, and describes the suspected illegal area.

[0027] like Figure 2As shown, the multi-modal feature fusion model includes a visual encoder, a visual-language adapter, and a large language model. The visual encoder is used for feature extraction of the building image, the visual-language adapter is used for fusion of the building image features and the illegal construction description to generate a mixed feature vector, and the large language model outputs the suspected illegal construction area and generates an output result including images and texts, i.e., the suspected illegal construction area and the description of the illegal construction area.

[0028] For example, the mixed feature vector obtained by the visual-language adapter and the prompt word are input into the large language model. The large language model processes the two inputs to obtain two results: the pixel coordinates of the illegal construction area in the image and the description text of the illegal construction area. As shown in the following figure, Figure 3 As shown, the output result is an example. The pixel coordinates of the illegal construction area are marked in the image by a rectangular frame, and the text result is: There is a house being built in the farmland.

[0029] The visual encoder processes the image based on the Vision Transformer (ViT). The input image is resized and segmented into image blocks (patches) to generate a set of building image features representing different regions and information in the image. A learnable query vector is cross-attentioned with the building image features to compress the building image features into a fixed-length sequence. The number of learnable query vectors is fixed at 256, so the building image features are compressed into a sequence of length 256. In order to not lose position information during compression, the adapter incorporates 2D absolute position encoding in the cross-attention calculation, i.e., by using 2D position encoding, the spatial position information in the preprocessed building image is preserved. The building image including spatial position information is input into the multi-layer self-attention mechanism of the Vit-bigG encoder to obtain the building image features.

[0030] The method of fusing the building image features with the illegal construction description is as follows: The illegal construction description is converted into a token sequence. The fixed-length sequence and the token sequence are fused, and the building image features and the illegal construction description are interacted through the cross-attention mechanism to find the corresponding region of the illegal construction description in the building image. The corresponding region is output in the form of a bounding box coordinate.

[0031] The multi-modal feature fusion model is trained in multiple stages and learns a large amount of data containing region description, visual question answering, etc. It can determine the target position in the image that matches the text description according to the fused features and output it in the form of a bounding box coordinate, realize the positioning and labeling of the illegal construction according to the input prompt word, and usually output a standardized JSON format document containing the bounding box coordinate and the text description information related to the illegal construction.

[0032] The method for generating an output result including a picture-text is: converting the illegal building description into a token sequence; inputting the token sequence into a Qwen Decoder layer to generate natural language through autoregressive decoding; and outputting a picture-text output result according to the natural language. The intermediate decoding result obtained by the Qwen Decoder layer through the multi-head attention mechanism is an intermediate process, and the final purpose is to generate natural language.

[0033] The multi-modal feature fusion model guides the detection of the target in the image through the illegal building description, detects the target according to the mixed feature vector of the illegal building description and the building image feature, and breaks through the limitation of the traditional closed-set detection model that can only identify predefined categories.

[0034] S3 converts the illegal building description into a semantic vector, compares the semantic vector with the content in the policy knowledge base, and obtains a plurality of policy contents with the highest similarity.

[0035] The method for constructing the policy knowledge base is: using the multi-modal analysis function of Qwen-VL (Qwen Large Vision Language Model, i.e., Qwen Large Vision Language Model) to extract the text and structural information of the policy file, wherein the policy file can be a PDF structure or a scan; intelligently segmenting the policy provisions in the policy file through the language model in Qwen-VL and automatically labeling key metadata; converting the text of the policy file into a high-quality vector through the semantic embedding technology of Qwen-VL; and storing the high-quality vector in a vector database to form a searchable policy knowledge base. In the embodiment, the Qwen-VL model is only illustrative, and other multi-modal models with visual positioning and picture-text processing capabilities, such as Qwen2-VL, Qwen2.5-VL, etc., can also be used.

[0036] The policy knowledge base includes segmented text, embedded vectors and metadata.

[0037] The method for comparing the semantic vector with the content in the policy knowledge base is: converting the illegal building description into a semantic vector, performing similarity retrieval through the policy knowledge base to recall Top-K candidate policy segments; then, using a large language model to finely screen and position the candidate content, directly outputting a judgment result while attaching key evidence, including: the file name of the relevant policy, the item number, the key sentence of the original text and the source file path.

[0038] The policy file includes structured or unstructured planning policy files, such as “XX City Residential Area Planning Management Regulations” and “XX Industrial Park Construction Standards”; policy provisions, including: building height restrictions, land use property requirements and approval process violation determination standard provisions, etc.; and key metadata, including validity period, issuing agency and applicable theme, etc.

[0039] S4 According to the policy content and the description of the suspected illegal area, it is comprehensively judged whether there is illegal construction, and finally it is determined whether the building is illegal.

[0040] According to the policy content, it is judged whether the illegal building description is illegal. For example, the illegal description is: building houses in cultivated land. The following policy is obtained by searching the policy library: "unauthorized building houses in cultivated land is illegal construction", and no approval procedures for building houses in this area are found. According to the policy content, it is judged that the area is indeed illegal construction.

[0041] Embodiment two Based on the same inventive concept, the embodiment discloses an open-vocabulary illegal building detection system, which comprises: an image-text input module, configured to obtain building images and natural language illegal building descriptions; a multi-modal feature fusion module, configured to input the building images and the illegal building descriptions into a multi-modal feature fusion model, frame the positions corresponding to the illegal building descriptions in the building images as suspected illegal areas, and describe the suspected illegal areas; a policy comparison module, configured to convert the illegal building descriptions into semantic vectors, compare the semantic vectors with the contents in a policy knowledge base, and obtain a plurality of policy contents with the highest similarity; a selection and positioning module, configured to comprehensively judge whether there is illegal construction according to the policy content and the description of the suspected illegal area, and finally determine whether the building is illegal.

[0042] Embodiment three Based on the same inventive concept, the embodiment discloses a computer-readable storage medium, and the computer-readable storage medium stores a computer program. The computer program is executed by a processor to implement the open-vocabulary illegal building detection method of any one of the above embodiments.

[0043] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0044] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0045] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0046] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0047] Finally, it should be noted that the above-mentioned embodiments are merely intended to illustrate the technical solutions of the present application, rather than limit the same. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the technical solutions of the present application can still be modified or equivalent replaced without departing from the spirit and scope of the present application, and any modification or equivalent replacement should be covered within the scope of protection of the present application. The above-mentioned content is merely a specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, and all these changes or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for detecting illegal buildings using open vocabulary, characterized in that, Includes the following steps: Obtain architectural images and natural language descriptions of illegal buildings; The building image and the description of the illegal building are input into a multimodal feature fusion model. The location in the building image that corresponds to the description of the illegal building is outlined as a suspected illegal area, and the suspected illegal area is described. The description of the illegal building is converted into a semantic vector, and the semantic vector is compared with the content in the policy knowledge base to obtain the policy content with the highest similarity. Based on the policy content and the description of the suspected illegal area, a comprehensive judgment is made as to whether there is any violation, and the final determination is whether the building is illegal.

2. The method for detecting illegal buildings using open vocabulary as described in claim 1, characterized in that, The building images were acquired by a drone equipped with a high-definition optical camera. The building images include orthophotos of the building, side views of the building, building materials, number of floors, relative positions of the building to green spaces, roads and fire lanes, and the surrounding environment of the building. The building image is preprocessed, and the preprocessing method includes denoising the building image, unifying the resolution, associating geographic information, and obtaining the building's latitude and longitude coordinates.

3. The method for detecting illegal buildings using open vocabulary as described in claim 1 or 2, characterized in that, The multimodal feature fusion model includes a visual encoder, a visual-language adapter, and a large language model. The visual encoder is used to extract features from the building image. The visual-language adapter is used to fuse the building image features with the description of illegal buildings to generate a hybrid feature vector. The large language model outputs the suspected illegal area and generates an output result including images and text.

4. The method for detecting illegal buildings using open vocabulary as described in claim 3, characterized in that, The method for feature extraction of the building image is as follows: cross-attention operation is performed between the query vector and the building image features to compress the building image features into a fixed-length sequence; spatial location information in the preprocessed building image is retained through 2D position encoding; and the building image including spatial location information is input into the multi-layer self-attention mechanism of the Vit-bigG encoder to obtain the building image features.

5. The method for detecting illegal buildings using open vocabulary as described in claim 3, characterized in that, The method for fusing building image features with descriptions of illegal buildings is as follows: Convert the description of the illegal building into a token sequence; The fixed-length sequence and the token sequence are fused together, and the building image features are interacted with the description of illegal buildings through a cross-attention mechanism to find the region corresponding to the description of illegal buildings in the building image. The corresponding region is output in the form of bounding box coordinates.

6. The method for detecting illegal buildings using open vocabulary as described in claim 3, characterized in that, The method for generating output results including text and images is as follows: Convert the description of the illegal building into a token sequence; Input the token sequence into the Qwen Decoder layer. Natural language is generated through autoregressive decoding; The output results are presented in the natural language output format.

7. The method for detecting illegal buildings using open vocabulary as described in claim 1 or 2, characterized in that, The method for constructing a policy knowledge base is as follows: Use Qwen-VL's multimodal parsing function to extract textual and structural information from policy documents; The policy clauses in the policy documents are intelligently segmented using a language model, and key metadata is automatically annotated. The text of the policy document is transformed into a high-quality vector using Qwen-VL semantic embedding technology; The high-quality vectors are stored in a vector database to form a searchable policy knowledge base.

8. The method for detecting illegal buildings using open vocabulary as described in claim 1 or 2, characterized in that, The policy documents include structured or unstructured planning policy documents; the policy provisions include: building height restrictions, land use requirements, and standards for identifying violations of approval procedures; the key metadata includes the effective period, issuing authority, and applicable subject.

9. An open-vocabulary system for detecting illegal buildings, characterized in that, include: The image and text input module is used to obtain architectural images and natural language descriptions of illegal buildings; The multimodal feature fusion module is used to input the building image and the description of the illegal building into the multimodal feature fusion model, to outline the position in the building image corresponding to the description of the illegal building as a suspected illegal area, and to describe the suspected illegal area; The policy comparison module is used to convert the description of the illegal building into a semantic vector, compare the semantic vector with the content in the policy knowledge base, and obtain the policy content with the highest similarity. The selected positioning module is used to comprehensively judge whether there is a violation based on the policy content and the description of the suspected illegal area, and finally determine whether the building is illegal.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the open-vocabulary method for detecting illegal buildings as described in any one of claims 1-8.

Images