Deep learning-based construction violation patrol method, system and device, and medium

Through deep learning technology, the automatic processing of illegal construction inspection videos has been achieved, efficient illegal construction inspection has been solved, the problems of high cost of manual inspection and high misjudgment rate have been solved, and the accuracy and efficiency of illegal construction inspection have been improved.

CN118917798APending Publication Date: 2024-11-08GUANGZHOU GAOKE COMM TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410989030.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the existing technology, illegal construction inspections rely on manual inspections, resulting in high costs and high misjudgment rates, affecting efficiency.

Method used

Using a deep learning-based method, by obtaining illegal patrol videos, video stream detection, image recognition and visual language model analysis are carried out to obtain and analyze automated patrol results.

Benefits of technology

It reduces labor costs, reduces the rate of misjudgment, and improves the accuracy and efficiency of illegal construction inspections.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917798A_ABST
    Figure CN118917798A_ABST
Patent Text Reader

Abstract

The invention discloses a construction violation patrol method, system and device based on deep learning, and a medium. The method comprises the following steps: obtaining a construction violation patrol video; sending the construction violation patrol video to a server, and performing video stream detection processing on the construction violation patrol video through the server to obtain a detection image; sending the detection image to a client, and performing image recognition processing on the detection image through the client to obtain a construction violation image; and inputting the construction violation image into a visual language model for construction violation analysis processing to obtain a construction violation patrol result. The embodiment of the invention can improve the accuracy and efficiency of detection, and can be widely applied to the technical field of illegal construction patrol.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of illegal construction inspection, and in particular to a method, system, equipment and medium for illegal construction inspection based on deep learning. Background Art

[0002] At present, there are few monitoring measures for land resources, and it is difficult to detect illegal buildings in a timely manner. In the related technology, there is a manual inspection method to inspect illegal buildings, that is, to patrol on the spot manually or obtain corresponding images through drones, and then make judgments manually. However, in actual applications, it is found that these solutions require high labor costs, and manual judgment is prone to misjudgment, which affects the efficiency of illegal building inspections. In summary, the technical problems existing in the related technology need to be improved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a method, system, equipment and medium for illegal building inspection based on deep learning, which can improve the efficiency of illegal building inspection.

[0004] To achieve the above purpose, one aspect of an embodiment of the present application proposes a method for inspecting illegal buildings based on deep learning, the method comprising:

[0005] Obtain illegal construction inspection videos;

[0006] The illegal building inspection video is sent to a server, and the server performs video stream detection processing on the illegal building inspection video to obtain a detection image;

[0007] The detection image is sent to a client, and the client performs image recognition processing on the detection image to obtain an illegal building image;

[0008] The illegal building image is input into a visual language model to perform illegal building analysis and processing to obtain an illegal building inspection result.

[0009] In some embodiments, the step of performing video stream detection processing on the illegal building inspection video by the server to obtain a detection image includes the following steps:

[0010] The server performs request monitoring processing to obtain the illegal building inspection video;

[0011] The initialized first image recognition model is loaded into the stream analysis framework, and the stream analysis framework is used to perform stream interruption detection processing on the illegal construction inspection video to obtain an interruption result;

[0012] According to the interruption result, multi-task image recognition processing is performed on the illegal construction inspection video through the first image recognition model to obtain a detection image.

[0013] In some embodiments, performing multi-task image recognition processing on the illegal construction inspection video through the first image recognition model according to the interruption result to obtain a detection image includes the following steps:

[0014] When the interruption result is a video stream interruption, the stream analysis framework is restarted by an auxiliary service to obtain a restarted algorithm service;

[0015] Performing inference alarm processing on the illegal building inspection video through the algorithm service to obtain alarm information;

[0016] Performing image interception processing on the illegal building inspection video according to the alarm information to obtain an alarm image;

[0017] The warning image is subjected to warning analysis processing by using the first image recognition model to obtain a detection image.

[0018] In some embodiments, performing image recognition processing on the detection image by the client to obtain the illegal building image includes the following steps:

[0019] Training and testing a plurality of neural networks according to the environment of the client, and selecting a second image recognition model from the plurality of neural networks;

[0020] The second image recognition model is deployed on the client, and the second image recognition model is enabled through the client to perform image recognition processing on the detection image to obtain the illegal building image.

[0021] In some embodiments, inputting the illegal building image into a visual language model to perform illegal building analysis and processing to obtain illegal building inspection results includes the following steps:

[0022] Obtaining an input question, and inputting the illegal building image into the visual language model;

[0023] Performing multiple rounds of dialogue processing on the visual language model according to the input question to obtain dialogue content;

[0024] The illegal building image is detected, analyzed and processed according to the conversation content to obtain an illegal building inspection result.

[0025] In some embodiments, before inputting the illegal building image into the visual language model for illegal building analysis, the method further includes pre-training the visual language model, specifically including the following steps:

[0026] Get training images;

[0027] Performing description generation processing on the training image to obtain a text description;

[0028] determining the training image and the text description as training data;

[0029] Performing task generation processing on the training data by using the visual language model to obtain a generation result;

[0030] The visual language model is subjected to parameter adjustment processing according to the generated result to obtain the trained visual language model.

[0031] In some embodiments, the step of obtaining the illegal building inspection video includes the following steps:

[0032] Get the patrol area;

[0033] Deploy drone inspection routes according to the inspection areas;

[0034] The illegal building inspection video is obtained by shooting with a drone according to the drone inspection route.

[0035] To achieve the above purpose, another aspect of the embodiment of the present application proposes a deep learning-based illegal building inspection system, the system comprising:

[0036] The first module is used to obtain illegal building inspection videos;

[0037] The second module is used to send the illegal building inspection video to the server, and the server performs video stream detection processing on the illegal building inspection video to obtain a detection image;

[0038] The third module is used to send the detection image to the client, and the client performs image recognition processing on the detection image to obtain an illegal building image;

[0039] The fourth module is used to input the illegal building image into the visual language model to perform illegal building analysis and processing to obtain illegal building inspection results.

[0040] In some embodiments, the first module is used to obtain illegal building inspection videos, including:

[0041] The first unit is used to obtain the patrol area;

[0042] The second unit is used to deploy the drone inspection route according to the inspection area;

[0043] The third unit is used to obtain illegal building inspection videos by shooting with a drone according to the drone inspection route.

[0044] In some embodiments, the second module is used to send the illegal building inspection video to the server, and the server performs video stream detection processing on the illegal building inspection video to obtain a detection image, including:

[0045] The fourth unit is used to perform request monitoring processing through the server to obtain the illegal building inspection video;

[0046] A fifth unit is used to load the initialized first image recognition model into a stream analysis framework, and perform a stream interruption detection process on the illegal construction inspection video through the stream analysis framework to obtain an interruption result;

[0047] The sixth unit is used to perform multi-task image recognition processing on the illegal construction inspection video through the first image recognition model according to the interruption result to obtain a detection image.

[0048] In some embodiments, the sixth unit is used to perform multi-task image recognition processing on the illegal construction inspection video through the first image recognition model according to the interruption result to obtain a detection image, including:

[0049] The first subunit is used for, when the interruption result is a video stream interruption, performing an algorithm service restart process on the stream analysis framework through an auxiliary service to obtain a restarted algorithm service;

[0050] The second subunit is used to perform reasoning and alarm processing on the illegal building inspection video through the algorithm service to obtain alarm information;

[0051] The third subunit is used to perform image capture processing on the illegal building inspection video according to the alarm information to obtain an alarm image;

[0052] The fourth subunit is used to perform alarm analysis processing on the alarm image through the first image recognition model to obtain a detection image.

[0053] In some embodiments, the third module is used to send the detection image to the client, and perform image recognition processing on the detection image by the client to obtain the illegal building image, including:

[0054] A seventh unit is used to train and test multiple neural networks according to the environment of the client, and select a second image recognition model from the multiple neural networks;

[0055] The eighth unit is used to deploy the second image recognition model on the client, and enable the second image recognition model through the client to perform image recognition processing on the detection image to obtain an illegal building image.

[0056] In some embodiments, the fourth module is used to input the illegal building image into the visual language model to perform illegal building analysis and processing to obtain illegal building inspection results, including:

[0057] A ninth unit is used to obtain an input question and input the illegal building image into the visual language model;

[0058] A tenth unit is used for performing multiple rounds of dialogue processing on the visual language model according to the input question to obtain dialogue content;

[0059] The eleventh unit is used to detect, analyze and process the illegal building image according to the conversation content to obtain illegal building inspection results.

[0060] In some embodiments, before the fourth module is used to input the illegal building image into the visual language model for illegal building analysis, the system further includes a fifth module for pre-training the visual language model, specifically including:

[0061] Unit 12, for acquiring training images;

[0062] A thirteenth unit is used to perform description generation processing on the training image to obtain a text description;

[0063] A fourteenth unit, configured to determine the training image and the text description as training data;

[0064] A fifteenth unit is used to perform task generation processing on the training data through the visual language model to obtain a generation result;

[0065] The sixteenth unit is used to perform parameter adjustment processing on the visual language model according to the generated result to obtain the trained visual language model.

[0066] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0067] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0068] The embodiments of the present application include at least the following beneficial effects: The present application provides a method, system, device and medium for illegal building inspection based on deep learning, which obtains illegal building inspection videos; sends the illegal building inspection videos to the server, performs video stream detection processing on the illegal building inspection videos by the server, and obtains detection images; sends the detection images to the client, performs image recognition processing on the detection images by the client, and obtains illegal building images; inputs the illegal building images into the visual language model for illegal building analysis processing, and obtains illegal building inspection results. The embodiments of the present application perform real-time detection of the video stream by the server, send the detected images to the client for secondary recognition, and perform illegal building analysis by the visual language model, which can alleviate the problem of high labor costs, reduce the probability of misjudgment, and help improve the accuracy and efficiency of illegal building inspections. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a flow chart of an illegal construction inspection method based on deep learning provided in an embodiment of the present application;

[0070] Figure 2 is a flow chart of a video stream detection provided by an embodiment of the present application;

[0071] Figure 3 It is a structural schematic diagram of an illegal construction inspection system based on deep learning provided in an embodiment of the present application;

[0072] Figure 4 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0074] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0075] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0077] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0078] 1) Artificial Intelligence (AI) is an important part of the intelligent discipline. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial Intelligence is a very broad science, including robotics, speech recognition, image recognition, natural language processing, expert systems, machine learning, computer vision, etc.

[0079] 2) Image recognition refers to the technology of using computers to process, analyze and understand images to identify targets and objects of various different patterns. It is a practical application of deep learning algorithms.

[0080] 3) Deep learning (DL) is a new research direction in the field of machine learning (ML). It is introduced into machine learning to make it closer to its original goal - artificial intelligence (AI). Deep learning is the learning of the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds.

[0081] In the related technology, there is a manual inspection method for illegal construction, that is, through manual on-site patrol or through drones to obtain image data for inspection, and then manually judge the illegal construction based on the images taken by the drone. However, in actual application, it is found that this method consumes high labor costs, and manual judgment is prone to misjudgment, which affects the efficiency of illegal construction inspection.

[0082] In view of this, an illegal construction inspection method, system, device and medium based on deep learning are provided in the embodiments of the present application. The scheme obtains illegal construction inspection videos; sends the illegal construction inspection videos to the server, performs video stream detection processing on the illegal construction inspection videos by the server, and obtains detection images; sends the detection images to the client, performs image recognition processing on the detection images by the client, and obtains illegal construction images; inputs the illegal construction images into the visual language model for illegal construction analysis processing, and obtains illegal construction inspection results. The embodiments of the present application perform real-time detection of the video stream by the server, send the detected images to the client for secondary recognition, and perform illegal construction analysis by the visual language model, which can alleviate the problem of high labor costs, reduce the probability of misjudgment, and help improve the accuracy and efficiency of illegal construction inspections.

[0083] The embodiment of the present application provides a method for inspecting illegal buildings based on deep learning, which relates to the field of artificial intelligence technology. The embodiment of the present application provides a method for inspecting illegal buildings based on deep learning, which can be applied to a terminal, a server, or software running in a terminal or a server. In some embodiments, the terminal can be a drone, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a method for inspecting illegal buildings based on deep learning, etc., but is not limited to the above forms.

[0084] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0085] Figure 1 is an optional flow chart of an illegal construction inspection method based on deep learning provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0086] Step S101, obtaining illegal building inspection video;

[0087] Step S102, sending the illegal building inspection video to a server, and performing video stream detection processing on the illegal building inspection video by the server to obtain a detection image;

[0088] Step S103, sending the detection image to a client, and performing image recognition processing on the detection image by the client to obtain an illegal building image;

[0089] Step S104: input the illegal building image into a visual language model to perform illegal building analysis and processing to obtain illegal building inspection results.

[0090] In the steps S101 to S104 shown in the embodiment of the present application, the illegal building inspection video is obtained by pre-deployed cameras or drones, and the illegal building inspection video is obtained by shooting or recording the buildings in the inspection area. Then the illegal building inspection is sent to the server for video stream detection, and the real-time video stream detection function is realized by the stream processing framework deployed on the server, and the illegal building phenomenon appearing in the illegal building inspection video is screenshotted to obtain a detection image. Then the detection image is sent to the client, and the detection image is processed by the client using an image recognition model to obtain an illegal building image, which is to perform image recognition on the detection image to detect whether there is an illegal building phenomenon, and the detection image with the illegal building phenomenon is identified as an illegal building image. Finally, the illegal building image is input into the visual language model for illegal building analysis, and the illegal building image is further detected by the visual language model to obtain the illegal building inspection result. It can be imagined that if the illegal building inspection result is that there is an illegal building behavior, the text-to-speech technology is used to play the illegal building inspection result in real time, such as the existence of illegal building behavior in the area, to remind the staff to review. The embodiment of the present application performs video stream detection on illegal building inspection videos through the server, and then performs image recognition processing through the client, and combines the visual language model to perform illegal building analysis processing, which can improve the accuracy of detection, reduce labor costs, and thus improve the efficiency of illegal building inspections.

[0091] In step S101 of some embodiments, obtaining the illegal building inspection video includes the following steps:

[0092] Get the patrol area;

[0093] Deploy drone inspection routes according to the inspection areas;

[0094] The illegal building inspection video is obtained by shooting with a drone according to the drone inspection route.

[0095] In the embodiment of the present application, the illegal building inspection video can be obtained by shooting with a drone, or by pre-deployed cameras. In the embodiment of the present application, a drone is used as an example. First, the inspection area where illegal building inspection is required is determined. Then, the inspection route of the drone is deployed according to the inspection area, so that the shooting area of ​​the drone can accurately cover the entire inspection area. Finally, the drone is dispatched to inspect illegal buildings according to the drone inspection route, and the illegal building inspection video is obtained by shooting. The embodiment of the present application combines the monitoring feedback data of high-altitude inspection machines such as drones, and performs illegal building analysis through artificial intelligence technology, which can replace manual inspections and improve the efficiency and accuracy of illegal building inspections.

[0096] In step S102 of some embodiments, performing video stream detection processing on the illegal building inspection video by the server to obtain a detection image includes the following steps:

[0097] The server performs request monitoring processing to obtain the illegal building inspection video;

[0098] The initialized first image recognition model is loaded into the stream analysis framework, and the stream analysis framework is used to perform stream interruption detection processing on the illegal construction inspection video to obtain an interruption result;

[0099] According to the interruption result, multi-task image recognition processing is performed on the illegal construction inspection video through the first image recognition model to obtain a detection image.

[0100] In the embodiment of the present application, after obtaining the illegal construction inspection video, the client will transmit the illegal construction inspection video to the service end, i.e., the server, and perform real-time video stream detection and processing through the server, and intercept the pictures of illegal construction behaviors in the illegal construction inspection video to obtain the detection image. Figure 2, the client encodes the illegal construction inspection video and sends it to the server. The first image recognition model is deployed in the server, wherein the first image recognition model is a YOLOv5s model, which has the advantage of fast speed, and its accuracy can meet the real-time detection requirements. The model is initialized, and then the client request is monitored, and the illegal construction inspection video is obtained by receiving the client request. The initialized first image recognition model is loaded into the stream analysis framework, and the stream analysis framework can adopt the DeepStream framework, which is a streaming analysis toolkit for building artificial intelligence-driven applications. The illegal construction inspection video is subjected to a push stream interruption detection process through the stream analysis framework to obtain an interruption result, and the illegal construction inspection video is subjected to a multi-task image recognition process through the first image recognition model according to the interruption result to obtain a detection image, and the illegal construction inspection video is specifically decoded, and the detection image is output after feature extraction and classification. The embodiment of the present application performs real-time video stream detection through the server, and can quickly and accurately detect images with illegal construction phenomena in the video stream.

[0101] In some embodiments, performing multi-task image recognition processing on the illegal construction inspection video through the first image recognition model according to the interruption result to obtain a detection image includes the following steps:

[0102] When the interruption result is a video stream interruption, the stream analysis framework is restarted by an auxiliary service to obtain a restarted algorithm service;

[0103] Performing inference alarm processing on the illegal building inspection video through the algorithm service to obtain alarm information;

[0104] Performing image interception processing on the illegal building inspection video according to the alarm information to obtain an alarm image;

[0105] The warning image is subjected to warning analysis processing by using the first image recognition model to obtain a detection image.

[0106] In an embodiment of the present application, when the interruption result is a video stream interruption, the algorithm service restart processing is performed on the stream analysis framework through the auxiliary service to obtain the restarted algorithm service, and the auxiliary service can be set to prevent the occurrence of interruption, and the interruption recovery capability can be realized immediately. Then, the algorithm service is used to perform reasoning alarm processing on the illegal construction inspection video, and the alarm logic is added to the framework. The alarm logic is used to determine whether an illegal construction behavior has occurred. For example, if a building appears in a specified area, an alarm processing is performed, and the corresponding alarm information is displayed. The alarm information is used to indicate the image of the illegal construction behavior and the corresponding illegal construction behavior, so that according to the alarm information, an alarm image is obtained from the illegal construction inspection video. The alarm image is an image where the alarm information appears. Finally, the alarm image is analyzed and processed by the first image recognition model, and the illegal construction behavior is identified and detected by the image recognition model to obtain a detection image. It should be noted that the embodiment of the present application can start multiple image recognition tasks if the server allows, which is not limited to the recognition of a single category, and can realize multi-category recognition at the same time.

[0107] In some embodiments, performing image recognition processing on the detection image by the client to obtain the illegal building image includes the following steps:

[0108] Training and testing a plurality of neural networks according to the environment of the client, and selecting a second image recognition model from the plurality of neural networks;

[0109] The second image recognition model is deployed on the client, and the second image recognition model is enabled through the client to perform image recognition processing on the detection image to obtain the illegal building image.

[0110] In an embodiment of the present application, multiple neural networks are trained and tested according to the environment of the client. The neural network is a YOLOv5 model, including multiple neural networks such as YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5 l and YOLv5x. Since the environment of each client is different, it is necessary to train and test multiple neural networks according to the actual scene during deployment, and select a second image recognition model from multiple neural networks according to the test results. Finally, the second image recognition model is deployed on the client, and the second image recognition model is enabled by the client to perform image recognition processing on the detection image to obtain an illegal building image. Since the video stream detection deployed on the server is a model that has been quantized and compressed, the detection accuracy is lower than that of the image recognition accuracy. The image recognition model deployed on the client is trained by the original pytorch framework, which has not been compressed and quantized by the model, and has the advantage of high accuracy, but the detection speed is slow and real-time video stream detection cannot be achieved. Therefore, the second image recognition model is deployed on the client to recognize the detection image again, the purpose is to reduce the impact of false alarms and improve the accuracy of illegal building inspections.

[0111] In some embodiments, inputting the illegal building image into a visual language model to perform illegal building analysis and processing to obtain illegal building inspection results includes the following steps:

[0112] Obtaining an input question, and inputting the illegal building image into the visual language model;

[0113] Performing multiple rounds of dialogue processing on the visual language model according to the input question to obtain dialogue content;

[0114] The illegal building image is detected, analyzed and processed according to the conversation content to obtain an illegal building inspection result.

[0115] In an embodiment of the present application, the image of the illegal building is input into the visual language model for illegal building analysis and processing. The visual language model is a large visual model of image-generated text. The model is a new generation of multimodal large model with powerful document image understanding capabilities, and can optimize the interaction between visual and language modalities through a deep fusion strategy. First, the image of the illegal building is input into the visual language model, and then the content of the conversation is determined by inputting questions or texts and conducting a single or multiple rounds of conversations with the visual language model. The illegal building image is detected and analyzed according to the content of the conversation through the visual language model, thereby obtaining the illegal building inspection results corresponding to the content of the conversation. For example, the input question is "Does the picture contain illegal construction?" The content of the conversation is understood through the visual language model, thereby analyzing the illegal building image and outputting the result of whether the illegal building image contains illegal construction. The embodiment of the present application can improve the accuracy of detection by using a visual language model for recognition and detection.

[0116] In some embodiments, before inputting the illegal building image into the visual language model for illegal building analysis, the method further includes pre-training the visual language model, specifically including the following steps:

[0117] Get training images;

[0118] Performing description generation processing on the training image to obtain a text description;

[0119] determining the training image and the text description as training data;

[0120] Performing task generation processing on the training data by using the visual language model to obtain a generation result;

[0121] The visual language model is subjected to parameter adjustment processing according to the generated result to obtain the trained visual language model.

[0122] In an embodiment of the present application, a training image with illegal construction is obtained, and a corresponding description generation process is performed according to the training image, that is, the corresponding text content is generated for the content of the training image. The text description can be generated according to the image recognition model, or it can be generated by artificial judgment. The text description is used to describe whether the image has illegal construction, and the text description is saved in the text, and the text is named with the name of the corresponding training image. The corresponding content can be obtained by reading the text content to the description element of the training image, so as to avoid the phenomenon of the content with the same name being covered. Then, the training data is task generated by the visual language model to obtain a generation result. The task generation can be a judgment result for generating illegal construction behavior, and the generation result is to judge whether the training image has illegal construction behavior. Finally, the loss function of the model is optimized according to the generation result, and the model loss of the loss function is back-propagated, and the model parameters are continuously adjusted until the similarity is greater than or equal to the similarity threshold, and the optimization of the model is stopped to obtain a visual language model that meets the requirements.

[0123] The following is a detailed description of the embodiments of the present invention with reference to specific application examples:

[0124] The embodiments of the present application can be applied to the scene of illegal construction detection, by deploying cameras in the detection area or using drones to cruise to obtain the corresponding illegal construction detection video, and by sending the illegal construction detection video to the server for real-time video stream detection processing, the image of illegal construction in the video is identified and intercepted to obtain the detection image. The detection image is returned to the client by the server, and an image recognition model is deployed in the client to perform secondary recognition on the detection image to obtain the illegal construction image. Among them, the image recognition models deployed in the server and the client are two different models. The image recognition model deployed on the server has the advantage of fast speed and can perform real-time detection, while the image recognition model deployed on the client has the advantage of high precision and can perform accurate recognition. The impact of false alarms can be reduced through secondary recognition. Finally, the illegal construction image is input into the visual language model for illegal construction analysis and processing, and the recognition analysis is performed by combining the deep fusion of visual language features to obtain the result of whether there is illegal construction in the detection area.

[0125] See also Figure 3 The embodiment of the present application also provides an illegal construction inspection system based on deep learning, which can implement the illegal construction inspection method based on deep learning. The system includes:

[0126] The first module 301 is used to obtain illegal building inspection videos;

[0127] The second module 302 is used to send the illegal building inspection video to the server, and the server performs video stream detection processing on the illegal building inspection video to obtain a detection image;

[0128] The third module 303 is used to send the detection image to the client, and the client performs image recognition processing on the detection image to obtain an illegal building image;

[0129] The fourth module 304 is used to input the illegal building image into the visual language model to perform illegal building analysis and processing to obtain illegal building inspection results.

[0130] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0131] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned illegal construction inspection method based on deep learning when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0132] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0133] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0134] The processor 401 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0135] The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 402, and the processor 401 calls and executes the illegal construction inspection method based on deep learning in the embodiment of this application;

[0136] Input / output interface 403, used to implement information input and output;

[0137] Communication interface 404, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0138] Bus 405 , which transmits information between various components of the device (e.g., processor 401 , memory 402 , input / output interface 403 , and communication interface 404 );

[0139] The processor 401 , the memory 402 , the input / output interface 403 and the communication interface 404 are connected to each other in communication within the device via the bus 405 .

[0140] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned illegal construction inspection method based on deep learning is implemented.

[0141] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0142] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0143] The embodiments of the present application provide a method, system, device and medium for illegal building inspection based on deep learning. The scheme obtains illegal building inspection videos; sends the illegal building inspection videos to the server, performs video stream detection processing on the illegal building inspection videos by the server, and obtains detection images; sends the detection images to the client, performs image recognition processing on the detection images by the client, and obtains illegal building images; inputs the illegal building images into the visual language model for illegal building analysis processing, and obtains illegal building inspection results. The embodiments of the present application perform real-time detection of the video stream by the server, send the detected images to the client for secondary recognition, and perform illegal building analysis through the visual language model, which can alleviate the problem of high labor costs, reduce the probability of misjudgment, and help improve the accuracy and efficiency of illegal building inspections.

[0144] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0145] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0146] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0147] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0148] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0149] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0150] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0151] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0154] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for inspecting illegal buildings based on deep learning, characterized in that: The method comprises the following steps: Obtain illegal construction inspection videos; The illegal building inspection video is sent to a server, and the server performs video stream detection processing on the illegal building inspection video to obtain a detection image; The detection image is sent to a client, and the client performs image recognition processing on the detection image to obtain an illegal building image; The illegal building image is input into a visual language model to perform illegal building analysis and processing to obtain an illegal building inspection result.

2. The method according to claim 1, characterized in that The step of performing video stream detection processing on the illegal building inspection video by the server to obtain a detection image includes the following steps: The server performs request monitoring processing to obtain the illegal building inspection video; The initialized first image recognition model is loaded into the stream analysis framework, and the stream analysis framework is used to perform stream interruption detection processing on the illegal construction inspection video to obtain an interruption result; According to the interruption result, multi-task image recognition processing is performed on the illegal construction inspection video through the first image recognition model to obtain a detection image.

3. The method according to claim 2, characterized in that The step of performing multi-task image recognition processing on the illegal construction inspection video by using the first image recognition model according to the interruption result to obtain a detection image comprises the following steps: When the interruption result is a video stream interruption, the stream analysis framework is restarted by an auxiliary service to obtain a restarted algorithm service; Performing inference alarm processing on the illegal building inspection video through the algorithm service to obtain alarm information; Performing image interception processing on the illegal building inspection video according to the alarm information to obtain an alarm image; The warning image is subjected to warning analysis processing by using the first image recognition model to obtain a detection image.

4. The method according to claim 1, characterized in that The step of performing image recognition processing on the detection image by the client to obtain an illegal building image comprises the following steps: Training and testing a plurality of neural networks according to the environment of the client, and selecting a second image recognition model from the plurality of neural networks; The second image recognition model is deployed on the client, and the second image recognition model is enabled through the client to perform image recognition processing on the detection image to obtain the illegal building image.

5. The method according to claim 1, characterized in that The step of inputting the illegal building image into a visual language model to perform illegal building analysis and processing to obtain illegal building inspection results comprises the following steps: Obtaining an input question, and inputting the illegal building image into the visual language model; Performing multiple rounds of dialogue processing on the visual language model according to the input question to obtain dialogue content; The illegal building image is detected, analyzed and processed according to the conversation content to obtain an illegal building inspection result.

6. The method according to claim 5, characterized in that Before inputting the illegal building image into the visual language model for illegal building analysis, the method further includes pre-training the visual language model, specifically including the following steps: Get training images; Performing description generation processing on the training image to obtain a text description; determining the training image and the text description as training data; Performing task generation processing on the training data by using the visual language model to obtain a generation result; The visual language model is subjected to parameter adjustment processing according to the generated result to obtain the trained visual language model.

7. The method according to any one of claims 1 to 6, characterized in that: The method of obtaining the illegal building inspection video includes the following steps: Get the patrol area; Deploy drone inspection routes according to the inspection areas; The illegal building inspection video is obtained by shooting with a drone according to the drone inspection route.

8. A deep learning-based illegal construction inspection system, characterized in that: The system comprises: The first module is used to obtain illegal building inspection videos; The second module is used to send the illegal building inspection video to the server, and the server performs video stream detection processing on the illegal building inspection video to obtain a detection image; The third module is used to send the detection image to the client, and the client performs image recognition processing on the detection image to obtain an illegal building image; The fourth module is used to input the illegal building image into the visual language model to perform illegal building analysis and processing to obtain illegal building inspection results.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.