Airport boundary detection method and system based on multiple modes

Abstract

The invention belongs to the technical field of airport boundary security and protection, and discloses an airport boundary detection method and system based on multiple modes. According to the method, the environment around the enclosure is analyzed in real time through the video personnel detection module, and whether a person approaches the enclosure or not is observed; detecting whether the prompt word and the content of the image are illegal or not, and if yes, not processing; analyzing the content in the image through the image understanding capability, the text understanding capability and the analysis capability of the multi-modal visual large model, and returning an analysis result; judging whether the input text content has the intention of damaging and invading the enclosure or not; if there is no intention of invading and damaging the enclosure, the abnormal alarm module is directly used. According to the method, a large model and traditional deep learning are combined, so that the accuracy and the overall efficiency of a security system can be remarkably improved. The ever-increasing safety requirement of the civil aviation industry is met, and new powerful power is injected for continuous and healthy development of the unmanned aerial vehicle.

Description

Technical Field

[0001] The present invention belongs to the technical field of airport perimeter security, and in particular relates to a multi-modal airport perimeter detection method and system. Background Art

[0002] With the vigorous development of the civil aviation industry, the scale of airports has continued to expand, and the passenger throughput has continued to soar. Against this background, the security needs of airports have become increasingly important. In this context, ensuring the safety of aircraft parking has become increasingly important. Unauthorized personnel illegally intruding into the airport runway or approaching parked aircraft poses a great threat to flight safety. These safety accidents may not only cause damage to the aircraft, but also cause casualties, bringing huge losses to airlines and society.

[0003] Traditional airport security measures, such as manual inspections and video camera monitoring, can play a preventive role to a certain extent, but in the face of the increasingly complex and changing security situation, these measures have become inadequate. Manual inspections are limited by manpower and time, and cannot provide 24-hour uninterrupted monitoring; and although video cameras can capture images in real time, their efficiency and accuracy are difficult to guarantee in the face of large-scale and multi-region monitoring needs. Therefore, in order to meet the needs of airport perimeter security, it is crucial to introduce more advanced and efficient security identification methods.

[0004] Through the above analysis, the problems and defects of the existing technology are as follows: At present, most airport perimeter security adopts video cameras plus visual deep learning algorithms to conduct real-time analysis of the airport perimeter range. This method can better detect whether there is any illegal crossing of the perimeter, but the traditional deep learning algorithm requires a large amount of data from related scenes for training. In the scenario of airport perimeters, it is difficult to provide a large amount of video data of illegal crossing of the perimeter, and it is impossible to provide data on scenes such as destruction, collision, and digging holes into the perimeter. In the face of growing safety and efficiency requirements, only using real-time video analysis methods to detect human intrusion cannot well meet the needs of diverse scenarios. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the disclosed embodiments of the present invention provide a multi-modal airport perimeter detection method and system.

[0006] The technical solution is as follows: A multi-modal airport perimeter detection method, comprising: S1, using the video personnel detection module to analyze the environment around the enclosure in real time to detect whether anyone is approaching the enclosure; S2, after detecting the information of the person approaching the fence, the image and the pre-set prompt words are transmitted to the legality analysis module to detect whether the prompt words and the content of the image are illegal. If they are illegal, no processing will be done; S3, transmits the image and prompt word content that have passed the legality detection to the multimodal image analysis module. The multimodal visual model uses the input data stage of the open source LLM to add a multimodal feature fusion layer to fuse the image features and prompt word vectors to form a multimodal feature representation; the image content is analyzed and the analysis results are returned through the image understanding, text understanding and analysis capabilities of the multimodal visual model; S4, transmitting the result of the multimodal image analysis module to the intention recognition module, and judging whether the input text content has the intention of destroying or invading the boundary through the intention recognition module; S5, if there is no intention to invade or damage the perimeter, the abnormal alarm module will be used to play a warning to leave through the loudspeakers around the perimeter; if there is a situation of destroying or invading the perimeter, the host computer system will alarm the airport's relevant supervisors and play a voice to stop the destruction and intrusion through the loudspeakers around the perimeter.

[0007] In step S1, it is detected whether someone is close to the fence, including: when someone is found near the fence, the multimodal image analysis module is triggered, the image data is transmitted to the legality analysis module, and the visual model based on the neural network is used to determine whether the image is legal content, so as to avoid transmitting illegal image data to the multimodal visual large model due to hackers. If it is legal content, the image is transmitted to the multimodal image analysis module.

[0008] In step S2, when the image and the prompt word pass the verification, the image and the prompt word are transmitted to the multimodal image analysis module.

[0009] In step S3, the image content is analyzed and the analysis results are returned through the image understanding, text understanding, and analysis capabilities of the multimodal visual large model, including: (1) Combining the received image data with the pre-customized prompt word; (2) The combined data is transmitted to the multimodal visual big model, and the content in the image is analyzed through the understanding and analysis capabilities of the multimodal visual big model; the preset boost words are combined with the input image, and the multimodal visual big model input with the prompt words and image data returns a description of the image; (3) The output content of the multimodal visual model is transmitted to the intention recognition algorithm. The intention recognition algorithm analyzes the intention of the transmitted content. If there is an intrusion intention, the system will alarm the user. The expression is: In the formula, express Model results, Indicates the results of keyword search. Represents the result of vector search, The value is 0 or 1, 0 means no intrusion intention, and 1 means there is intrusion intention.

[0010] In step (2), the content in the image is analyzed through the understanding and analysis capabilities of the multimodal visual large model, including: (2.1) Fence image data collection: train the multimodal vision big model through unsupervised methods to enhance the multimodal vision big model's understanding of the fence, and collect fence images around multiple airports, fence images generated by image enhancement technology, fence images generated by AIGC technology, and fence image data crawled by crawlers; (2.2) Dataset generation: annotating the collected image data of the boundary, including the questions and answers corresponding to the images; (2.3) Fine-tune the algorithm and align it. Fine-tune the multimodal visual model using the generated dataset, adjust the model hyperparameters, and input the labeled data into the multimodal visual model in batches for training. This ensures that the output of the multimodal visual model is consistent with the content of the perimeter security scenario.

[0011] In step S4, the intention recognition module determines whether the input text content has the intention of destroying or invading the boundary, including: 4.1, Collection of intent data, collection through prompt words and large language model to generate corpus, manual writing, natural language data enhancement to generate corresponding text data; through manual writing and setting prompt words and using large models to imitate the manually written prompt words to generate approximate data; 4.2, Data annotation, first set the prompt word prompt with the pre-set intent category to use the large language model to determine which intent the input corpus belongs to, and organize the data set according to the results returned by the large model language type; after the data set is generated, manual error correction is performed to correct the incorrectly identified intent; 4.3, Intent recognition algorithm training, use transformer architecture and multiple open source algorithms for training, organize the data according to Example 2, and after training, inherit the model to the intent recognition module.

[0012] Another object of the present invention is to provide a multi-modal airport fence detection system, which implements the multi-modal airport fence detection method, and the system comprises: The video personnel detection module is used to detect whether there is a person approaching the fence. When a person is found near the fence, the multimodal image analysis module is triggered to transmit the image data to the legality analysis module to determine whether the image is legal content to avoid transmitting illegal image data to the multimodal visual model due to hackers. If it is legal content, the image is transmitted to the multimodal image analysis module; The legality analysis module is composed of two deep learning algorithm models, including a trasnformer structure NLP model responsible for judging whether the text prompt word is illegal, and a neural network visual classification model for judging whether the image is illegal. The image and prompt word are transmitted to the legality analysis module for unified recognition and judgment. If there is illegal information, it will not be processed directly. It is used to analyze whether the received image and prompt word are legal. The algorithm model is used to detect whether the input image and prompt word contain illegal content such as pornography, violence and terrorism. Illegal content will be directly returned without processing. When the image and prompt word pass the verification, the image and prompt word are transmitted to the multimodal image analysis module. The multimodal image analysis module includes a multimodal visual model and a deep learning intention recognition algorithm. The multimodal visual model is responsible for analyzing and describing the objects and behaviors in the image through images and prompt words, and the intention recognition algorithm is responsible for performing intention analysis on the description content output by the multimodal visual model through pre-set behavior scenarios. The intention recognition module is used to judge the intention of the output of the multimodal image analysis module to determine whether the output content has the intention of invading the fence. If it is confirmed that people, cars and other objects in the image are invading the fence, a red alarm is directly sent to the user in the system and a warning sound is sent in the invading location area; if there is no invading intention in the image but there is a stay or wait-and-see situation, a prompt alarm is sent in the system and a warning sound is sent in the corresponding location area to stay away from the fence.

[0013] Furthermore, the multimodal-based airport perimeter detection system is mounted on a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, and the processor implements the functions of the multimodal-based airport perimeter detection system described above when executing the computer program.

[0014] Furthermore, the multimodal-based airport perimeter detection system is mounted on a computer-readable storage medium, and the computer-readable storage medium stores a computer program, which can realize the functions of the above-mentioned multimodal-based airport perimeter detection system when executed by a processor.

[0015] Furthermore, the multimodal airport perimeter detection system is mounted on an information data processing terminal, and the information data processing terminal is used to provide a user input interface to implement the functions of the multimodal airport perimeter detection system when executed on an electronic device.

[0016] Combining all the above technical solutions, the beneficial effects of the present invention are as follows: The present invention provides a multimodal-based method that utilizes the understanding and analysis capabilities of a large model combined with traditional deep learning to solve the problem of not being able to provide a large amount of training data and low recognition accuracy in perimeter security scenarios, and meet the needs of airport perimeter security for diversified and multi-scenario security. The present invention uses a method that combines a large model with traditional deep learning, which can significantly improve the accuracy and overall efficiency of the security system. This not only meets the growing safety needs of the civil aviation industry, but also injects a strong new impetus into its sustainable and healthy development.

[0017] The present invention improves the accuracy of intrusion behavior detection through a multi-module and multi-process approach, reducing the drawbacks of traditional visual solutions that require massive amounts of data for training and the problem that training data for multiple scenarios is difficult to collect. The method is highly scalable and can self-select to replace each algorithm in each module. Currently, the perimeters of major airports are mostly processed using manual inspections and video recognition solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure; Figure 1 is a flow chart of a multi-modal airport boundary detection method provided by an embodiment of the present invention; Figure 2 is a schematic diagram of a multi-modal airport perimeter detection method provided by an embodiment of the present invention; Figure 3 is a schematic diagram of a multi-modal airport perimeter detection system provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of the optimization principle of multimodal visual large model training provided by an embodiment of the present invention; In the figure: 1. Video person detection module; 2. Legality analysis module; 3. Multimodal image analysis module; 4. Intent recognition module. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0020] The innovation of the present invention is that the present invention solves the problem that traditional security perimeter intrusion detection uses visual algorithms for identification and analysis, but there is little training data, resulting in insufficient coverage of intrusion scenarios and inaccurate identification. This method combines the ability of large models to understand images and texts to cover and analyze a variety of complex intrusion behavior scenarios that are difficult to occur in normal situations. It refines the intrusion detection scenarios, improves the detection accuracy, and reduces the false alarm rate.

[0021] Embodiment 1, as Figure 1 As shown, the airport boundary detection method based on multi-modality provided by the embodiment of the present invention includes: S1, using the video personnel detection module to analyze the environment around the enclosure in real time to detect whether anyone is approaching the enclosure; S2, after detecting the information of the person approaching the fence, the image and the pre-set prompt words are transmitted to the legality analysis module to detect whether the prompt words and the content of the image are illegal. If they are illegal, no processing will be done; The legality analysis module consists of two deep learning algorithm models and a vector library of illegal words. The vector library of illegal words mainly consists of some texts. The text is vectorized by the model and stored in the vector database. The deep learning algorithm model includes an NLP model based on the trasnformer structure to determine whether the text prompt word is illegal, and a neural network visual classification model to determine whether the image is illegal. After the text data is vectorized at the same time, it will be similar to the data in the vector library. If the similarity is greater than the threshold, the task has illegal content. The image and prompt word text are passed to the legality analysis module. The NLP model and visual classification model in the legality analysis and the vector library retrieval method are used to determine whether the text and image are illegal. Voting is performed based on the results of the three methods. If one of the results shows a violation, the alarm content is illegal. Verifying the content in three ways improves the stability of the module.

[0022] S3, transmits the image and prompt word content that have passed the legality detection to the multimodal image analysis module. The multimodal visual model uses the input data stage of the open source LLM to add a multimodal feature fusion layer to fuse the image features and prompt word vectors to form a multimodal feature representation; the image content is analyzed and the analysis results are returned through the image understanding, text understanding and analysis capabilities of the multimodal visual model; S4, transmitting the result of the multimodal image analysis module to the intention recognition module, and judging whether the input text content has the intention of destroying or invading the boundary through the intention recognition module; The intention recognition module has three submodules: nlp intrusion intention classification model, keyword retrieval module and vector similarity matching module. The nlp intrusion intention classification model is a classification model based on the trasnformer structure to determine whether the input text has intrusion intention. The keyword retrieval module searches the input text through the preset security intrusion keywords. If there are more than 3 or more, the task has intrusion intention. The intention recognition vector library converts the historical and collected intrusion intention texts into vectors through the model and stores them in the vector database. When verifying the input text, the input text is first converted into a vector, and then the vector is compared with the data stored in the vector library. If it exceeds the threshold, it is considered that there is an intrusion intention. If there are multiple data exceeding the threshold, the stronger the intrusion intention of the input text is considered. The output data of these three modules are voted. If there are two votes to determine that there is an intrusion intention, the behavior of destroying and intruding the boundary is returned; S5, if there is no intention to invade or damage the perimeter, the abnormal alarm module will be used to play a warning to leave through the loudspeakers around the perimeter; if there is a situation of destroying or invading the perimeter, the host computer system will alarm the airport's relevant supervisors and play a voice to stop the destruction and intrusion through the loudspeakers around the perimeter.

[0023] For example, Figure 2 This is the principle of the airport perimeter detection method based on multi-modality provided by an embodiment of the present invention.

[0024] Embodiment 2, as Figure 3 As shown, the airport perimeter detection method system based on multi-modality provided by the embodiment of the present invention includes: The video personnel detection module 1 is responsible for detecting whether there are people approaching the fence. Through the traditional visual target detection algorithm, it accesses the real-time video data stream for analysis to detect whether there are people staying or wandering around the fence. When someone is found near the fence, the multimodal image analysis module is triggered, and the image data is transmitted to the legality analysis module to determine whether the image is legal content to avoid transmitting illegal image data to the multimodal visual large model due to hackers and other reasons. If it is legal content, the image is transmitted to the multimodal image analysis module.

[0025] Legality analysis module 2: This module mainly analyzes whether the received image and prompt word are legal. It uses the algorithm model to detect whether the input image and prompt word contain illegal content such as pornography, violence and terrorism. Illegal content will be directly returned without processing. When the image and prompt word pass the verification, the image and prompt word are transmitted to the multimodal image analysis module.

[0026] Multimodal image analysis module 3, which mainly includes two parts: a multimodal visual model and a deep learning intention recognition algorithm. The multimodal visual model is mainly responsible for analyzing and describing the objects, behaviors, etc. in the image through images and prompt words, and the intention recognition algorithm is mainly responsible for performing intention analysis on the description content output by the multimodal visual model through pre-set behavior scenarios; The multimodal visual big model uses the open source LLM. In the input data stage of the open source LLM, a multimodal feature fusion layer is added to fuse the image features and the prompt word vector to form a multimodal feature representation. The multimodal feature fusion layer uses the complementarity of images and texts to enhance the big model's ability to understand text and image information. After the image and the prompt word enter the model, the multimodal feature fusion layer is used to perform feature fusion, and the feature vector of the input text is fused with the feature map of the input image. The feature vector is transformed to the same dimension and input to the LLM. According to the pre-set scene prompt words, multiple prompt words and images are sent to the multimodal big model respectively. According to the return value of the big model, it is judged whether there are several preset behaviors of invading the enclosure in this image. For example: intention scenes such as climbing over, hitting, destroying, digging holes, and staying. If the intention analyzed by the algorithm contains the situation of invading the enclosure, an alarm prompt will be issued in the system.

[0027] The processing flow includes: (1) Combine the received image data with the pre-customized prompt word prompt.

[0028] The image and each prompt word form a key-value pair and request the multimodal large model separately. Separate requests can improve the accuracy of the multimodal large model.

[0029] (2) The multimodal visual big model is requested for the combined data, and the content in the image is analyzed through the understanding and analysis capabilities of the multimodal visual big model. The input image is combined with the preset boost words, for example, describing the objects in the image and the behavior in the objects. The multimodal visual big model input with the prompt words and image data will return a description of the image.

[0030] (3) The output content of the multimodal visual model is transmitted to the intention recognition algorithm, which analyzes the intention of the transmitted content. If there is an intrusion intention, the system will alarm the user.

[0031] , express Model results, Indicates the results of keyword search. Represents the result of vector search, , , The value of is 0 or 1. 0 means no intrusion intention, and 1 means there is intrusion intention.

[0032] Intent recognition has three sub-modules including Intrusion intention classification model, keyword retrieval module and vector similarity matching module, The intrusion intention classification model is a classification model based on the trasnformer structure to determine whether the input text has intrusion intention. The keyword retrieval module searches the input text through the preset security intrusion keywords. If there are more than 3 or more keywords, the task has intrusion intention. The intention recognition vector library converts the historical and collected intrusion intention texts into vectors through the model and stores them in the vector database. When verifying the input text, the input text is first converted into a vector, and then the vector is compared with the data stored in the vector library. If it exceeds the threshold, it is considered that there is intrusion intention. The output data of these three modules are voted. If there are two votes to determine that there is intrusion intention, the behavior of destroying and intruding the boundary is returned.

[0033] Among them, in step (2), when the multimodal visual model analyzes the content in the image, it is necessary to perform multimodal visual model training optimization, such as Figure 4 As shown, including: (2.1) Fence image data collection: First, the multimodal visual big model is trained through unsupervised methods. Because the multimodal visual big model is mostly pre-trained through massive unsupervised data, it is mainly aimed at airport fence scenes. Therefore, it is necessary to strengthen the multimodal visual big model's understanding of the fence. By collecting fence images around multiple airports, fence images generated by image enhancement technology, fence images generated by AIGC technology (artificial intelligence content automatic generation technology) and fence-related image data crawled by crawlers.

[0034] By crawling the Internet to search for fence-related image data and fence images collected at the airport and fence image data generated by a public visual large model, these data are trained through unsupervised methods and contrastive learning to let the model learn how to distinguish similar samples from dissimilar samples. By maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs. Through contrastive learning, let the model learn how to distinguish similar samples from dissimilar samples, by maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs. The collected airport fence data crawler crawled data and self-made data are combined with a variety of preset intrusion prompt words. Through the multimodal fusion layer of multimodal multi-model, the features of different modal data are fused into the same dimension and analyzed and processed through LLM.

[0035] (2.2) Dataset generation: annotate the collected boundary-related image data, and judge which behaviors and objects exist in the image. Use this as an annotation to annotate each image in the form of Example 1, and fill the path of the image, the content of the question, and the answer that the large model should answer into the corresponding position in Example 1. Include the question and answer corresponding to the image. Create a corresponding dataset for each image according to the format of Example 1.

[0036] Example 1: {"id": id, "imagePath": "image path", "dialogues": [{"from": "user", "value": " \nQuestion content"}, {"from": "assistant", "value": "Answer content"}]} (2.3) Fine-tune the algorithm and align it. Fine-tune the multimodal visual model using the generated dataset. By adjusting the model hyperparameters, input the labeled data (Example 1) into the multimodal visual model in batches for training. This makes the output of the multimodal visual model more consistent with the content of the perimeter security scenario.

[0037] Intention recognition module 4, this module mainly performs intention discrimination on the output of the multimodal image analysis module to determine whether the output content has the intention of invading the fence. If it is confirmed that people, cars and other objects in the image are invading the fence, a red alarm will be directly sent to the user in the system and a warning sound will be sent in the invading location area. If there is no invading intention in the image but there is a situation of staying or watching, a prompt alarm will be sent in the system, and a warning sound will be sent in the corresponding location area to stay away from the fence.

[0038] The intention determination includes: 4.1. Collection of intent data. The main methods of data collection include generating corpus through prompt words and large language models, manual writing, and natural language data enhancement to generate corresponding text data.

[0039] The intent identification module classifies and judges the text output by the multimodal large model to determine whether the output content contains the intention of invading the boundary. Therefore, when training the intent identification module, it is necessary to collect corresponding training data, generate corpus through manual writing and existing mature LLM, and generate corpus through the prompt word prompt and the large language model as follows: Generate multiple semantic expressions of behaviors that intrude the boundary. Generate multiple corpora through LLM as the data set for intent recognition.

[0040] 4.2. Data labeling. To save labeling time, we first set the prompt word prompt to pre-set intent categories and use the large language model to determine which specific intent the input corpus belongs to. The text information returned by the multimodal large model is input into the trained intent recognition model, and the model will determine whether there is an intrusion. The data set is organized according to the results returned by the large language model. Because the large language model may make mistakes when determining intent, manual correction is required after the data set is generated. The training data corpus is generated through LLM, and it is necessary to manually check whether the corpus is correct and modify the incorrect ones to correct the incorrectly identified intents.

[0041] Example 2: One person uses a stick to hit the boundary 1 again; The first half of this example is the training data input data, 1 is the labeled result, and 1 indicates the presence of intrusion intention and 0 indicates no intrusion intention.

[0042] 4.3, Intent recognition algorithm training, the intent recognition algorithm training can be trained by various open source algorithms such as transfomer, and the data is organized into key and value forms, where the key is a piece of collected corpus and the value is the judgment of whether there is intrusion behavior. Several key and value pairs are organized and sent to training and iterative optimization in batches through transfomer. After the algorithm training is completed, the intent recognition algorithm model trained by transfomer is inherited to the intent recognition module.

[0043] This method uses a multi-stage and multi-method approach to perimeter intrusion detection and recognition, including the front-end using a target detection algorithm to perform an initial screening of the identification of humans approaching the perimeter, and adding a legitimacy analysis module to prevent criminals from calling a multimodal large model to generate illegal content. The multimodal large model is used to analyze the content in the input image, and the generated text content is handed over to the intent recognition module to determine whether there is an intrusion intention. This method enhances the accuracy of intrusion detection and the coverage of multiple intrusion behaviors in multiple forms, which is a significant improvement over the existing method of detecting intrusion behaviors through a single visual solution.

[0044] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered within the protection scope of the present invention.

Claims

1. A multi-modal airport perimeter detection method, It is characterized in that The method includes: S1, using the video personnel detection module to analyze the environment around the enclosure in real time to detect whether anyone is approaching the enclosure; S2, after detecting the information of the person approaching the fence, the image and the pre-set prompt words are transmitted to the legality analysis module to detect whether the prompt words and the content of the image are illegal. If they are illegal, no processing will be done; S3, transmits the image and prompt word content that have passed the legality detection to the multimodal image analysis module. The multimodal visual model uses the input data stage of the open source LLM to add a multimodal feature fusion layer to fuse the image features and prompt word vectors to form a multimodal feature representation; the image content is analyzed and the analysis results are returned through the image understanding, text understanding and analysis capabilities of the multimodal visual model; S4, transmitting the result of the multimodal image analysis module to the intention recognition module, and judging whether the input text content has the intention of destroying or invading the boundary through the intention recognition module; S5, if there is no intention to invade or damage the perimeter, the abnormal alarm module will be used to play a warning to leave through the loudspeakers around the perimeter; if there is a situation of destroying or invading the perimeter, the host computer system will alarm the airport's relevant supervisors and play a voice to stop the destruction and intrusion through the loudspeakers around the perimeter.

2. The airport perimeter detection method based on multimodality according to claim 1, It is characterized in that In step S1, it is detected whether someone is close to the fence, including: when someone is found near the fence, the multimodal image analysis module is triggered, the image data is transmitted to the legality analysis module, and the visual model based on the neural network is used to determine whether the image is legal content, so as to avoid transmitting illegal image data to the multimodal visual large model due to hackers. If it is legal content, the image is transmitted to the multimodal image analysis module.

3. The airport perimeter detection method based on multimodality according to claim 1, It is characterized in that In step S2, when the image and the prompt word pass the verification, the image and the prompt word are transmitted to the multimodal image analysis module.

4. The airport perimeter detection method based on multimodality according to claim 1, It is characterized in that In step S3, the image content is analyzed and the analysis results are returned through the image understanding, text understanding, and analysis capabilities of the multimodal visual large model, including: (1) Combining the received image data with the pre-customized prompt word; (2) The combined data is transmitted to the multimodal visual big model, and the content in the image is analyzed through the understanding and analysis capabilities of the multimodal visual big model; the preset boost words are combined with the input image, and the multimodal visual big model input with the prompt words and image data returns a description of the image; (3) The output content of the multimodal visual model is transmitted to the intention recognition algorithm. The intention recognition algorithm analyzes the intention of the transmitted content. If there is an intrusion intention, the system will alarm the user. The expression is: In the formula, express Model results, Indicates the results of keyword search. Represents the result of vector search, The value is 0 or 1, 0 means no intrusion intention, and 1 means there is intrusion intention.

5. The airport perimeter detection method based on multimodality according to claim 4, It is characterized in that in, In step (2), the content in the image is analyzed through the understanding and analysis capabilities of the multimodal visual large model, including: (2.1) Fence image data collection: train the multimodal vision big model through unsupervised methods to enhance the multimodal vision big model's understanding of the fence, and collect fence images around multiple airports, fence images generated by image enhancement technology, fence images generated by AIGC technology, and fence image data crawled by crawlers; (2.2) Dataset generation: annotating the collected image data of the boundary, including the questions and answers corresponding to the images; (2.3) Fine-tune the algorithm and align it. Fine-tune the multimodal visual model using the generated dataset, adjust the model hyperparameters, and input the labeled data into the multimodal visual model in batches for training. This ensures that the output of the multimodal visual model is consistent with the content of the perimeter security scenario.

6. The airport perimeter detection method based on multimodality according to claim 1, It is characterized in that In step S4, the intention recognition module determines whether the input text content has the intention of destroying or invading the boundary, including: (4.1) Intent data collection: collect the corresponding text data by using prompts and large language models to generate corpus, manually write, and enhance natural language data; generate approximate data by manually writing and setting prompts and using large models to imitate manually written prompts; (4.2) Data annotation: First, set the prompt word prompt to the pre-set intent category to use the large language model to determine which intent the input corpus belongs to, and organize the data set according to the results returned by the large language model; after the data set is generated, perform manual error correction to correct the incorrectly identified intent; (4.3) Intent recognition algorithm training, using transformer architecture and multiple open source algorithms for training, organize the data according to Example 2, and after training, inherit the model to the intent recognition module.

7. A multi-modal airport perimeter detection system, It is characterized in that The system implements the airport perimeter detection method based on multimodality as described in any one of claims 1 to 6, and the system comprises: The video person detection module (1) is used to detect whether a person is close to the fence. When a person is found near the fence, the multimodal image analysis module is triggered to transmit the image data to the legality analysis module to determine whether the image is legal content to avoid transmitting illegal image data to the multimodal visual model due to hacking. If the image is legal content, the image is transmitted to the multimodal image analysis module; The legality analysis module (2) is composed of two deep learning algorithm models, including a trasnformer structure NLP model responsible for judging whether the text prompt word is illegal, and a neural network visual classification model for judging whether the image is illegal. The image and prompt word are transmitted to the legality analysis module for unified recognition and judgment. If there is illegal information, it is directly not processed. It is used to analyze whether the received image and prompt word are legal. The algorithm model is used to detect whether the input image and prompt word contain illegal content such as pornography, violence and terrorism. Illegal content will be directly returned without processing. When the image and prompt word pass the verification, the image and prompt word are transmitted to the multimodal image analysis module. The multimodal image analysis module (3) includes a multimodal visual large model and a deep learning intention recognition algorithm. The multimodal visual large model is responsible for analyzing and describing the objects and behavior content in the image through images and prompt words, and the intention recognition algorithm is responsible for performing intention analysis on the description content output by the multimodal visual large model through pre-set behavior scenarios; The intention recognition module (4) is used to perform intention discrimination on the output of the multimodal image analysis module (3) to determine whether the output content has the intention of invading the boundary. If it is confirmed that the object in the image is invading the boundary, a red alarm is directly sent to the user in the system and a warning sound is sent in the invading location area; if there is no invading intention in the image but there is a stay or wait-and-see situation, a prompt alarm is sent in the system and a warning sound is sent in the corresponding location area to stay away from the boundary.

8. The multi-modal airport perimeter detection system according to claim 7, It is characterized in that The multimodal airport perimeter detection system is mounted on a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, the functions of the multimodal airport perimeter detection system described above are implemented.

9. The multi-modal airport perimeter detection system according to claim 7, It is characterized in that The multimodal-based airport perimeter detection system is mounted on a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the functions of the multimodal-based airport perimeter detection system can be realized.

10. The multi-modal airport perimeter detection system according to claim 7, It is characterized in that The multimodal airport perimeter detection system is mounted on an information data processing terminal, and the information data processing terminal is used to provide a user input interface to implement the functions of the multimodal airport perimeter detection system when executed on an electronic device.

Images