An airport security situation awareness system and method based on deep learning

Abstract

The present invention discloses an airport security situation awareness method and system based on deep learning, which relates to the field of image processing technology. The present invention collects continuous picture sets of airport security passengers and pictures of luggage and articles in the past, constructs a first passenger picture matrix and a picture set of luggage and articles, divides the first passenger picture matrix into a training matrix and a test matrix, and divides the picture set of luggage and articles into a training set and a test set; sets corresponding label matrices and label sets; trains, tests and optimizes the CL network model and the CNN neural network model to obtain the final CL network model and the final CNN neural network model; transmits the picture set of passengers and the pictures of luggage and articles collected in real time to the final CL network model and the final CNN convolutional neural network model respectively to obtain the security situation results and make decisions; brings the features of real-time abnormal passengers into the trained final recurrent neural network prediction model to obtain the security situation of real-time abnormal passengers and make decisions.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing. Specifically, it particularly relates to a method and system for airport security situation based on deep learning. Background Art

[0002] With the rapid development of the global aviation industry, airport security, as the first line of defense to ensure aviation safety, is becoming increasingly important. Currently, airport security systems mainly rely on manual inspections combined with traditional equipment such as X-ray machines and metal detectors. Although these methods can identify prohibited luggage items to a certain extent, they have disadvantages such as low processing efficiency, high false alarm rates, and insufficient recognition capabilities for complex scenarios. Especially when facing increasingly diverse terrorist threat means, traditional security inspection methods seem inadequate.

[0003] In recent years, deep learning technology has made remarkable progress in fields such as image recognition and pattern classification, providing new solutions for airport security situation awareness. However, currently, most of the security inspection systems based on deep learning in the market focus on the recognition of single luggage items, lacking the comprehensive awareness and dynamic evaluation capabilities of the security situation in the entire security inspection area, and there is poor information sharing among systems, making it difficult to form an efficient and collaborative security inspection network. Summary of the Invention

[0004] The present invention proposes a method and system for airport security situation awareness based on deep learning, aiming to achieve comprehensive, accurate, and dynamic awareness of the security situation in the airport security inspection area by integrating multi-source data fusion, deep learning model optimization, and real-time situation analysis technologies. The specific technical solutions include:

[0005] S1. Collect continuous picture sets of past airport security passengers to construct a first passenger picture matrix; collect pictures of past airport security luggage items to construct a luggage item picture set; preprocess the first passenger picture matrix and the luggage item picture set respectively to obtain a preprocessed first passenger picture matrix and a preprocessed luggage item picture set;

[0006] S2. Construct an initial CL network model; train and optimize the initial CL network model through the preprocessed first passenger picture matrix to obtain a final CL network model; construct an initial CNN convolutional neural network model; train and optimize the initial CNN convolutional neural network model through the preprocessed luggage item picture set to obtain a final CNN convolutional neural network model;

[0007] S3. Perform real-time image acquisition within the airport to obtain a real-time passenger image matrix and a real-time luggage and item image set, and transmit them to the final CL network model and the final CNN convolutional neural network model respectively to obtain the real-time security situation result and make a decision; for passengers with an abnormal real-time security situation, collect the features extracted from the final CL network model and construct a real-time abnormal passenger feature matrix;

[0008] S4. Collect past abnormal passenger data and construct an abnormal passenger feature matrix; construct an initial recurrent neural network prediction model, and train and optimize the initial recurrent neural network prediction model based on the abnormal passenger feature matrix to obtain the final recurrent neural network prediction model; substitute the real-time abnormal passenger feature matrix into the final recurrent neural network prediction model to obtain the prediction result of the real-time abnormal passenger and make a decision.

[0009] Preferably, the S1 includes the following steps:

[0010] S11. Collect the continuous image sets of each passenger before and during security inspection in past security accidents, abnormal situations, and normal situations that occurred during airport security inspections, remove the incorrect, duplicate, and irrelevant images in each image set, and construct the first passenger image matrix A as follows:

[0011]

[0012] where A in represents the nth image of the ith passenger in the first passenger image matrix A, N represents a total of N passengers; collect the images of common dangerous luggage and items, non-dangerous luggage and items, and unidentifiable luggage and items during past airport security inspections, and construct a luggage and item image set a = {a1, a2,..., a i ,..., a m}, where a i represents the ith image in the luggage and item image set a, and m represents a total of m images in the luggage and item image set a;

[0013] S12. Preprocess the first passenger image matrix, including the following steps: divide the first passenger image matrix A into a passenger image training matrix B and a passenger image test matrix D, and set label matrices respectively to obtain a passenger image training label matrix C and a passenger image test label matrix E as follows:

[0014]

[0015] where B in represents the nth image of the ith passenger in the passenger image training matrix, O represents a total of O passengers in the passenger image training matrix B, C io represents the oth label of the ith passenger in the passenger image training label matrix; Din denotes the n-th picture of the i-th passenger in the passenger picture test matrix; P represents that there are P passengers in the passenger picture test matrix D, E io denotes the o-th label of the i-th passenger in the passenger picture test label matrix;

[0016] S13. Preprocess the luggage item picture set, including the following steps: Divide the luggage item picture set a = {a1, a2,..., a i ,..., a m} into a luggage item picture training set b = {b1, b2,..., b i ,..., b p} and a luggage item picture test set c = {c1, c2,..., c i ,..., c q}; respectively set label sets to obtain a luggage item picture training label set d = {d1, d2,..., d i ,..., d p} and a luggage item picture test label set e = {e1, e2,..., e i ,..., e q}; where b i represents the i-th picture in the luggage item picture training set, p represents that there are p pictures in the luggage item picture training set; a i represents the i-th picture in the luggage item picture test set, q represents that there are q pictures in the luggage item picture training set; d i represents the i-th label in the luggage item picture training label set, e i represents the i-th label in the luggage item picture test label set;

[0017] By collecting past airport passenger pictures and luggage item pictures, a first passenger picture matrix and a luggage item picture set are constructed. The first passenger picture matrix is divided into a training matrix and a test matrix, and the luggage item picture set is divided into a training set and a test set, and the corresponding label matrix and label set are set, providing a data basis for subsequent model training and testing, and ensuring the training quality and generalization ability of the model.

[0018] Preferably, the S2 includes the following steps:

[0019] S21. The CNN convolutional neural network is used for image recognition, and the LSTM long short-term memory network processes time series data. The CNN convolutional neural network and the LSTM long short-term memory network are serially fused to construct an initial CL network model;

[0020] S22. Train the initial CL network model using the passenger picture training matrix B and the passenger picture training label matrix C; after the training is completed, obtain the trained CL network model, then input the passenger picture test matrix D and the passenger picture test label matrix E into the trained CL network model for testing, and optimize the trained CL network model according to the test results to obtain the final CL network model;

[0021] S23. Construct an initial CNN convolutional neural network model, and use the luggage item picture training set, the luggage item picture test set, the luggage item picture training label set, and the luggage item picture test label set to train and optimize the initial CNN convolutional neural network model to obtain the final CNN convolutional neural network model;

[0022] Construct a CL convolutional recurrent neural network model by combining the CNN convolutional neural network and the LSTM long short-term memory network to better process continuous picture features and changes; by adjusting the model parameters, enable the model to learn from the training data, test the model to evaluate the generalization ability of the model, and ensure that the model can perform well in actual applications; optimizing the model is to adjust the model according to the test results to improve its performance.

[0023] Preferably, the S22 includes the following steps:

[0024] S221. Set the number of iterations for training the initial CL network model to g, the batch size to u, the optimizer to the adam optimizer, and the initial learning rate to α;

[0025] S222. Set the current training iteration number to z and the training error threshold to l; input the passenger picture training matrix B and the passenger picture training label matrix C into the initial CL network model for training according to the batch size u, the adam optimizer, and the initial learning rate α. When the training error of the initial CL network model < l or when z ≥ g, stop training to obtain the trained CL network model;

[0026] S223. Set the accuracy threshold w, input the passenger picture test matrix D and the passenger picture test label matrix E into the trained CL network model for testing to obtain the test accuracy k; when k ≥ w, use the trained CL network model as the final CL network model; when k < w, optimize the initial learning rate α of the trained CL network model to obtain the optimized CL network model, and use the optimized CL network model as the final CL network model;

[0027] By setting the relevant parameters of the CL network model, and then inputting the passenger picture training matrix into the CL network model for training, a trained CL network model is obtained. The passenger picture test matrix is used to test the trained CL network model. According to the test results, the trained CL network model is further optimized to make the CL network model more accurate.

[0028] Preferably, the training error of the initial CL network model in S222 is calculated using a loss function, and the function expression is as follows:

[0029]

[0030] where x is the number of pictures in the passenger picture training matrix; y i is the true label of the i-th picture, taking values 0 or 1; is the predicted probability of the i-th sample, representing the probability that the model predicts this sample belongs to class 1;

[0031] By calculating the training error of the initial CL network model using the loss function, the obtained result is used as an important indicator in the model training process to more accurately evaluate the performance of the model and guide the optimization and adjustment of the model.

[0032] Preferably, optimizing the initial learning rate α of the trained CL network model in S223 to obtain an optimized CL network model includes the following steps:

[0033] S2231. Randomly initialize the positions of the particle swarm h = {h1, h2,..., h i ,..., h Np}, where h i represents the initial position of the i-th particle, and Np is the number of particles; initialize the velocities of the particles v = {v1, v2,..., v i ,..., v Np}, where v i represents the i-th velocity; initialize the best positions of each particle f = {f1, f2,..., f i ,..., f Np}, where f i represents the best position reached by the i-th particle; initialize the global best position l as any position in f; set the maximum number of iterations as f′, and the performance of the global best position reaches the target value f″;

[0034] S2232. For each particle, update the velocity where t represents the current moment, t + 1 represents the next moment, j is the inertia weight, c1 and c2 are learning factors, r1 and r2 are random numbers in the range of 0 to 1, and f it is the best position reached by the i-th particle at time t, l t is the global best position reached by the i-th particle at time t;

[0035] Update the position The updated position is used as the new learning rate to obtain the updated learning rate. Use the updated learning rate to train the trained CL network model and evaluate the performance. If the updated position has better performance than f i t , then update f i . If the of the new position does not have better performance than f i t , keep the original f i ; At time t + 1, traverse the historical best positions f i of all particles and select the position with the best performance as the new global best position l t+1 ;

[0036] S2233. Repeat S2232. When the performance of the global best position reaches the target value f″ or the number of iterations reaches f′, stop the iteration process and output the learning rate value corresponding to the global best position to obtain the optimal learning rate;

[0037] S2234. Use the optimal learning rate as the learning rate of the trained CL network model to obtain an optimized CL network model;

[0038] By using the particle swarm optimization algorithm to optimize the learning rate parameter in the trained CL network model, and using the performance of the CL network model as the optimization index during optimization, the performance of the CL network model can be continuously improved, making the CL network model more accurate in recognizing passenger features in images.

[0039] Preferably, the S3 includes the following steps:

[0040] S31. Obtain a real-time continuous picture set of each passenger in the airport through video monitoring; construct a real-time passenger picture matrix F as follows,

[0041]

[0042] where F iG refers to the G-th picture of the i-th passenger in the real-time passenger picture matrix F, and L represents the total number of passengers in the current airport;

[0043] Substitute the real-time passenger picture matrix F into the final CL network model to obtain the security situation of each passenger in the current airport;

[0044] S32. Obtain pictures of luggage items through video monitoring and sensor monitoring to obtain a real-time luggage item picture set where K i refers to the i-th picture in the real-time luggage item picture set, and n p represents the total number of pictures in the real-time luggage item picture set;

[0045] Substitute the real-time luggage item picture set K into the final CNN convolutional neural network model to obtain the security situation of each luggage item;

[0046] S33. Conduct normal security checks on passengers and luggage items with a real-time security situation of safe; immediately trigger an emergency response mechanism for passengers and luggage items with a real-time security situation of unsafe; conduct focused inspections on luggage items with a real-time security situation of abnormal; for passengers with a real-time security situation of abnormal, collect the features extracted from the final CL network model to obtain a real-time abnormal passenger feature matrix Y, as follows,

[0047]

[0048] where Y is refers to the S-th feature of the i-th passenger in the real-time abnormal passenger feature matrix Y, and R represents the total number of real-time abnormal passengers;

[0049] By transmitting the real-time collected passenger picture set and luggage item pictures to the final CL network model and the final CNN convolutional neural network model respectively, obtaining the security situation results and making decisions, the real-time monitoring and early warning functions are realized, and the security inspection efficiency and safety are improved.

[0050] Preferably, S4 includes the following steps:

[0051] S41. Collect past abnormal passenger data, extract features to construct an abnormal passenger feature matrix M, as follows,

[0052]

[0053] where M iT refers to the T-th feature of the i-th passenger in the abnormal passenger feature matrix M, and U represents the total number of abnormal passengers in the abnormal passenger feature matrix;

[0054] Divide the abnormal passenger feature matrix M into an abnormal passenger feature training matrix X and an abnormal passenger feature test matrix Z, as follows,

[0055]

[0056] where X iT refers to the T-th feature of the i-th passenger in the abnormal passenger feature training matrix X, and V represents the total number of abnormal passengers in the abnormal passenger feature test matrix; ZiT It refers to the T-th feature of the i-th passenger in the abnormal passenger feature test matrix Z, and Q represents the total number of abnormal passengers in the abnormal passenger feature test matrix;

[0057] Set a label for the final security situation of each abnormal passenger to obtain the abnormal passenger feature training label set and the abnormal passenger feature test label set. The abnormal passenger feature training label set q = {q1, q2,..., q i ,..., q V} and the abnormal passenger feature test label set r = {r1, r2,..., r i ,..., r Q}, where q i represents the i-th label in the abnormal passenger feature training label set; r i represents the i-th label in the abnormal passenger feature test label set;

[0058] S42. Construct an initial recurrent neural network prediction model, and use the abnormal passenger feature training matrix X, the abnormal passenger feature test matrix Z, and their corresponding abnormal passenger feature training label set and abnormal passenger feature test label set to train and optimize the initial recurrent neural network prediction model to obtain the final recurrent neural network prediction model;

[0059] S43. Substitute the real-time abnormal passenger feature matrix Y into the final recurrent neural network prediction model to obtain the prediction result of the real-time abnormal passenger security situation. For passengers with an unsafe prediction result, immediately trigger the emergency response mechanism; for passengers with a safe prediction result, still focus on investigation to avoid machine omissions;

[0060] By collecting the characteristics of past abnormal passengers, construct an abnormal passenger feature matrix, and divide the abnormal passenger feature matrix into a training matrix and a test matrix; construct an initial recurrent neural network prediction model, and use the abnormal passenger feature training matrix, the abnormal passenger feature test matrix, and their corresponding abnormal passenger feature training label set and abnormal passenger feature test label set to train and optimize the initial recurrent neural network prediction model to obtain the final recurrent neural network prediction model; bring the characteristics of real-time abnormal passengers into the trained final recurrent neural network prediction model to obtain the security situation of real-time abnormal passengers, and make decisions to achieve the prediction function, which can identify the trend of abnormal behavior development in advance, so as to take preventive measures and reduce the occurrence of security incidents;

[0061] An airport security inspection security situation awareness system based on deep learning, including a data collection and preprocessing module, a deep learning model training and optimization module, a deep learning model application module, and a security situation prediction module;

[0062] The data collection and preprocessing module is used to collect consecutive picture sets of passengers and pictures of luggage and items during past airport security checks, construct the first passenger picture matrix and the picture set of luggage and items, divide the first passenger picture matrix into a training matrix and a test matrix, and divide the picture set of luggage and items into a training set and a test set; set the corresponding label matrix and label set;

[0063] The deep learning model training and optimization module is used to train, test and optimize the CL network model and the CNN neural network model to obtain the final CL network model and the final CNN neural network model;

[0064] The deep learning model application module transmits the real-time collected picture sets of passengers and pictures of luggage and items to the final CL network model and the final CNN convolutional neural network model respectively to obtain the security situation results and make decisions;

[0065] The security situation prediction module is used to bring the features of real-time abnormal passengers into the trained final recurrent neural network prediction model to obtain the security situation of real-time abnormal passengers and make decisions.

[0066] The present invention has the following beneficial effects:

[0067] By collecting the pictures of passengers and pictures of luggage and items on past public airports and airplanes, constructing the first passenger picture matrix and the picture set of luggage and items, dividing the first passenger picture matrix into a training matrix and a test matrix, and dividing the picture set of luggage and items into a training set and a test set, and setting the corresponding label matrix and label set, it provides a data basis for subsequent model training and testing, and ensures the training quality and generalization ability of the model.

[0068] By training, testing and optimizing the CL network model and the CNN neural network model to obtain the final CL network model and the final CNN neural network model, the recognition accuracy and efficiency of prohibited items and abnormal behaviors are improved, and the performance of the model in practical applications is enhanced.

[0069] By transmitting the real-time collected picture sets of passengers and pictures of luggage and items to the final CL network model and the final CNN convolutional neural network model respectively to obtain the security situation results and make decisions, the functions of real-time monitoring and early warning are realized, the security inspection efficiency and safety are improved, and the occurrence of security incidents is reduced.

[0070] By bringing the features of real-time abnormal passengers into the trained final recurrent neural network prediction model to obtain the security situation of real-time abnormal passengers and make decisions; it can further identify abnormal behaviors in advance, so as to take preventive measures, reduce the occurrence of security incidents, and improve the initiative and preventive nature of security management.

[0071] Of course, it is not necessary for any product implementing the present invention to simultaneously achieve all the above-mentioned advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the invention, the following will briefly introduce the drawings required for describing the embodiments. Obviously, the drawings in the following description are only some embodiments of the invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0073] Figure 1 It is a flowchart for identifying the real-time security situation of luggage and items in a method for airport security situation awareness based on deep learning provided by the present invention;

[0074] Figure 2 It is a flowchart for identifying the real-time security situation of passengers in a method for airport security situation awareness based on deep learning provided by the present invention;

[0075] Figure 3 It is a flowchart for identifying the real-time security situation of abnormal passengers in a method for airport security situation awareness based on deep learning provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0076] The following will clearly and completely describe the technical solutions in the embodiments of the invention with reference to the drawings in the embodiments of the invention. Obviously, the described embodiments are only some embodiments of the invention, rather than all embodiments. Based on the embodiments of the invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the invention.

[0077] In the description of the present invention, it should be understood that the terms "opening", "upper", "lower", "top", "middle", "inner", etc. indicating orientation or positional relationships are only for the convenience of describing the invention and simplifying the description, rather than indicating or implying that the components or elements referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention.

[0078] Embodiment 1:

[0079] Please refer to Figures 1-3 , the present invention discloses a method for airport security situation awareness based on deep learning, including the following steps:

[0080] S1. Collect a continuous set of pictures of past airport security passengers to construct the first passenger picture matrix; collect pictures of past airport security luggage items to construct a luggage item picture set; preprocess the first passenger picture matrix and the luggage item picture set respectively to obtain the preprocessed first passenger picture matrix and the preprocessed luggage item picture set;

[0081] The S1 includes the following steps:

[0082] S11. Collect a continuous set of pictures of each passenger before and during security check in the past airport security accidents, abnormal situations and normal situations, remove the wrong, repeated and irrelevant pictures in each picture set, and construct the first passenger picture matrix A as follows,

[0083]

[0084] where A in represents the nth picture of the ith passenger in the first passenger picture matrix A, and N represents that there are N passengers in total;

[0085] Collect pictures of common dangerous luggage items, non-dangerous luggage items and unidentifiable luggage items during past airport security checks to construct a luggage item picture set a = {a1, a2,..., a i ,..., a m}, where a i represents the ith picture in the luggage item picture set a, and m represents that there are m pictures in the luggage item picture set a;

[0086] S12. Preprocess the first passenger picture matrix, including the following steps: divide the first passenger picture matrix A into a passenger picture training matrix B and a passenger picture test matrix D, and set label matrices respectively to obtain a passenger picture training label matrix C and a passenger picture test label matrix E as follows,

[0087]

[0088] where B in represents the nth picture of the ith passenger in the passenger picture training matrix, O represents that there are O passengers in the passenger picture training matrix B, and C io represents the oth label of the ith passenger in the passenger picture training label matrix; D in represents the nth picture of the ith passenger in the passenger picture test matrix; P represents that there are P passengers in the passenger picture test matrix D, and E io represents the oth label of the ith passenger in the passenger picture test label matrix;

[0089] S13. Preprocess the luggage item picture set, including the following steps: Divide the luggage item picture set a = {a1, a2,..., a i ,..., a m} into a luggage item picture training set b = {b1, b2,..., b i ,..., b p} and a luggage item picture test set c = {c1, c2,..., c i ,..., c q}; Set the label sets respectively to obtain a luggage item picture training label set d = {d1, d2,..., d i ,..., d p} and a luggage item picture test label set e = {e1, e2,..., e i ,..., e q}; where b i represents the i-th picture in the luggage item picture training set, p represents the total number of pictures in the luggage item picture training set; a i represents the i-th picture in the luggage item picture test set, q represents the total number of pictures in the luggage item picture training set; d i represents the i-th label in the luggage item picture training label set, e i represents the i-th label in the luggage item picture test label set;

[0090] S2. Construct an initial CL network model; Train and optimize the initial CL network model with the preprocessed first passenger picture matrix to obtain the final CL network model; Construct an initial CNN convolutional neural network model; Train and optimize the initial CNN convolutional neural network model with the preprocessed luggage item picture set to obtain the final CNN convolutional neural network model;

[0091] The S2 includes the following steps:

[0092] S21. The CNN convolutional neural network is used for image recognition, and the LSTM long short-term memory network processes time series data. Serialize and fuse the CNN convolutional neural network and the LSTM long short-term memory network to construct an initial CL network model;

[0093] S22. Use the passenger picture training matrix B and the passenger picture training label matrix C to train the initial CL network model; After training, obtain the trained CL network model, and then input the passenger picture test matrix D and the passenger picture test label matrix E into the trained CL network model for testing, and optimize the trained CL network model according to the test results to obtain the final CL network model;

[0094] The S22 includes the following steps:

[0095] S221. Construct an initial CL network model, and set the number of iterations for training the initial CL network model to g, the batch size to u, the optimizer to the adam optimizer, and the initial learning rate to α;

[0096] S222. Set the current training iteration number to z and the training error threshold to l; input the passenger picture training matrix B and the passenger picture training label matrix C into the initial CL network model for training according to the batch size u, the adam optimizer, and the initial learning rate α. When the training error of the initial CL network model < l or when z ≥ g, stop training to obtain the trained CL network model;

[0097] The training error of the initial CL network model in S222 is calculated using a loss function, and the function expression is as follows:

[0098]

[0099] where x is the number of pictures in the passenger picture training matrix; y i is the true label of the i-th picture, taking values 0 or 1; is the predicted probability of the i-th sample, indicating the probability that the model predicts this sample belongs to class 1;

[0100] S223. Set the accuracy threshold w, input the passenger picture test matrix D and the passenger picture test label matrix E into the trained CL network model for testing to obtain the test accuracy k; when k ≥ w, take the trained CL network model as the final CL network model; when k < w, optimize the initial learning rate α of the trained CL network model to obtain an optimized CL network model, and take the optimized CL network model as the final CL network model;

[0101] Optimizing the initial learning rate α of the trained CL network model in S223 to obtain an optimized CL network model includes the following steps:

[0102] S2231. Randomly initialize the positions of the particle swarm h = {h1, h2,..., h i ,..., h Np}, where h i represents the initial position of the i-th particle, and Np is the number of particles; initialize the velocities of the particles v = {v1, v2,..., v i ,..., v Np}, where v i represents the i-th velocity; initialize the best position of each particle f = {f1, f2,..., f i ,..., f Np}, where f i represents the best position reached by the i-th particle; initialize the global best position l to any position in f; set the maximum number of iterations to f′, and the performance of the global best position reaches the target value f″;

[0103] S2232. For each particle, update the velocity where t represents the current moment, t + 1 represents the next moment, j is the inertia weight, c1 and c2 are learning factors, r1 and r2 are random numbers in the range from 0 to 1, and f i t is the best position reached by the i-th particle at moment t, and l t is the global best position reached by the i-th particle at moment t;

[0104] Update the position The updated position is used as the new learning rate to obtain the updated learning rate. Use the updated learning rate to train the trained CL network model and evaluate the performance. If the performance of the updated position is better than f i t , then update f i , if the performance of the new position is not better than f i t , keep the original f i ; at moment t + 1, traverse the historical best positions f i of all particles, and select the position with the best performance as the new global best position l t+1 ;

[0105] S2233. Repeat S2232. When the performance of the global best position reaches the target value f″ or the number of iterations reaches f′, stop the iteration process, output the learning rate value corresponding to the global best position, and obtain the optimal learning rate;

[0106] S2234. Use the optimal learning rate as the learning rate of the trained CL network model to obtain an optimized CL network model;

[0107] S23. Construct an initial CNN convolutional neural network model, and use the luggage item picture training set, luggage item picture test set, luggage item picture training label set, and luggage item picture test label set to train and optimize the initial CNN convolutional neural network model to obtain a final CNN convolutional neural network model;

[0108] S3. Conduct real-time image acquisition within the airport to obtain a real-time passenger image matrix and a real-time luggage and item image set, and transmit them to the final CL network model and the final CNN convolutional neural network model respectively to obtain the real-time security situation results and make decisions; for passengers with an abnormal real-time security situation, collect the features extracted from the final CL network model to construct a real-time abnormal passenger feature matrix;

[0109] S3 includes the following steps:

[0110] S31. Obtain a real-time continuous image set of each passenger within the airport through video surveillance; construct a real-time passenger image matrix F as follows,

[0111]

[0112] where F iG refers to the Gth image of the ith passenger in the real-time passenger image matrix F, and L represents the total number of passengers within the current airport;

[0113] Substitute the real-time passenger image matrix F into the final CL network model to obtain the security situation of each passenger within the current airport;

[0114] S32. Obtain images of luggage and items through video surveillance and sensor monitoring to obtain a real-time luggage and item image set where K i refers to the ith image in the real-time luggage and item image set, and n p represents the total number of images in the real-time luggage and item image set;

[0115] Substitute the real-time luggage and item image set K into the final CNN convolutional neural network model to obtain the security situation of each luggage and item;

[0116] S33. Conduct normal security checks on passengers and luggage and items with a safe real-time security situation; immediately trigger the emergency response mechanism for passengers and luggage and items with an unsafe real-time security situation; conduct focused inspections on luggage and items with an abnormal real-time security situation; for passengers with an abnormal real-time security situation, collect the features extracted from the final CL network model to obtain a real-time abnormal passenger feature matrix Y as follows,

[0117]

[0118] where Y is refers to the Sth feature of the ith passenger in the real-time abnormal passenger feature matrix Y, and R represents the total number of real-time abnormal passengers;

[0119] S4. Collect historical data of abnormal passengers to construct a feature matrix of abnormal passengers; construct an initial recurrent neural network prediction model, and optimize the initial recurrent neural network prediction model based on the feature matrix of abnormal passengers to obtain a final recurrent neural network prediction model; substitute the real-time feature matrix of abnormal passengers into the final recurrent neural network prediction model to obtain the prediction result of real-time abnormal passengers and make a decision.

[0120] S4 includes the following steps:

[0121] S41. Collect historical data of abnormal passengers, extract features to construct a feature matrix M of abnormal passengers as follows:

[0122]

[0123] where M iT represents the T-th feature of the i-th passenger in the feature matrix M of abnormal passengers, and U represents the total number of abnormal passengers in the feature matrix of abnormal passengers;

[0124] Divide the feature matrix M of abnormal passengers into a training matrix X of abnormal passenger features and a test matrix Z of abnormal passenger features as follows:

[0125]

[0126] where X iT represents the T-th feature of the i-th passenger in the training matrix X of abnormal passenger features, and V represents the total number of abnormal passengers in the training matrix of abnormal passenger features; Z iT represents the T-th feature of the i-th passenger in the test matrix Z of abnormal passenger features, and Q represents the total number of abnormal passengers in the test matrix of abnormal passenger features;

[0127] Set a label for the final safety situation of each abnormal passenger as safe and unsafe to obtain a training label set q = {q1, q2,..., q i ,..., q V} and a test label set r = {r1, r2,..., r i ,..., r Q}, where q i represents the i-th label in the training label set of abnormal passenger features; r i represents the i-th label in the test label set of abnormal passenger features;

[0128] S42. Construct an initial recurrent neural network prediction model, and use the training matrix X of abnormal passenger features, the test matrix Z of abnormal passenger features, and their corresponding training label sets and test label sets of abnormal passenger features to train and optimize the initial recurrent neural network prediction model to obtain a final recurrent neural network prediction model;

[0129] S43. Substitute the real-time abnormal passenger feature matrix Y into the final recurrent neural network prediction model to obtain the prediction result of the safety situation of real-time abnormal passengers. For passengers with an unsafe prediction result, immediately trigger the emergency response mechanism; for passengers with a safe prediction result, still focus on investigation to avoid machine errors and omissions.

[0130] Embodiment 2:

[0131] The present invention also discloses an airport security safety situation awareness system based on deep learning, including a data collection and preprocessing module, a deep learning model training and optimization module, a deep learning model application module, and a safety situation prediction module;

[0132] The data collection and preprocessing module is used to collect continuous picture sets of passengers and pictures of luggage and items during past airport security checks, construct the first passenger picture matrix and the luggage and item picture set, divide the first passenger picture matrix into a training matrix and a test matrix, and divide the luggage and item picture set into a training set and a test set; set the corresponding label matrix and label set;

[0133] The deep learning model training and optimization module is used to train, test and optimize the CL network model and the CNN neural network model to obtain the final CL network model and the final CNN neural network model;

[0134] The deep learning model application module transmits the picture set of passengers and the pictures of luggage and items collected in real time to the final CL network model and the final CNN convolutional neural network model respectively to obtain the safety situation result and make a decision;

[0135] The safety situation prediction module is used to bring the features of real-time abnormal passengers into the trained final recurrent neural network prediction model to obtain the safety situation of real-time abnormal passengers and make a decision.

[0136] In the description of this specification, the descriptions referring to terms such as "one embodiment", "example", "specific example", etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0137] The preferred embodiments of the invention disclosed above are only used to help illustrate the invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of this specification. These embodiments are selected and specifically described in this specification in order to better explain the principle and practical application of the invention, so that those skilled in the art can well understand and utilize the invention.

Claims

1. A method for airport security situation awareness based on deep learning, characterized in that, It includes the following steps: S1. Collect a continuous picture set of past airport security passengers, and construct a first passenger picture matrix; Collect pictures of past airport security luggage items, and construct a luggage item picture set; preprocess the first passenger picture matrix and the luggage item picture set respectively to obtain a preprocessed first passenger picture matrix and a preprocessed luggage item picture set; Among them, preprocessing the first passenger picture matrix and the luggage item picture set respectively includes the following steps: Divide the first passenger picture matrix into a passenger picture training matrix and a passenger picture test matrix, and set label matrices respectively to obtain a passenger picture training label matrix and a passenger picture test label matrix; Divide the luggage item picture set into a luggage item picture training set and a luggage item picture test set, and set label sets respectively to obtain a luggage item picture training label set and a luggage item picture test label set; S2. Construct an initial CL network model; train and optimize the initial CL network model with the preprocessed first passenger picture matrix to obtain a final CL network model; construct an initial CNN convolutional neural network model; train and optimize the initial CNN convolutional neural network model with the preprocessed luggage item picture set to obtain a final CNN convolutional neural network model; The S2 includes the following steps: S21. Serially fuse a CNN convolutional neural network and an LSTM long short-term memory network to construct an initial CL network model; S22. Use the passenger picture training matrix and the passenger picture training label matrix to train the initial CL network model; after training, obtain a trained CL network model, then input the passenger picture test matrix and the passenger picture test label matrix into the trained CL network model for testing, and optimize the trained CL network model according to the test results to obtain a final CL network model; S23. Construct an initial CNN convolutional neural network model, and use the luggage item picture training set, the luggage item picture test set, the luggage item picture training label set, and the luggage item picture test label set to train and optimize the initial CNN convolutional neural network model to obtain a final CNN convolutional neural network model; S3. Conduct real-time picture collection in the airport to obtain a real-time passenger picture matrix and a real-time luggage item picture set, and transmit them to the final CL network model and the final CNN convolutional neural network model respectively to obtain a real-time security situation result and make a decision; for passengers with an abnormal real-time security situation, collect the features extracted from the final CL network model to construct a real-time abnormal passenger feature matrix; S4. Collect past abnormal passenger data to construct an abnormal passenger feature matrix; construct an initial recurrent neural network prediction model, and train and optimize the initial recurrent neural network prediction model based on the abnormal passenger feature matrix to obtain a final recurrent neural network prediction model; substitute the real-time abnormal passenger feature matrix into the final recurrent neural network prediction model to obtain a prediction result for the real-time abnormal passenger and make a decision.

2. The method for airport security situation awareness based on deep learning according to claim 1, characterized in that The S1 also includes the following steps: Collect a continuous set of pictures of each passenger before and during airport security checks for past security incidents, abnormal situations, and normal situations that occurred during airport security checks. Remove incorrect, duplicate, and irrelevant pictures from each set of pictures to construct a first passenger picture matrix; collect pictures of common dangerous luggage items, non-dangerous luggage items, and unidentifiable luggage items that occurred during past airport security checks to construct a luggage item picture set.

3. The method for airport security situation awareness based on deep learning according to claim 2, wherein, The S22 includes the following steps: S221. Set the number of iterations for training the initial CL network model to g, the batch size to u, the optimizer to the adam optimizer, and the initial learning rate to α; S222. Set the current training iteration number to z and the training error threshold to l; input the passenger picture training matrix and the passenger picture training label matrix into the initial CL network model for training according to the batch size u, the adam optimizer, and the initial learning rate α. When the training error of the initial CL network model < l or when z ≥ g, stop training to obtain the trained CL network model; S223. Set the accuracy threshold w, input the passenger picture test matrix D and the passenger picture test label matrix E into the trained CL network model for testing to obtain the test accuracy k; when k ≥ w, use the trained CL network model as the final CL network model; when k < w, optimize the initial learning rate α of the trained CL network model to obtain an optimized CL network model, and use the optimized CL network model as the final CL network model.

4. The method for airport security situation awareness based on deep learning according to claim 3, characterized in that The training error of the initial CL network model in S222 is calculated using a loss function, and the function expression is as follows: where x is the number of pictures in the passenger picture training matrix; y i is the true label of the i-th picture, taking values 0 or 1; is the predicted probability of the i-th sample, representing the probability that the model predicts this sample belongs to class 1.

5. The method for airport security situation awareness based on deep learning according to claim 4, characterized in that The optimization of the initial learning rate α of the trained CL network model in S223 to obtain an optimized CL network model includes the following steps: S2231. Randomly initialize the positions of the particle swarm \(h = \{h_1, h_2, \ldots, h i , \ldots, h Np}\), where \(h i \) represents the initial position of the \(i\)-th particle, and \(N_p\) is the number of particles; initialize the velocities of the particles \(v = \{v_1, v_2, \ldots, v i , \ldots, v Np}\), where \(v i \) represents the \(i\)-th velocity; initialize the best positions of each particle \(f = \{f_1, f_2, \ldots, f i , \ldots, f Np}\), where \(f i \) represents the best position reached by the \(i\)-th particle; initialize the global best position \(l\) as any position in \(f\); set the maximum number of iterations as \(f'\), and the performance of the global best position reaches the target value \(f''\); S2232. For each particle, update the velocity where t represents the current moment, t + 1 represents the next moment, j is the inertia weight, c1 and c2 are learning factors, r1 and r2 are random numbers in the range of 0 to 1, f i t is the best position reached by the i-th particle at moment t, and l t is the global best position reached by the i-th particle at moment t; Updated Position The updated position is used as the new learning rate to obtain the updated learning rate. The trained CL network model is trained using the updated learning rate, and the performance is evaluated. If the updated position has better performance than f i t , then f is updated i . If the performance of the new position is not better than f i t , the original f is maintained i ; At time t+1, traverse the historical best positions f i of all particles, and select the position with the best performance as the new global best position l t+1 ; S2233. Repeat S2232. When the performance at the global best position reaches the target value f″ or the number of iterations reaches f′, stop the iteration process, output the learning rate value corresponding to the global best position to obtain the optimal learning rate; S2234. Use the optimal learning rate as the learning rate of the trained CL network model to obtain an optimized CL network model.

6. The method for airport security situation awareness based on deep learning according to claim 5, wherein The S3 includes the following steps: S31. Obtain a real-time continuous set of pictures of each passenger in the airport through video monitoring; construct a real-time passenger picture matrix, and substitute the real-time passenger picture matrix into the final CL network model to obtain the security situation of each passenger in the current airport; S32. Obtain pictures of luggage items through video monitoring and sensor monitoring to obtain a real-time luggage item picture set; substitute the real-time luggage item picture set into the final CNN convolutional neural network model to obtain the security situation of each luggage item. S33. Conduct normal security checks on passengers and luggage with a real-time security situation of safe; immediately trigger the emergency response mechanism for passengers and luggage with a real-time security situation of unsafe; conduct focused inspections on luggage with a real-time security situation of abnormal; for passengers with a real-time security situation of abnormal, collect the features extracted from the final CL network model to obtain the real-time abnormal passenger feature matrix.

7. The method for airport security situation awareness based on deep learning according to claim 6, wherein S4 includes the following steps: S41. Collect past abnormal passenger data, extract features to construct an abnormal passenger feature matrix, and divide the abnormal passenger feature matrix into an abnormal passenger feature training matrix and an abnormal passenger feature test matrix; Set the abnormal passenger feature training label set and the abnormal passenger feature test label set; S42. Construct an initial recurrent neural network prediction model, and use the abnormal passenger feature training matrix, the abnormal passenger feature test matrix and their corresponding abnormal passenger feature training label set and abnormal passenger feature test label set to train and optimize the initial recurrent neural network prediction model to obtain the final recurrent neural network prediction model; S43. Substitute the real-time abnormal passenger feature matrix into the final recurrent neural network prediction model to obtain the prediction result of the real-time abnormal passenger security situation. For passengers with a prediction result of unsafe, immediately trigger the emergency response mechanism; for passengers with a prediction result of safe, still conduct focused inspections.

8. A system for the method of airport security situation awareness based on deep learning according to any one of claims 1-7, characterized in that, It includes a data collection and preprocessing module, a deep learning model training and optimization module, a deep learning model application module, and a security situation prediction module.

Images