Airport security check security situation awareness system and method based on deep learning

Abstract

The invention discloses an airport security check security situation awareness method and system based on deep learning, and relates to the technical field of image processing.The airport security check security situation awareness method comprises the steps that a first passenger picture matrix and a luggage picture set are constructed by collecting a continuous picture set and luggage pictures of passing airport security check passengers; dividing the first passenger picture matrix into a training matrix and a test matrix, and dividing the luggage picture set into a training set and a test set; setting a corresponding label matrix and a label set; training, testing and optimizing the CL network model and the CNN neural network model to obtain a final CL network model and a final CNN neural network model; respectively transmitting a passenger picture set and a luggage picture which are collected in real time to the final CL network model and the final CNN convolutional neural network model to obtain a security situation result and make a decision; and substituting the characteristics of the real-time abnormal passenger into the trained final recurrent neural network prediction model to obtain the safety situation of the real-time abnormal passenger, and making a decision.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and more specifically, relates to a method and system for airport security inspection based on deep learning. Background Art

[0002] With the rapid development of the global aviation industry, airport security, as the first line of defense to ensure aviation safety, is becoming increasingly important. At present, airport security systems mainly rely on manual inspection combined with traditional equipment such as X-ray machines and metal detectors. Although these methods can identify prohibited luggage items to a certain extent, they have disadvantages such as low processing efficiency, high false alarm rate, and insufficient recognition of complex scenes. In particular, in the face of increasingly diverse terrorist threats, traditional security methods seem to be unable to cope with the situation.

[0003] In recent years, deep learning technology has made significant progress in image recognition, pattern classification and other fields, providing a new solution for airport security situation awareness. However, the current deep learning-based security inspection systems on the market focus on the identification of a single piece of luggage, lacking the ability to fully perceive and dynamically evaluate the security situation of the entire security inspection area, and poor information sharing between systems, making it difficult to form an efficient and coordinated security inspection network. Summary of the invention

[0004] This invention proposes a deep learning-based airport security situation awareness system and method, which aims to achieve comprehensive, accurate and dynamic awareness of the security situation in the airport security area by integrating multi-source data fusion, deep learning model optimization and real-time situation analysis technology. The specific technical solutions include:

[0005] S1, collect continuous picture sets of passengers who have passed airport security checks in the past to construct a first passenger picture matrix; collect pictures of luggage items that have passed airport security checks in the past to construct a baggage item picture set; pre-process the first passenger picture matrix and the baggage item picture set respectively to obtain a pre-processed first passenger picture matrix and a pre-processed baggage item picture set;

[0006] S2, constructing an initial CL network model; training and optimizing the initial CL network model through the preprocessed first passenger image matrix to obtain a final CL network model; constructing an initial CNN convolutional neural network model; training and optimizing the initial CNN convolutional neural network model through the preprocessed luggage item image set to obtain a final CNN convolutional neural network model;

[0007] S3. Collect real-time pictures in the airport, obtain the real-time passenger picture matrix and the real-time baggage picture set, and transmit them to the final CL network model and the final CNN convolutional neural network model respectively, obtain the real-time security situation results and make decisions; for passengers with abnormal real-time security situation, collect the features extracted from the final CL network model and construct the real-time abnormal passenger feature matrix;

[0008] S4. Collect past abnormal passenger data and construct an abnormal passenger feature matrix; construct an initial recursive neural network prediction model, train and optimize the initial recursive neural network prediction model based on the abnormal passenger feature matrix to obtain the final recursive neural network prediction model; substitute the real-time abnormal passenger feature matrix into the final recursive neural network prediction model to obtain the prediction results for real-time abnormal passengers and make decisions.

[0009] Preferably, the S1 comprises the following steps:

[0010] S11, collect the continuous picture sets of each passenger before and during the security check in the past security accidents, abnormal situations and normal situations during the airport security check, remove the wrong, repeated and irrelevant pictures in each picture set, and construct the first passenger picture matrix A, as follows,

[0011]

[0012] Among them, A in represents the nth image of the ith passenger in the first passenger image matrix A, where N represents the total number of N passengers; collect images of dangerous luggage items, non-dangerous luggage items, and unrecognizable luggage items commonly seen in airport security checks in the past, and construct a luggage item image set a = {a1, a2, ..., a i ,...,a m}, where a i represents the i-th image in the luggage item image set a, and m represents the total number of m images in the luggage item image set a;

[0013] S12, preprocessing the first passenger picture matrix, including the following steps: dividing the first passenger picture matrix A into a passenger picture training matrix B and a passenger picture test matrix D, setting label matrices respectively, and obtaining a passenger picture training label matrix C and a passenger picture test label matrix E, as follows,

[0014]

[0015] Among them B in represents the nth image of the ith passenger in the passenger image training matrix, O represents the total number of O passengers in the passenger image training matrix B, C io represents the oth label of the ith passenger in the passenger image training label matrix; Din represents the nth image of the ith passenger in the passenger image test matrix; P represents the total number of P passengers in the passenger image test matrix D, E io represents the oth label of the ith passenger in the passenger image test label matrix;

[0016] S13, preprocessing the luggage item picture set, including the following steps: i ,...,a m} is divided into luggage item image training set b = {b1, b2, ..., b i ,...,b p} and the luggage item image test set c = {c1,c2,...,c i ,...,c q}; Set the label sets respectively, and obtain the luggage item image training label set d = {d1, d2, ..., d i ,...,d p} and luggage item image test label set e = {e1,e2,...,e i ,...,e q}; where b i represents the i-th image in the luggage item image training set, and p represents the total number of p images in the luggage item image training set; a i represents the i-th image in the luggage item image test set, q represents the total number of q images in the luggage item image training set; d i represents the i-th label in the training label set of luggage item images, e i represents the i-th label in the luggage item image test label set;

[0017] By collecting past airport passenger pictures and luggage pictures, we build the first passenger picture matrix and luggage picture set. We divide the first passenger picture matrix into a training matrix and a test matrix, and the luggage picture set into a training set and a test set, and set the corresponding label matrix and label set, which provide a data basis for subsequent model training and testing, and ensure the training quality and generalization ability of the model.

[0018] Preferably, S2 comprises the following steps:

[0019] S21, CNN convolutional neural network is used for image recognition, LSTM long short-term memory network is used to process time series data, CNN convolutional neural network and LSTM long short-term memory network are serially fused to construct the initial CL network model;

[0020] S22. Train the initial CL network model using the passenger picture training matrix B and the passenger picture training label matrix C; after the training is completed, obtain the trained CL network model, then input the passenger picture test matrix D and the passenger picture test label matrix E into the trained CL network model for testing, and optimize the trained CL network model according to the test results to obtain the final CL network model;

[0021] S23. Construct an initial CNN convolutional neural network model, and use the luggage item picture training set, the luggage item picture test set, the luggage item picture training label set, and the luggage item picture test label set to train and optimize the initial CNN convolutional neural network model to obtain the final CNN convolutional neural network model;

[0022] Construct a CL convolutional recurrent neural network model by combining the CNN convolutional neural network and the LSTM long short-term memory network to better process continuous picture features and changes; by adjusting the model parameters, enable the model to learn from the training data, test the model to evaluate the generalization ability of the model, and ensure that the model can perform well in practical applications; optimize the model by adjusting the model according to the test results to improve its performance.

[0023] Preferably, the S22 includes the following steps:

[0024] S221. Set the number of iterations for training the initial CL network model to g, the batch size to u, the optimizer to the adam optimizer, and the initial learning rate to α;

[0025] S222. Set the current training iteration number to z and the training error threshold to l; input the passenger picture training matrix B and the passenger picture training label matrix C into the initial CL network model for training according to the batch size u, the adam optimizer, and the initial learning rate α. When the training error of the initial CL network model < l or when z ≥ g, stop training to obtain the trained CL network model;

[0026] S223. Set the accuracy threshold w, input the passenger picture test matrix D and the passenger picture test label matrix E into the trained CL network model for testing to obtain the test accuracy k; when k ≥ w, use the trained CL network model as the final CL network model; when k < w, optimize the initial learning rate α of the trained CL network model to obtain the optimized CL network model, and use the optimized CL network model as the final CL network model;

[0027] By setting the relevant parameters of the CL network model, the passenger image training matrix is ​​input into the CL network model for training to obtain a trained CL network model. The passenger image test matrix is ​​used to test the trained CL network model. According to the test results, the trained CL network model is optimized to make the CL network model more accurate.

[0028] Preferably, the training error of the initial CL network model in S222 is calculated using a missing function, and the function expression is as follows:

[0029]

[0030] Where x is the number of images in the passenger image training matrix; y i is the true label of the i-th image, which takes the value 0 or 1; is the predicted probability of the i-th sample, indicating the probability that the model predicts that the sample belongs to category 1;

[0031] The training error of the initial CL network model is calculated by applying the missing function, and the result is used as an important indicator in the model training process to more accurately evaluate the performance of the model and guide the optimization and adjustment of the model.

[0032] Preferably, the step of optimizing the initial learning rate α of the trained CL network model in S223 to obtain the optimized CL network model comprises the following steps:

[0033] S2231, randomly initialize the position of the particle swarm h={h1,h2,...,h i ,...,h Np}, where h i represents the initial position of the ith particle, Np is the number of particles; the initialization particle velocity v = {v1,v2,...,v i ,...,v Np}, where v i represents the i-th velocity; initialize the optimal position of each particle f={f1,f2,...,f i ,...,f Np}, where f i represents the best position reached by the ith particle; initialize the global best position l to any position in f; set the maximum number of iterations to f′, and the performance of the global best position reaches the target value f″;

[0034] S2232. For each particle, update the speed Where t represents the current moment, t+1 represents the next moment, j is the inertia weight, c1 and c2 are learning factors, r1 and r2 are random numbers ranging from 0 to 1, and f it is the optimal position reached by the ith particle at time t, l t is the global optimal position reached by the ith particle at time t;

[0035] Update Location The updated position is used as the new learning rate to obtain the updated learning rate, and the updated learning rate is used to train the trained CL network model and evaluate the performance. If the updated position The performance is better than f i t , then update f i , if the new position The performance is not better than f i t , keep the original f i ; At time t+1, traverse the historical best position f of all particles i , and select the position with the best performance as the new global optimal position l t+1 ;

[0036] S2233, repeat S2232, when the performance of the global best position reaches the target value f″ or the number of iterations reaches f′, stop the iteration process, output the learning rate value corresponding to the global best position, and obtain the optimal learning rate;

[0037] S2234, using the optimal learning rate as the learning rate of the trained CL network model to obtain an optimized CL network model;

[0038] The particle swarm optimization algorithm is used to optimize the learning rate parameters in the trained CL network model. The performance of the CL network model is used as the optimization indicator during the optimization, thereby continuously improving the performance of the CL network model and making the CL network model more accurate in identifying passenger features in images.

[0039] Preferably, S3 comprises the following steps:

[0040] S31, obtain a real-time continuous picture set of each passenger in the airport through video surveillance; construct a real-time passenger picture matrix F as follows,

[0041]

[0042] Among them, F iG It refers to the Gth picture of the i-th passenger in the real-time passenger picture matrix F, and L represents the total number of passengers in the current airport;

[0043] Substitute the real-time passenger image matrix F into the final CL network model to obtain the safety status of each passenger in the current airport;

[0044] S32, obtaining pictures of luggage items through video monitoring and sensor monitoring, and obtaining a real-time luggage item picture set Where K i refers to the i-th image in the real-time luggage item image set, n p Indicates the total number of images in the real-time luggage item image set;

[0045] Substitute the real-time luggage item image set K into the final CNN convolutional neural network model to obtain the safety status of each luggage item;

[0046] S33, for passengers and luggage whose real-time security situation is safe, normal security inspection is carried out; for passengers and luggage whose real-time security situation is unsafe, the emergency response mechanism is immediately triggered; for luggage whose real-time security situation is abnormal, the luggage is checked intensively; for passengers whose real-time security situation is abnormal, the features extracted from the final CL network model are collected to obtain the real-time abnormal passenger feature matrix Y, as follows,

[0047]

[0048] Among them, Y is It refers to the Sth feature of the i-th passenger in the real-time abnormal passenger feature matrix Y, and R represents the total number of real-time abnormal passengers;

[0049] By transmitting the real-time collected passenger picture sets and luggage item pictures to the final CL network model and the final CNN convolutional neural network model respectively, the security situation results are obtained and decisions are made, realizing real-time monitoring and early warning functions, and improving security inspection efficiency and safety.

[0050] Preferably, S4 comprises the following steps:

[0051] S41, collect past abnormal passenger data, extract features and construct abnormal passenger feature matrix M, as follows:

[0052]

[0053] Among them, M iT It refers to the Tth feature of the i-th passenger in the abnormal passenger feature matrix M, and U represents the total number of abnormal passengers in the abnormal passenger feature matrix;

[0054] The abnormal passenger feature matrix M is divided into an abnormal passenger feature training matrix X and an abnormal passenger feature test matrix Z, as follows:

[0055]

[0056] Among them, X iT It refers to the Tth feature of the i-th passenger in the abnormal passenger feature training matrix X, V represents the total number of abnormal passengers in the abnormal passenger feature test matrix; ZiT It refers to the Tth feature of the i-th passenger in the abnormal passenger feature test matrix Z, and Q represents the total number of abnormal passengers in the abnormal passenger feature test matrix;

[0057] Set a label for the final safety situation of each abnormal passenger, obtain the abnormal passenger feature training label set and the abnormal passenger feature test label set, and obtain the abnormal passenger feature training label set q = {q1, q2, ..., q i ,...,q V} and the abnormal passenger feature test label set r = {r1,r2,...,r i ,...,r Q}, where q i represents the i-th label in the abnormal passenger feature training label set; r i represents the i-th label in the abnormal passenger feature test label set;

[0058] S42, constructing an initial recursive neural network prediction model, and using the abnormal passenger feature training matrix X, the abnormal passenger feature test matrix Z and its corresponding abnormal passenger feature training label set and abnormal passenger feature test label set to train and optimize the initial recursive neural network prediction model to obtain a final recursive neural network prediction model;

[0059] S43, substituting the real-time abnormal passenger feature matrix Y into the final recursive neural network prediction model to obtain the prediction results of the safety situation of the real-time abnormal passengers, and immediately triggering the emergency response mechanism for the passengers whose prediction results are unsafe; and focusing on checking the passengers whose prediction results are safe to avoid machine errors;

[0060] By collecting the features of past abnormal passengers, an abnormal passenger feature matrix is ​​constructed, and the abnormal passenger feature matrix is ​​divided into a training matrix and a test matrix; an initial recursive neural network prediction model is constructed, and the initial recursive neural network prediction model is trained and optimized using the abnormal passenger feature training matrix, the abnormal passenger feature test matrix and its corresponding abnormal passenger feature training label set and abnormal passenger feature test label set to obtain the final recursive neural network prediction model; the features of real-time abnormal passengers are introduced into the trained final recursive neural network prediction model to obtain the safety situation of real-time abnormal passengers, and make decisions to realize the prediction function, which can identify the development trend of abnormal behavior in advance, so as to take preventive measures and reduce the occurrence of safety incidents;

[0061] A deep learning-based airport security situation awareness system, including a data collection and preprocessing module, a deep learning model training and optimization module, a deep learning model application module, and a security situation prediction module;

[0062] The data collection and preprocessing module is used to collect continuous passenger picture sets and luggage picture sets during airport security checks in the past, construct a first passenger picture matrix and a luggage picture set, divide the first passenger picture matrix into a training matrix and a test matrix, and divide the luggage picture set into a training set and a test set; set a corresponding label matrix and label set;

[0063] The deep learning model training and optimization module is used to train, test and optimize the CL network model and the CNN neural network model to obtain the final CL network model and the final CNN neural network model;

[0064] The deep learning model application module transmits the real-time collected passenger picture sets and luggage item pictures to the final CL network model and the final CNN convolutional neural network model respectively, obtains the security situation results and makes decisions;

[0065] The safety situation prediction module is used to bring the characteristics of real-time abnormal passengers into the trained final recursive neural network prediction model, obtain the safety situation of real-time abnormal passengers, and make decisions.

[0066] The present invention has the following beneficial effects:

[0067] By collecting previously publicly available photos of passengers and luggage items at airports and airplanes, we constructed the first passenger photo matrix and luggage item photo set. The first passenger photo matrix was divided into a training matrix and a test matrix, and the luggage item photo set was divided into a training set and a test set. We also set the corresponding label matrix and label set, which provided a data basis for subsequent model training and testing, and ensured the training quality and generalization ability of the model.

[0068] By training, testing and optimizing the CL network model and the CNN neural network model, the final CL network model and the final CNN neural network model are obtained, which improve the recognition accuracy and efficiency of contraband and abnormal behaviors and enhance the performance of the model in practical applications.

[0069] By transmitting the real-time collected passenger picture sets and luggage item pictures to the final CL network model and the final CNN convolutional neural network model respectively, the security situation results are obtained and decisions are made, realizing real-time monitoring and early warning functions, improving security inspection efficiency and safety, and reducing the occurrence of security incidents.

[0070] By bringing the characteristics of real-time abnormal passengers into the trained final recursive neural network prediction model, the safety situation of real-time abnormal passengers can be obtained and decisions can be made; abnormal behaviors can be further identified in advance, so that preventive measures can be taken to reduce the occurrence of safety incidents, thereby improving the initiative and preventive nature of safety management.

[0071] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the invention, the drawings required for describing the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, they can also obtain drawings based on these drawings without paying creative work.

[0073] Figure 1 A flowchart of identifying the real-time security situation of luggage items in a method for airport security inspection security situation awareness based on deep learning provided by the present invention;

[0074] Figure 2 A flowchart of identifying real-time passenger safety situation in a method for airport security situation awareness based on deep learning provided by the present invention;

[0075] Figure 3 A flowchart for identifying real-time abnormal passenger safety situation in a deep learning-based airport security situation awareness method provided by the present invention. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the invention to clearly and completely describe the technical solutions in the embodiments of the invention. Obviously, the described embodiments are only part of the embodiments of the invention, not all of the embodiments. Based on the embodiments in the invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the invention.

[0077] In the description of the present invention, it is necessary to understand that the terms "opening", "upper", "lower", "top", "middle", "inside" and the like indicating orientation or positional relationship are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the components or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the invention.

[0078] Embodiment 1:

[0079] See also Figure 1-3 The present invention discloses a method for airport security situation awareness based on deep learning, comprising the following steps:

[0080] S1, collect continuous picture sets of passengers who have passed airport security checks in the past to construct a first passenger picture matrix; collect pictures of luggage items that have passed airport security checks in the past to construct a baggage item picture set; pre-process the first passenger picture matrix and the baggage item picture set respectively to obtain a pre-processed first passenger picture matrix and a pre-processed baggage item picture set;

[0081] The S1 comprises the following steps:

[0082] S11, collect the continuous picture sets of each passenger before and during the security check in the past security accidents, abnormal situations and normal situations during the airport security check, remove the wrong, repeated and irrelevant pictures in each picture set, and construct the first passenger picture matrix A, as follows,

[0083]

[0084] Among them, A in represents the nth picture of the ith passenger in the first passenger picture matrix A, where N represents a total of N passengers;

[0085] Collect pictures of dangerous luggage items, non-dangerous luggage items, and unrecognizable luggage items that are commonly seen during airport security checks, and construct a luggage item picture set a = {a1, a2, ..., a i ,...,a m}, where a i represents the i-th image in the luggage item image set a, and m represents the total number of m images in the luggage item image set a;

[0086] S12, preprocessing the first passenger picture matrix, including the following steps: dividing the first passenger picture matrix A into a passenger picture training matrix B and a passenger picture test matrix D, setting label matrices respectively, and obtaining a passenger picture training label matrix C and a passenger picture test label matrix E, as follows,

[0087]

[0088] Among them B in represents the nth image of the ith passenger in the passenger image training matrix, O represents the total number of O passengers in the passenger image training matrix B, C io represents the oth label of the ith passenger in the passenger image training label matrix; D in represents the nth image of the ith passenger in the passenger image test matrix; P represents the total number of P passengers in the passenger image test matrix D, E io represents the oth label of the ith passenger in the passenger image test label matrix;

[0089] S13, preprocessing the luggage item picture set, including the following steps: i ,...,a m} is divided into luggage item image training set b = {b1, b2, ..., b i ,...,b p} and the luggage item image test set c = {c1,c2,...,c i ,...,c q}; Set the label sets respectively, and obtain the luggage item image training label set d = {d1, d2, ..., d i ,...,d p} and luggage item image test label set e = {e1,e2,...,e i ,...,e q}; where b i represents the i-th image in the luggage item image training set, and p represents the total number of p images in the luggage item image training set; a i represents the i-th image in the luggage item image test set, q represents the total number of q images in the luggage item image training set; d i represents the i-th label in the training label set of luggage item images, e i represents the i-th label in the luggage item image test label set;

[0090] S2, constructing an initial CL network model; training and optimizing the initial CL network model through the preprocessed first passenger image matrix to obtain a final CL network model; constructing an initial CNN convolutional neural network model; training and optimizing the initial CNN convolutional neural network model through the preprocessed luggage item image set to obtain a final CNN convolutional neural network model;

[0091] The S2 comprises the following steps:

[0092] S21, CNN convolutional neural network is used for image recognition, LSTM long short-term memory network is used to process time series data, CNN convolutional neural network and LSTM long short-term memory network are serially fused to construct the initial CL network model;

[0093] S22, using the passenger image training matrix B and the passenger image training label matrix C to train the initial CL network model; after the training is completed, a trained CL network model is obtained, and then the passenger image test matrix D and the passenger image test label matrix E are input into the trained CL network model for testing, and the trained CL network model is optimized according to the test results to obtain a final CL network model;

[0094] The S22 comprises the following steps:

[0095] S221. Construct an initial CL network model, and set the number of iterations for training the initial CL network model to g, the batch size to u, the optimizer to the adam optimizer, and the initial learning rate to α;

[0096] S222. Set the current training iteration number to z and the training error threshold to l; input the passenger picture training matrix B and the passenger picture training label matrix C into the initial CL network model for training according to the batch size u, the adam optimizer, and the initial learning rate α. When the training error of the initial CL network model < l or when z ≥ g, stop training to obtain the trained CL network model;

[0097] The training error of the initial CL network model in S222 is calculated using a loss function, and the function expression is as follows:

[0098]

[0099] where x is the number of pictures in the passenger picture training matrix; y i is the true label of the i-th picture, taking values 0 or 1; is the predicted probability of the i-th sample, representing the probability that the model predicts this sample belongs to class 1;

[0100] S223. Set the accuracy threshold w, input the passenger picture test matrix D and the passenger picture test label matrix E into the trained CL network model for testing to obtain the test accuracy k; when k ≥ w, use the trained CL network model as the final CL network model; when k < w, optimize the initial learning rate α of the trained CL network model to obtain an optimized CL network model, and use the optimized CL network model as the final CL network model;

[0101] Optimizing the initial learning rate α of the trained CL network model in S223 to obtain an optimized CL network model includes the following steps:

[0102] S2231. Randomly initialize the positions of the particle swarm h = {h1, h2,..., h i ,..., h Np}, where h i represents the initial position of the i-th particle, and Np is the number of particles; initialize the velocities of the particles v = {v1, v2,..., v i ,..., v Np}, where v i represents the i-th velocity; initialize the best position of each particle f = {f1, f2,..., f i ,..., f Np}, where f i represents the best position reached by the ith particle; initialize the global best position l to any position in f; set the maximum number of iterations to f′, and the performance of the global best position reaches the target value f″;

[0103] S2232. For each particle, update the speed Where t represents the current moment, t+1 represents the next moment, j is the inertia weight, c1 and c2 are learning factors, r1 and r2 are random numbers ranging from 0 to 1, and f i t is the optimal position reached by the ith particle at time t, l t is the global optimal position reached by the ith particle at time t;

[0104] Update Location The updated position is used as the new learning rate to obtain the updated learning rate, and the updated learning rate is used to train the trained CL network model and evaluate the performance. If the updated position The performance is better than f i t , then update f i , if the new position The performance is not better than f i t , keep the original f i ; At time t+1, traverse the historical best position f of all particles i , and select the position with the best performance as the new global optimal position l t+1 ;

[0105] S2233, repeat S2232, when the performance of the global best position reaches the target value f″ or the number of iterations reaches f′, stop the iteration process, output the learning rate value corresponding to the global best position, and obtain the optimal learning rate;

[0106] S2234, using the optimal learning rate as the learning rate of the trained CL network model to obtain an optimized CL network model;

[0107] S23, constructing an initial CNN convolutional neural network model, and using the luggage item image training set, the luggage item image test set, the luggage item image training label set, and the luggage item image test label set to train and optimize the initial CNN convolutional neural network model to obtain a final CNN convolutional neural network model;

[0108] S3. Collect real-time pictures in the airport, obtain the real-time passenger picture matrix and the real-time baggage picture set, and transmit them to the final CL network model and the final CNN convolutional neural network model respectively, obtain the real-time security situation results and make decisions; for passengers with abnormal real-time security situation, collect the features extracted from the final CL network model and construct the real-time abnormal passenger feature matrix;

[0109] The S3 comprises the following steps:

[0110] S31, obtain a real-time continuous picture set of each passenger in the airport through video surveillance; construct a real-time passenger picture matrix F as follows,

[0111]

[0112] Among them, F iG It refers to the Gth picture of the i-th passenger in the real-time passenger picture matrix F, and L represents the total number of passengers in the current airport;

[0113] Substitute the real-time passenger image matrix F into the final CL network model to obtain the safety status of each passenger in the current airport;

[0114] S32, obtaining pictures of luggage items through video monitoring and sensor monitoring, and obtaining a real-time luggage item picture set Where K i refers to the i-th image in the real-time luggage item image set, n p Indicates the total number of images in the real-time luggage item image set;

[0115] Substitute the real-time luggage item image set K into the final CNN convolutional neural network model to obtain the safety status of each luggage item;

[0116] S33, for passengers and luggage whose real-time security situation is safe, normal security inspection is carried out; for passengers and luggage whose real-time security situation is unsafe, the emergency response mechanism is immediately triggered; for luggage whose real-time security situation is abnormal, the luggage is checked intensively; for passengers whose real-time security situation is abnormal, the features extracted from the final CL network model are collected to obtain the real-time abnormal passenger feature matrix Y, as follows,

[0117]

[0118] Among them, Y is It refers to the Sth feature of the i-th passenger in the real-time abnormal passenger feature matrix Y, and R represents the total number of real-time abnormal passengers;

[0119] S4. Collect past abnormal passenger data and construct an abnormal passenger feature matrix; construct an initial recursive neural network prediction model, train and optimize the initial recursive neural network prediction model based on the abnormal passenger feature matrix to obtain a final recursive neural network prediction model; substitute the real-time abnormal passenger feature matrix into the final recursive neural network prediction model to obtain the prediction results of the real-time abnormal passengers and make a decision;

[0120] S4 contains the following steps:

[0121] S41, collect past abnormal passenger data, extract features and construct abnormal passenger feature matrix M, as follows:

[0122]

[0123] Among them, M iT It refers to the Tth feature of the i-th passenger in the abnormal passenger feature matrix M, and U represents the total number of abnormal passengers in the abnormal passenger feature matrix;

[0124] The abnormal passenger feature matrix M is divided into an abnormal passenger feature training matrix X and an abnormal passenger feature test matrix Z, as follows:

[0125]

[0126] Among them, X iT It refers to the Tth feature of the i-th passenger in the abnormal passenger feature training matrix X, V represents the total number of abnormal passengers in the abnormal passenger feature training matrix; Z iT It refers to the Tth feature of the i-th passenger in the abnormal passenger feature test matrix Z, and Q represents the total number of abnormal passengers in the abnormal passenger feature test matrix;

[0127] Set a label for each abnormal passenger's final safety situation as safe or unsafe, and obtain the abnormal passenger feature training label set q = {q1, q2, ..., q i ,...,q V} and the abnormal passenger feature test label set r = {r1,r2,...,r i ,...,r Q}, where q i represents the i-th label in the abnormal passenger feature training label set; r i represents the i-th label in the abnormal passenger feature test label set;

[0128] S42, constructing an initial recursive neural network prediction model, and using the abnormal passenger feature training matrix X, the abnormal passenger feature test matrix Z and its corresponding abnormal passenger feature training label set and abnormal passenger feature test label set to train and optimize the initial recursive neural network prediction model to obtain a final recursive neural network prediction model;

[0129] S43. Substitute the real-time abnormal passenger feature matrix Y into the final recursive neural network prediction model to obtain the prediction results of the safety situation of the real-time abnormal passengers. For passengers whose prediction results are unsafe, the emergency response mechanism is immediately triggered; for passengers whose prediction results are safe, they are still checked intensively to avoid machine errors.

[0130] Embodiment 2:

[0131] The present invention also discloses a deep learning-based airport security situation awareness system, which includes a data collection and preprocessing module, a deep learning model training and optimization module, a deep learning model application module, and a security situation prediction module;

[0132] The data collection and preprocessing module is used to collect continuous passenger picture sets and luggage picture sets during airport security checks in the past, construct a first passenger picture matrix and a luggage picture set, divide the first passenger picture matrix into a training matrix and a test matrix, and divide the luggage picture set into a training set and a test set; set a corresponding label matrix and label set;

[0133] The deep learning model training and optimization module is used to train, test and optimize the CL network model and the CNN neural network model to obtain the final CL network model and the final CNN neural network model;

[0134] The deep learning model application module transmits the real-time collected passenger picture sets and luggage item pictures to the final CL network model and the final CNN convolutional neural network model respectively, obtains the security situation results and makes decisions;

[0135] The safety situation prediction module is used to bring the characteristics of real-time abnormal passengers into the trained final recursive neural network prediction model, obtain the safety situation of real-time abnormal passengers, and make decisions.

[0136] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0137] The preferred embodiments of the invention disclosed above are only used to help explain the invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the invention, so that those skilled in the art can understand and use the invention well.

Claims

1. A method for airport security situation awareness based on deep learning, characterized in that: The following steps are involved: S1, collect continuous pictures of passengers who have passed airport security checks, and construct the first passenger picture matrix; Collecting past airport security baggage pictures and constructing a baggage picture set; preprocessing the first passenger picture matrix and the baggage picture set respectively to obtain a preprocessed first passenger picture matrix and a preprocessed baggage picture set; S2, constructing an initial CL network model; training and optimizing the initial CL network model through the preprocessed first passenger image matrix to obtain a final CL network model; constructing an initial CNN convolutional neural network model; training and optimizing the initial CNN convolutional neural network model through the preprocessed luggage item image set to obtain a final CNN convolutional neural network model; S3, collect real-time pictures in the airport, obtain the real-time passenger picture matrix and the real-time luggage picture set, and transmit them to the final CL network model and the final CNN convolutional neural network model respectively, obtain the real-time security situation results and make decisions; For passengers whose real-time safety situation is abnormal, the features extracted from the final CL network model are collected to construct a real-time abnormal passenger feature matrix; S4, collecting past abnormal passenger data and constructing an abnormal passenger feature matrix; constructing an initial recursive neural network prediction model, training and optimizing the initial recursive neural network prediction model based on the abnormal passenger feature matrix, and obtaining a final recursive neural network prediction model; Substitute the real-time abnormal passenger feature matrix into the final recursive neural network prediction model to obtain the prediction results of real-time abnormal passengers and make decisions.

2. The airport security situation awareness method based on deep learning according to claim 1 is characterized in that: The S1 comprises the following steps: S11, collecting continuous picture sets of each passenger before and during security checks in security accidents, abnormal situations and normal situations that occurred during past airport security checks, removing erroneous, repeated and irrelevant pictures in each picture set, and constructing a first passenger picture matrix; collecting pictures of common dangerous luggage items, non-dangerous luggage items and unrecognizable luggage items during past airport security checks, and constructing a baggage item picture set; S12, preprocessing the first passenger image matrix, including the following steps: The first passenger picture matrix is ​​divided into a passenger picture training matrix and a passenger picture test matrix, and label matrices are set respectively to obtain a passenger picture training label matrix and a passenger picture test label matrix; S13, pre-processing the luggage item image set, including the following steps: The luggage item image set is divided into a luggage item image training set and a luggage item image test set; and label sets are set respectively to obtain a luggage item image training label set and a luggage item image test label set.

3. The airport security situation awareness method based on deep learning according to claim 2 is characterized in that: The S2 comprises the following steps: S21, serially fuse the CNN convolutional neural network and the LSTM long short-term memory network to construct the initial CL network model; S22. Train the initial CL network model using the passenger picture training matrix and the passenger picture training label matrix; after the training is completed, obtain the trained CL network model, then input the passenger picture test matrix and the passenger picture test label matrix into the trained CL network model for testing, and optimize the trained CL network model according to the test results to obtain the final CL network model; S23. Construct the initial CNN convolutional neural network model, and use the luggage item picture training set, the luggage item picture test set, the luggage item picture training label set, and the luggage item picture test label set to train and optimize the initial CNN convolutional neural network model to obtain the final CNN convolutional neural network model.

4. The airport security situation awareness method based on deep learning according to claim 3 is characterized in that: The S22 includes the following steps: S221. Set the number of iterations for training the initial CL network model as g, the batch size as u, the optimizer as the adam optimizer, and the initial learning rate as α; S222. Set the current training iteration number as z and the training error threshold as l; input the passenger picture training matrix and the passenger picture training label matrix into the initial CL network model for training according to the batch size u, the adam optimizer, and the initial learning rate α. When the training error of the initial CL network model < l or when z ≥ g, stop the training to obtain the trained CL network model; S223. Set the accuracy threshold w, input the passenger picture test matrix D and the passenger picture test label matrix E into the trained CL network model for testing to obtain the test accuracy k; when k ≥ w, use the trained CL network model as the final CL network model; when k < w, optimize the initial learning rate α of the trained CL network model to obtain the optimized CL network model, and use the optimized CL network model as the final CL network model.

5. The airport security situation awareness method based on deep learning according to claim 4 is characterized in that: The training error of the initial CL network model in S222 is calculated using the loss function, and the function expression is as follows: Where x is the number of images in the passenger image training matrix; y i is the true label of the i-th image, which takes the value of 0 or 1; is the predicted probability of the i-th sample, indicating the probability that the model predicts that the sample belongs to category 1.

6. The airport security situation awareness method based on deep learning according to claim 5 is characterized in that: The optimization of the initial learning rate α of the trained CL network model in S223 to obtain the optimized CL network model includes the following steps: S2231, randomly initialize the position of the particle swarm h={h1,h2,...,h i ,...,h Np }, where h i represents the initial position of the ith particle, Np is the number of particles; the initialization particle velocity v = {v1,v2,...,v i ,...,v Np }, where v i represents the i-th velocity; initialize the optimal position of each particle f={f1,f2,...,f i ,...,f Np }, where f i represents the best position reached by the ith particle; initialize the global best position l to any position in f; set the maximum number of iterations to f′, and the performance of the global best position reaches the target value f″; S2232. For each particle, update the speed Where t represents the current moment, t+1 represents the next moment, j is the inertia weight, c1 and c2 are learning factors, r1 and r2 are random numbers ranging from 0 to 1, and f i t is the optimal position reached by the ith particle at time t, l t is the global optimal position reached by the ith particle at time t; Update Location The updated position is used as the new learning rate to obtain the updated learning rate, and the updated learning rate is used to train the trained CL network model and evaluate the performance. If the updated position The performance is better than f i t , then update f i If the new position The performance is not better than f i t , keep the original f i ; At time t+1, traverse the historical best position f of all particles i , and select the position with the best performance as the new global optimal position l t+1 ; S2233. Repeat S2232. When the performance at the global best position reaches the target value f″ or the number of iterations reaches f′, stop the iteration process, output the learning rate value corresponding to the global best position to obtain the optimal learning rate; S2234. Use the optimal learning rate as the learning rate of the trained CL network model to obtain the optimized CL network model.

7. The airport security situation awareness method based on deep learning according to claim 6 is characterized in that: The S3 includes the following steps: S31. Obtain the real-time continuous picture set of each passenger in the airport through video monitoring; construct the real-time passenger picture matrix, and substitute the real-time passenger picture matrix into the final CL network model to obtain the security situation of each passenger in the current airport; S32. Obtain the pictures of luggage items through video monitoring and sensor monitoring to obtain the real-time luggage item picture set; substitute the real-time luggage item picture set into the final CNN convolutional neural network model to obtain the security situation of each luggage item; S33. Perform normal security checks on passengers and luggage items whose real-time security status is safe; immediately trigger the emergency response mechanism for passengers and luggage items whose real-time security status is unsafe; focus on checking luggage items whose real-time security status is abnormal; for passengers whose real-time security status is abnormal, collect the features extracted from the final CL network model to obtain the real-time abnormal passenger feature matrix.

8. The airport security situation awareness method based on deep learning according to claim 7 is characterized in that: S4 contains the following steps: S41, collecting past abnormal passenger data, extracting features to construct an abnormal passenger feature matrix, and dividing the abnormal passenger feature matrix into an abnormal passenger feature training matrix and an abnormal passenger feature test matrix; Setting an abnormal passenger feature training label set and an abnormal passenger feature test label set; S42, constructing an initial recursive neural network prediction model, and using the abnormal passenger feature training matrix, the abnormal passenger feature test matrix and its corresponding abnormal passenger feature training label set, and the abnormal passenger feature test label set to train and optimize the initial recursive neural network prediction model to obtain a final recursive neural network prediction model; S43. Substitute the real-time abnormal passenger feature matrix into the final recursive neural network prediction model to obtain the prediction results of the safety situation of the real-time abnormal passengers. For passengers predicted to be unsafe, the emergency response mechanism is immediately triggered; for passengers predicted to be safe, they are still subject to intensive investigation.

9. A system for airport security situation awareness method based on deep learning as described in any one of claims 1 to 8, characterized in that: It includes data collection and preprocessing module, deep learning model training and optimization module, deep learning model application module, and security situation prediction module.

Images