Verification method and system for self-service security check

Abstract

The invention discloses a verification method and system for self-service security check, and the method comprises the following steps: collecting a front face image and a side face image of a sample object as a sample image, carrying out the preprocessing of the sample image, and carrying out the extraction, replacement and integration of shallow features, and obtaining a sample fusion face image shallow feature matrix; constructing a sample fusion face image shallow feature matrix of a certain scale as a training data set, and inputting the training data set into the MobileNet network model to obtain a deep feature extraction model; and performing similarity comparison on the second fusion face image deep feature matrix and each first fusion face image deep feature matrix of the current boarding flight in the passenger feature library, setting a comparison similarity threshold value, if the similarity is higher than a preset similarity threshold value, passing verification, and if the similarity is lower than the preset similarity threshold value, not passing verification.

Description

Technical Field

[0001] The present invention relates to the technical field of security inspection, and particularly to a verification method and system for self-service security inspection. Background Art

[0002] Airport security inspection is an essential part for passengers to take civil aviation flights. It mainly involves airport management personnel manually verifying the identities of passing passengers and then inspecting the passengers' carry-on luggage. However, with the continuous acceleration of China's urbanization process, the urban population density has increased sharply, and the passenger flow in large and medium-sized airports, high-speed railway stations and other public transportation places has also been increasing year by year. This has brought new challenges to airport security, and at the same time forced airlines and airports to improve the work efficiency of each link. Therefore, it is particularly important to simplify the process after passengers arrive at the airport and reduce the queuing time for security inspection, and to ensure the authenticity and security of passengers boarding the plane.

[0003] To solve the problems of low efficiency of manual identity verification and long queuing time for security inspection, the current airport has proposed a method of introducing face recognition into airport security inspection for identity verification. For example, the existing patent CN109117727A - A method for verifying the authenticity of passengers at the boarding gate based on face recognition technology discloses that when passengers check in, the camera automatically collects the face images of passengers, the identity card recognition device automatically discriminates the authenticity of the identity card and automatically recognizes the identity card number. When passengers pass through airport security inspection, the barcode scanner scans and associates the departure date and flight number on the boarding pass. This scheme compares the face photo on the passenger's identity card with the face presented by the passenger on the spot, that is, identity verification, and then scans the boarding pass to complete ticket verification.

[0004] The current problems are as follows: The identity verification method is a system that requires passengers to actively cooperate. Passengers need to face the face collection device (camera) to collect a frontal face, and at the same time need to carry valid identity documents such as identity cards or boarding passes with them for verification. When passing through airport security inspection, passengers need to take out valid identity documents from their bags and then actively cooperate with face collection. This process takes a long time, and the security inspection speed cannot meet the extremely large passenger flow in the airport. At the same time, due to the existence of certain common features in similar faces, it is easy to make misidentifications, resulting in low recognition accuracy, only reaching 97%. The authenticity of information cannot be guaranteed, which may lead to criminals using others' identity documents to sneak into the entrance channel and board the plane, posing a great threat to the lives and property safety of the people. Summary of the Invention

[0005] The first object of the present invention is to provide a verification method for self-service security inspection, and the second object of the present invention is to provide a verification system for self-service security inspection.

[0006] The first object of the present invention is implemented by the following technical solution: A verification method for self-service security inspection, comprising the following steps:

[0007] S1. Establish a passenger feature library and extract the deep feature matrix of the first fused face image:

[0008] Using the document information of the passenger's identity document as the registration information, collect the frontal face image and the profile face image of the passenger as the registration images. After preprocessing the registration images, perform shallow feature extraction, replacement, and integration to obtain the shallow feature matrix of the first fused face image. The shallow feature matrix of the first fused face image is subjected to deep feature extraction using a deep feature extraction model to obtain the deep feature matrix of the first fused face image. Store the registration information and the deep feature matrix of the first fused face image in the passenger feature library;

[0009] S2. Associate the passenger feature library with the aviation information platform;

[0010] S3. Extract the deep feature matrix of the second fused face image:

[0011] Collect the frontal face image and the profile face image of the passenger at the security inspection site as real-time images. After preprocessing the real-time images, perform shallow feature extraction, replacement, and integration to obtain the shallow feature matrix of the second fused face image. The shallow feature matrix of the second fused face image is subjected to deep feature extraction using a deep feature extraction model to obtain the deep feature matrix of the second fused face image;

[0012] S4. Similarity comparison:

[0013] Retrieve the corresponding deep feature matrix of the first fused face image in the passenger feature library according to the boarding pass information of the current flight. Compare the deep feature matrix of the second fused face image with the retrieved deep feature matrix of the first fused face image in the passenger feature library. Set a comparison similarity threshold. If the similarity is higher than the preset similarity threshold, the verification passes; if the similarity is lower than the preset similarity threshold, the verification fails.

[0014] Furthermore, to construct the deep feature extraction model, the specific steps include: collecting the frontal face image and the profile face image of the sample object as the sample images. After preprocessing the sample images, perform shallow feature extraction, replacement, and integration to obtain the shallow feature matrix of the sample fused face image;

[0015] Construct a certain scale of the shallow feature matrix of the sample fused face image as the training data set, and input the training data set into the MobileNet network model to complete the training of the MobileNet network model and obtain the deep feature extraction model.

[0016] Furthermore, for the image preprocessing, the specific steps include:

[0017] Perform face detection on the frontal face image and the profile face image to obtain the frontal face region image and the profile face region image;

[0018] Expand the frontal face region image and the profile face region image outward by a certain multiple, and then intercept the expanded frontal face region image and profile face region image to obtain the cropped frontal face region image and the cropped profile face region image;

[0019] Perform face key point detection on the cropped profile face region image to obtain three key points of the upper, middle and lower parts of the ear. Construct a rectangular region with the upper and lower two key points as diagonal anchor points to obtain the ear region image. Expand the ear region image outward by a certain multiple, and then intercept the expanded ear region image to obtain the cropped ear region image;

[0020] Scale the cropped frontal face region image, the cropped profile face region image and the cropped ear region image to a fixed size to obtain the scaled frontal face region image, the scaled profile face region image and the scaled ear region image.

[0021] Furthermore, the frontal face region image, the profile face region image and the ear region image are expanded outward by a certain multiple, and the specific multiple is 1.2.

[0022] Furthermore, scaling the cropped frontal face region image, the cropped profile face region image and the cropped ear region image to a fixed size, specifically, the size of the scaled frontal face region image is (W×H), retaining the three color channels RGB of the color image (W×H×3); the size of the scaled profile face region image is (W×H), converted from the color image to a grayscale image, retaining only one color channel (W×H×1); according to the scaling ratio of the profile face region image, the size of the scaled ear region image is (w×h), converted from the color image to a grayscale image, retaining only one color channel (w×h×1).

[0023] Furthermore, the specific steps of the shallow feature extraction, replacement and integration include:

[0024] Extract shallow features from the scaled frontal face region image, the scaled profile face region image and the scaled ear region image respectively through three pre-trained fully convolutional neural networks, while keeping the dimension of the output feature matrix the same as that of the input image, and obtain the frontal face shallow feature matrix with a size of (W×H×3), the profile face shallow feature matrix with a size of (W×H×1), and the ear shallow feature matrix with a size of (w×h×1);

[0025] Replace the elements in the corresponding relative positions of the ear region in the lateral face shallow feature matrix with the elements in the ear shallow feature matrix one by one, and after integration, the fused side face image shallow feature matrix is obtained;

[0026] Merge the color channels of the previously extracted frontal face shallow feature matrix and the integrated fused side face image shallow feature matrix and stack them together to obtain the fused face image shallow feature matrix with a size of (W×H×4).

[0027] Furthermore, the passenger feature library is associated with the aviation information platform, including the registration information of the passengers in the passenger feature library, and the first fused face image deep feature matrix is bound to the boarding pass information of the aviation information platform.

[0028] Furthermore, the frontal face image and the lateral face image of the passengers at the security inspection site are collected through a frontal camera and a lateral camera respectively.

[0029] The second object of the present invention is implemented by the following technical solution: A verification system for self-service security inspection, which includes:

[0030] A registration module, which is used to collect the document information of the passengers, the frontal face image and the lateral face image of the passengers, and obtain the registration information and the registration image;

[0031] A passenger feature library, which is used to store the first fused face image deep feature matrix extracted from the registration image through an image processing and feature extraction module and the associated registration information. The passenger feature library binds the registration information of the passengers, the first fused face image deep feature matrix to the boarding pass information of the aviation information platform;

[0032] An acquisition module, including a frontal camera and a lateral camera, which is used to collect the frontal face image and the lateral face image of the passengers at the security inspection site and obtain the real-time image;

[0033] An image processing and feature extraction module, which is used to expand, intercept, and scale the collected frontal face image, lateral face image, and the constructed ear region image, and then extract the shallow features of the scaled frontal face region image, scaled lateral face region image, and scaled ear region image through a convolutional neural network. After shallow feature extraction, replacement, and integration, a fused face image shallow feature matrix is obtained, and a deep feature extraction model is used to perform deep feature extraction on the fused face image shallow feature matrix to obtain the fused face image deep feature matrix;

[0034] A verification module, configured to compare the deep feature matrix of the second fused face image with each deep feature matrix of the first fused face image of the current boarding flight in the passenger feature database, set a comparison similarity threshold, and send an instruction to the control module according to the similarity comparison result;

[0035] A control module, configured to control the turnstile or the alarm. If the similarity is higher than the preset similarity threshold, the control module controls the turnstile to open. If the similarity is lower than the preset similarity threshold, the control module controls the alarm to give an alarm;

[0036] The output end of the registration module is communicatively connected to the input end of the image processing and feature extraction module. The output end of the image processing and feature extraction module is communicatively connected to the input end of the passenger feature database. The output end of the acquisition module is communicatively connected to the input end of the image processing and feature extraction module. The output end of the image processing and feature extraction module, the output end of the passenger feature database are communicatively connected to the input end of the verification module. The output end of the verification module is communicatively connected to the input end of the control module. The output end of the control module is communicatively connected to the input ends of the turnstile and the alarm.

[0037] Advantages of the present invention:

[0038] By comparing the deep feature matrix of the second fused face image with each deep feature matrix of the first fused face image of the current boarding flight in the passenger feature database, the verification of the passenger's identity is realized. Since the passenger feature database is associated with the aviation information platform, the passenger feature database binds the passenger's registration information, the deep feature matrix of the first fused face image with the boarding pass information of the aviation information platform, and retrieves the corresponding deep feature matrix of the first fused face image in the passenger feature database according to the boarding pass information of the current flight of the aviation information platform. The deep feature matrix of the second fused face image is compared with the retrieved deep feature matrix of the first fused face image in the passenger feature database to complete the verification of the person and the ticket, shorten the security check time and boarding time of the passenger, improve the efficiency of the security check, without repeated verification of the identity information, and at the same time, only those who meet the conditions of the current flight can pass the security check, also avoiding the problem of passengers boarding the wrong plane;

[0039] By collecting the frontal face and side face of passengers at the security check site and preprocessing the pictures, the scaled frontal face area image, the scaled side face area image, and the scaled ear area image are obtained. Then, through shallow feature extraction, replacement, and integration, the shallow feature matrix of the second fused face image is obtained. By performing deep feature extraction operations on the shallow feature matrix of the second fused face image, the deep feature matrix of the second fused face image is obtained. Based on the fusion of ear information including the front and side, the deep feature matrix of the second fused face image is obtained, realizing the fusion of overall and local face features. By comparing the deep feature matrix of the second fused face image with the deep feature matrix of the first fused face image, the accuracy of verification is improved, the common features of similar faces are effectively reduced, the incidence of misidentification is reduced, the recognition accuracy of security check is improved, and the recognition accuracy reaches more than 99.5%, ensuring the safety of people's lives and property;

[0040] Using the system of the present invention, each component module cooperates with each other to realize the comparison between the deep feature matrix of the second fused face image of passengers at the security check site and the deep feature matrix of the first fused face image in the passenger feature library, improving the accuracy of security check; at the same time, under the association of the passenger feature library and the aviation information platform, the registration information, the deep feature matrix of the first fused face image, and the boarding pass information are associated. When the deep feature matrix of the second fused face image matches the deep feature matrix of the first fused face image, it can be default that the person-certificate-ticket verification is passed, completing the security check, accelerating the security check speed, optimizing the security check process, and saving the manpower of airport security check. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0042] Figure 1 It is a flowchart of the verification method for self-service security check in Embodiment 1;

[0043] Figure 2 It is a flowchart for constructing the deep feature extraction model in Embodiment 1;

[0044] Figure 3 It is a flowchart of the verification method for self-service security check in Embodiment 1;

[0045] Figure 4 It is a principle block diagram of the verification system for self-service security check in Embodiment 2; DETAILED DESCRIPTION OF THE INVENTION

[0046] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0047] Embodiment 1:

[0048] As Figures 1-3 shown, a verification method for self-service security inspection includes the following steps:

[0049] S1. Establish a passenger feature library and extract the deep feature matrix of the first fused face image:

[0050] Establish a passenger feature library. Use the document information of the passenger's identity document as the registration information, collect the frontal face image and the side face image of the passenger as the registration images. After preprocessing the registration images, perform shallow feature extraction, replacement, and integration to obtain the shallow feature matrix of the first fused face image. The shallow feature matrix of the first fused face image uses a deep feature extraction model for deep feature extraction to obtain the deep feature matrix of the first fused face image, and associate the registration information with the deep feature matrix of the first fused face image and store it in the passenger feature library.

[0051] S2. Associate the passenger feature library with the aviation information platform;

[0052] Bind the registration information of the passenger in the passenger feature library, the deep feature matrix of the first fused face image, and the boarding pass information of the aviation information platform. Bind the registration information of the passenger, the deep feature matrix of the first fused face image, and the flight information. Retrieve the corresponding deep feature matrix of the first fused face image in the passenger feature library according to the boarding pass information of the current flight on the aviation information platform.

[0053] S3. Extract the deep feature matrix of the second fused face image:

[0054] Collect the frontal face image and the side face image of the passenger at the security inspection site as real-time images. The frontal face image and the side face image are collected by a frontal camera and a side camera respectively. After preprocessing the real-time images, perform shallow feature extraction, replacement, and integration to obtain the shallow feature matrix of the second fused face image. The shallow feature matrix of the second fused face image uses a deep feature extraction model for deep feature extraction to obtain the deep feature matrix of the second fused face image.

[0055] S4. Similarity comparison:

[0056] Retrieve the corresponding first fused face image deep feature matrix in the passenger feature library according to the boarding pass information of the current flight. Compare the second fused face image deep feature matrix with the retrieved first fused face image deep feature matrix in the passenger feature library. Set a comparison similarity threshold. If the similarity is higher than the preset similarity threshold, the verification passes; if the similarity is lower than the preset similarity threshold, the verification fails. Perform the comparison based on the finally fused deep features to achieve passenger identity verification. The first fused face image deep feature matrix and the second fused face image deep feature matrix are one-dimensional vectors of fixed length. After normalizing all feature vectors (the feature vectors are converted into direction-invariant unit vectors by dividing by their own lengths), calculate the cosine similarity between each pair (the radian value of the angle opened by two unit vectors) for comparison.

[0057] Construct a deep feature extraction model. The specific steps include: Collect the frontal face images and profile face images of the sample objects as sample images. After preprocessing the sample images through image preprocessing, then perform shallow feature extraction, replacement, and integration to obtain the sample fused face image shallow feature matrix. Construct a sample fused face image shallow feature matrix of a certain scale as the training data set. Input the training data set into the MobileNet network model to complete the training of the MobileNet network model and obtain the deep feature extraction model.

[0058] The above-mentioned image preprocessing specifically includes: Perform face detection on the frontal face image and the profile face image to obtain the frontal face region image and the profile face region image.

[0059] Expand the frontal face region image and the profile face region image outward by a certain multiple. Expand the frontal face region image, the profile face region image, and the ear region image outward by a certain multiple. The specific multiple is 1.2. Expanding the face region image outward by a part can include more comprehensive face edge information (such as ears, hair, accessories, etc.) based on the face facial information, which helps to obtain more accurate recognition results.

[0060] Then intercept the expanded frontal face region image and the profile face region image to obtain the cropped frontal face region image and the cropped profile face region image, and crop the corresponding images according to the obtained regions to exclude irrelevant redundant information for subsequent use.

[0061] Perform face key point detection (35 points) on the cropped side face region image to obtain three key points on the upper, middle, and lower parts of the ear. Use the upper and lower key points as diagonal anchor points to construct a rectangular region. If the width of the rectangular region is too narrow, expand the width of the rectangle to at least half of the height to obtain the ear region image. Obtain face key points for ear region positioning. In the 35-point face key points, a single ear region contains two key points, the upper and the lower. (If these two points are approximately on a vertical line, the located ear region may be too narrow in width. In this case, its width needs to be appropriately expanded to include a more complete ear region.) Expand the ear region image outward by a certain multiple, and then intercept the expanded ear region image to obtain the cropped ear region image;

[0062] Scale the cropped front face region image, the cropped side face region image, and the cropped ear region image to a fixed size. The purpose is to facilitate the input into the subsequent neural network model. Since the input image size of the neural network model is fixed, the images must be scaled to the corresponding size to be input. Here, the "fixed size" corresponds to the input image size of the neural network model, specifically the size (W×H) of the scaled front face region image, retaining the three color channels RGB of the color image (W×H×3). Retaining the RGB color channels can retain more and more original input information. That is, in the subsequent recognition process, the front face image is the main information;

[0063] The size of the scaled side face region image is (W×H). Convert it from a color image to a grayscale image, retaining only one color channel (W×H×1). Scale the image to a fixed size to facilitate the input into the subsequent neural network model. Retaining only a single grayscale channel can compress the input information to a certain extent. That is, in the subsequent recognition process, the side face image is the secondary information;

[0064] According to the scaling ratio of the side face region image, the face image and the ear region image are scaled proportionally. The purpose is to ensure that there is no misalignment before and after image scaling, and the ear region can correspond to an area with the same relative position in the face region, facilitating subsequent processing in the same size dimension. The size of the scaled ear region image is (w×h). Convert it from a color image to a grayscale image, retaining only one color channel (w×h×1) to obtain the scaled front face region image, the scaled side face region image, and the scaled ear region image;

[0065] The above-mentioned shallow feature extraction, replacement, and integration specifically include the following steps: Shallow features are extracted from the scaled frontal face region image, scaled side face region image, and scaled ear region image respectively through three pre-trained fully convolutional neural networks, while keeping the dimension of the output feature matrix the same as that of the input image and the size of the output features consistent with that of the input image, so as to facilitate the subsequent replacement process of the elements in the shallow feature matrix. The size of the shallow feature matrix of the frontal face is (W×H×3), the size of the shallow feature matrix of the side face is (W×H×1), and the size of the shallow feature matrix of the ear is (w×h×1). The training of the FCN pre-trained on ImageNet can be carried out by adding a fully connected layer at the end of the network through the labels of existing images. After the training is completed, the fully connected layer can be discarded. The reason for using the FCN is that both the input and output of this model are matrix images, and the size before and after feature extraction can be kept the same, which is convenient for subsequent operations.

[0066] One by one, the elements within the relative position corresponding to the ear region in the shallow feature matrix of the side face are replaced with the elements in the shallow feature matrix of the ear. Replacing the elements in the relative position of the shallow features corresponding to the ear region in the shallow features of the side face is equivalent to performing more refined calculation and processing on this part of the shallow features. On this basis, deep features are further extracted to obtain more accurate recognition results. After integration, the shallow feature matrix of the fused side face image is obtained.

[0067] The shallow feature matrix of the frontal face extracted above and the shallow feature matrix of the fused side face image obtained through integration are combined by stacking the color channels together to obtain the shallow feature matrix of the fused face image, with a size of (W×H×4). Combining the shallow feature matrix of the frontal face and the shallow feature matrix of the fused side face image gives the finally integrated shallow features of the face, and the total number of channels is 3 (frontal) + 1 (side) = 4.

[0068] A computer program is stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned verification method for self-service security inspection are implemented. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more integrated available media, and can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a solid-state drive.

[0069] By collecting the frontal face and side face of passengers at the security check site, and preprocessing the pictures at the same time, the scaled frontal face area image, the scaled side face area image and the scaled ear area image are obtained. Then, through shallow feature extraction, replacement and integration, the shallow feature matrix of the second fused face image is obtained. After performing deep feature extraction on the shallow feature matrix of the second fused face image, the shallow feature matrix of the second fused face image is obtained. Based on the fusion of ear information including the front and side, the deep feature matrix of the second fused face image is obtained, realizing the fusion of overall and local face features, which is beneficial to improving the accuracy of verification, effectively reducing the common features of similar faces, reducing the incidence of misidentification errors, improving the recognition accuracy, and ensuring the safety of people's lives and property;

[0070] According to the boarding pass information of the current flight, the corresponding deep feature matrix of the first fused face image in the passenger feature library is retrieved. The deep feature matrix of the second fused face image is compared with the retrieved deep feature matrix of the first fused face image in the passenger feature library to verify the identity of the passenger. Since the passenger feature library is associated with the aviation information platform, the passenger feature library binds the passenger's registration information, the deep feature matrix of the first fused face image with the boarding pass information of the aviation information platform. According to the boarding pass information of the current flight on the aviation information platform, the deep feature matrix of the first fused face image in the passenger feature library is retrieved and compared with the deep feature matrix of the first fused face image of the current flight to complete the verification of the person and the ticket. Passengers do not need to present their ID cards during the security check process, shortening the security check time and boarding time of passengers, accelerating the security check efficiency, eliminating the need for repeated verification of identity information, and at the same time, only those who meet the conditions of the current flight can pass the security check, also avoiding the problem of passengers boarding the wrong plane.

[0071] Embodiment 2:

[0072] As Figure 4 shown, based on the same inventive concept as the verification method for self-service security check provided in Embodiment 1, Embodiment 2 of the present invention also provides a verification system for self-service security check, including: It includes:

[0073] A registration module, which is used to collect the passenger's certificate information, frontal face image and side face image of the passenger to obtain registration information and registration images;

[0074] A passenger feature library, which is used to store the deep feature matrix of the first fused face image obtained by extracting the registration image through the image processing and feature extraction module and the associated registration information. The passenger feature library binds the passenger's registration information, the deep feature matrix of the first fused face image with the boarding pass information of the aviation information platform;

[0075] The acquisition module, including a front camera and a side camera, is used to acquire the front face image and the side face image of passengers at the security check site to obtain real-time images;

[0076] The image processing and feature extraction module is used to expand, intercept, and scale the acquired front face image, side face image, and the constructed ear region image, and then extract shallow features from the scaled front face region image, scaled side face region image, and scaled ear region image through a convolutional neural network. After that, through shallow feature extraction, replacement, and integration, a shallow feature matrix of the fused face image is obtained. Then, a deep feature extraction model is used to perform deep feature extraction on the shallow feature matrix of the fused face image to obtain a deep feature matrix of the fused face image; The image processing and feature extraction module specifically includes an image acquisition device, an image preprocessing unit, and a convolutional neural network model 、 The deep feature extraction model.

[0077] The verification module is used to compare the similarity between the second deep feature matrix of the fused face image and each first deep feature matrix of the fused face image of the current boarding flight in the passenger feature library, set a comparison similarity threshold, and send an instruction to the control module according to the similarity comparison result;

[0078] The control module is used to control the turnstile or the alarm. If the similarity is higher than the preset similarity threshold, the control module controls the turnstile to open. If the similarity is lower than the preset similarity threshold, the control module controls the alarm to sound;

[0079] The output end of the registration module is communicatively connected to the input end of the image processing and feature extraction module. The output end of the image processing and feature extraction module is communicatively connected to the input end of the passenger feature library. The output end of the acquisition module is communicatively connected to the input end of the image processing and feature extraction module. The output ends of the image processing and feature extraction module and the passenger feature library are communicatively connected to the input end of the verification module. The output end of the verification module is communicatively connected to the input end of the control module. The output end of the control module is communicatively connected to the input ends of the turnstile and the alarm.

[0080] The specific operation process of this embodiment:

[0081] 1. Preliminary preparation:

[0082] Before purchasing an airplane ticket, a passenger needs to register a personal account through the user terminal. The registration module collects the document information of the passenger's identity document as the registration information, and collects the front face image and the side face image of the passenger as the registration images;

[0083] The output end of the registration module is connected to the input end of the image processing and feature extraction module. After processing the registration image through the image processing and feature extraction module, a first fused face image deep feature matrix is obtained. Then, the first fused face image deep feature matrix and its associated registration information are stored in the passenger feature database. The passenger feature database is associated with the aviation information platform, so that the passenger feature database binds the passenger's registration information, the first fused face image deep feature matrix with the boarding pass information of the aviation information platform;

[0084] 2. On-site security check:

[0085] The front camera and side camera of the acquisition module are used to acquire the front face image and side face image of the passenger at the security check site, obtaining real-time images;

[0086] The output end of the acquisition module is connected to the input end of the image processing and feature extraction module. After processing the real-time image through the image processing and feature extraction module, a second fused face image deep feature matrix is obtained;

[0087] 3. Similarity verification:

[0088] The second fused face image deep feature matrix is sent to the verification module. The verification module compares the second fused face image deep feature matrix with each of the first fused face image deep feature matrices of the current boarding flight. If the similarity is higher than the preset similarity threshold, the control module controls the turnstile to open. If the similarity is lower than the preset similarity threshold, the control module controls the alarm to sound.

[0089] Through the collaborative cooperation among the above-mentioned various component modules, the comparison between the second fused face image deep feature matrix of the passenger at the security check site and the first fused face image deep feature matrix in the passenger feature database is realized, improving the accuracy of the security check. At the same time, under the association between the passenger feature database and the aviation information platform, the registration information, the first fused face image deep feature matrix and the boarding pass information are associated. When the second fused face image deep feature matrix matches the first fused face image deep feature matrix, it can be defaulted that the person-certificate-ticket verification is passed, completing the security check, accelerating the security check speed, optimizing the security check process, and saving the airport security manpower.

[0090] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A verification method for self-service security check, characterized in that: The following steps are involved: S1. Establish a passenger feature database and extract the deep feature matrix of the first fused face image: Using the ID card information of the passenger as the registration information, collecting the front face image and the side face image of the passenger as the registration image, performing image preprocessing on the registration image, and then performing shallow feature extraction, replacement and integration to obtain a first fused face image shallow feature matrix, performing deep feature extraction on the first fused face image shallow feature matrix using a deep feature extraction model to obtain a first fused face image deep feature matrix, and storing the registration information and the first fused face image deep feature matrix in the passenger feature library; S2. The passenger feature database is associated with the aviation information platform; S3, extract the deep feature matrix of the second fused face image: Collecting frontal face images and side face images of passengers at the security check site as real-time images, performing image preprocessing on the real-time images, and then performing shallow feature extraction, replacement and integration to obtain a shallow feature matrix of a second fused face image, and performing deep feature extraction on the shallow feature matrix of the second fused face image using a deep feature extraction model to obtain a deep feature matrix of the second fused face image; S4. Similarity comparison: According to the boarding pass information of the current flight, the corresponding first fused face image deep feature matrix in the passenger feature library is retrieved, and the second fused face image deep feature matrix is ​​compared with the retrieved first fused face image deep feature matrix of the passenger feature library, and a comparison similarity threshold is set. If the similarity is higher than the preset similarity threshold, the verification is passed; if the similarity is lower than the preset similarity threshold, the verification fails.

2. The verification method for self-service security check according to claim 1, characterized in that: Constructing the deep feature extraction model, the specific steps include: collecting the front face image and the side face image of the sample object as sample images, performing image preprocessing on the sample images, and then performing shallow feature extraction, replacement and integration to obtain a shallow feature matrix of the sample fusion face image; A shallow feature matrix of the sample fusion face image of a certain scale is constructed as a training data set, and the training data set is input into the MobileNet network model to complete the training of the MobileNet network model to obtain a deep feature extraction model.

3. The verification method for self-service security check according to claim 2, characterized in that: The picture preprocessing specifically includes the following steps: Performing face detection on the front face image and the side face image to obtain a front face region image and a side face region image; Expanding the front face region image and the side face region image outwardly according to a certain multiple, and then intercepting the expanded front face region image and the side face region image to obtain a cropped front face region image and a cropped side face region image; Performing facial key point detection in the cropped side face region image to obtain three key points of the upper, middle and lower ears, constructing a rectangular region with the upper and lower key points as diagonal anchor points to obtain an ear region image, expanding the ear region image outward by a certain multiple, and then intercepting the expanded ear region image to obtain a cropped ear region image; The cropped front face region image, the cropped side face region image and the cropped ear region image are scaled to a fixed size to obtain a scaled front face region image, a scaled side face region image and a scaled ear region image.

4. The verification method for self-service security check according to claim 3, characterized in that: The front face region image, the side face region image and the ear region image are expanded outwards at a certain multiple, and the specific multiple is 1.

2.

5. The verification method for self-service security check according to claim 4, characterized in that: The cropped front face area image, the cropped side face area image and the cropped ear area image are scaled to a fixed size, specifically, the size of the scaled front face area image (W×H) is retained, and the three color channels (W×H×3) of the color image RGB are retained; the size of the scaled side face area image (W×H) is converted from a color image to a grayscale image, and only one color channel (W×H×1) is retained; according to the scaling ratio of the side face area image, the size of the scaled ear area image (w×h) is converted from a color image to a grayscale image, and only one color channel (w×h×1) is retained.

6. The verification method for self-service security check according to claim 2, characterized in that: The shallow feature extraction, replacement and integration specifically include the following steps: Extract shallow features from the scaled front face region image, the scaled side face region image, and the scaled ear region image respectively through three pre-trained fully convolutional neural networks, while keeping the dimension of the output feature matrix the same as that of the input image, and obtain the shallow feature matrix size of the front face (W×H×3), the shallow feature matrix size of the side face (W×H×1), and the shallow feature matrix size of the ear (w×h×1); The elements in the relative position corresponding to the ear region in the side face shallow feature matrix are replaced one by one with the elements in the ear shallow feature matrix, and the shallow feature matrix of the fused side face image is obtained through integration; The shallow feature matrix of the frontal face extracted above is integrated with the shallow feature matrix of the fused side face image, and the color channels are combined and stacked together to obtain the shallow feature matrix of the fused face image, with a size of (W×H×4).

7. The verification method for self-service security check according to claim 1, characterized in that: The passenger feature database is associated with the aviation information platform, including the registration information of the passengers in the passenger feature database, and the first fused face image deep feature matrix is ​​bound to the boarding pass information of the aviation information platform.

8. The verification method for self-service security check according to claim 1, characterized in that: The frontal facial image and side facial image of passengers at the security check site are collected by the front camera and the side camera respectively.

9. A verification system for self-service security check, characterized in that: These include: A registration module is used to collect the passenger's ID information and the passenger's front face image and side face image to obtain registration information and registration image; A passenger feature database is used to store the first fused face image deep feature matrix and the registration information associated therewith, which are obtained by extracting the registration image through the image processing and feature extraction module. The passenger feature database binds the passenger's registration information, the first fused face image deep feature matrix and the boarding pass information of the aviation information platform; The acquisition module includes a front camera and a side camera, which are used to acquire front and side facial images of passengers at the security check site to obtain real-time images; The image processing and feature extraction module is used to expand, intercept and scale the collected frontal face image, side face image and constructed ear region image, extract shallow features from the scaled frontal face region image, scaled side face region image and scaled ear region image through a convolutional neural network, and then extract, replace and integrate the shallow features to obtain a shallow feature matrix of the fused face image, and use a deep feature extraction model to perform deep feature extraction on the shallow feature matrix of the fused face image to obtain a deep feature matrix of the fused face image; A verification module, used for performing a similarity comparison between the second fused face image deep feature matrix and each of the first fused face image deep feature matrices of the current boarding flight in the passenger feature library, setting a comparison similarity threshold, and sending instructions to the control module according to the similarity comparison result; A control module, used to control the gate or the alarm. If the similarity is higher than a preset similarity threshold, the control module controls the gate to open. If the similarity is lower than the preset similarity threshold, the control module controls the alarm to sound an alarm. The output end of the registration module is communicatively connected to the input end of the image processing and feature extraction module, the output end of the image processing and feature extraction module is communicatively connected to the input end of the passenger feature library, the output end of the acquisition module is communicatively connected to the input end of the image processing and feature extraction module, the output end of the image processing and feature extraction module and the output end of the passenger feature library are communicatively connected to the input end of the verification module, the output end of the verification module is communicatively connected to the input end of the control module, and the output end of the control module is communicatively connected to the input end of the gate and the alarm.

Images