Security check identification and determination method and system

Abstract

The present application belongs to the field of passenger security check, and particularly relates to a security check identification and determination method and system. The present application extracts part of video frames in a video stream for corresponding processing and target detection to obtain an identification result, and calculates the identification result of the video frames not extracted according to the moving speed of the conveyor belt and the identification result of the extracted video frames. Thus, through frame extraction identification and calculation of the remaining frames, the identification result of all video frames can be obtained by identifying a small number of frames, which reduces the requirement for the algorithm processing efficiency and total processing amount during processing and target detection, and avoids processing lag. In the processing step of the push stream, if the number of video frames in the processing result queue of the previous processing step exceeds a set number threshold, the set time interval between two frames during the push stream is reduced to play the video frames at double speed, so as to ensure the synchronization of the video frames after processing and target detection when the luggage frame passes through the security check machine.

Description

Technical Field

[0001] This invention belongs to the field of passenger security inspection, specifically relating to a security inspection identification and judgment method and system. Background Technology

[0002] Intelligent passenger security check system is a research topic in the application of airport passenger security check and control system. On January 26, 2022, the "Notice on Doing a Good Job in the Construction of Baggage Full-Process Tracking System" clearly required that by the end of 2023, airports with less than 10 million passengers should complete the construction of passenger baggage full-process tracking system. According to the survey, 85.4% of domestic airports with less than 10 million passengers have such systems, and most of them do not have information technology and intelligent equipment, making it difficult to realize functions such as baggage box information recognition, baggage tracking, human-computer interaction, and real-time display.

[0003] Chinese invention patent application CN112934728A discloses a technical solution for an airport baggage security check status tracking system and method. This solution, when baggage is detected entering the primary tag recognition area, controls a primary tag reader via a host computer to identify the tag information on the baggage and generates baggage tracking information based on the tag information recognition result. When baggage is detected leaving the primary tag recognition area, the host computer associates and stores the current baggage tracking information with the tag information on the baggage. When baggage is detected entering the security check area, the host computer controls the security check equipment to start, performs X-ray scanning and photographs the baggage, and records the images on its image station. The system displays baggage images and obtains security check status information. When baggage is detected leaving the security check area, the host computer associates and stores the received security check status information with the current baggage tracking information. However, this solution relies heavily on the real-time nature of tag information recognition, scanning, and baggage image display, as well as the image recognition method for security check identification, based on real-time acquired information (especially image information). Therefore, to ensure the recognition effect, the algorithm processing efficiency and total processing volume are both high. If only conventional computing power data processing equipment is used, problems such as processing lag and data backlog may occur during the recognition process. Summary of the Invention

[0004] The purpose of this invention is to provide a security inspection identification and judgment method and system to solve the problems of processing lag and data backlog that may occur in the real-time identification and judgment process of existing security inspection identification and judgment methods.

[0005] To achieve the above objectives, the present invention provides a security inspection and identification method, comprising the following steps:

[0006] 1) Acquire the video stream corresponding to the security check in real time, process the video frames in the video stream and perform target detection to obtain the corresponding recognition results; the recognition results include the location and number of the luggage frame in the identified video frame;

[0007] The processed and target-detected video frames are played in real time to perform security checks on the luggage frames and obtain the luggage frame security check results.

[0008] 2) Based on the recognition results corresponding to the video frames in the video stream and the security check judgment results of the luggage frame, obtain the security check recognition judgment result information;

[0009] The method for processing video frames in the video stream includes: extracting video frames from the video stream at set intervals, performing corresponding processing and target detection on the extracted video frames to obtain recognition results; and calculating the recognition results of the unextracted video frames based on the moving speed of the conveyor belt and the recognition results of the extracted video frames.

[0010] The beneficial effects of the above technical solution are as follows: by extracting frames for recognition and calculating the remaining frames, only a small number of frames need to be actually recognized to obtain the recognition results of all video frames, which reduces the requirements for algorithm processing efficiency and total processing volume when processing video frames and detecting targets in the video stream, while avoiding processing lag.

[0011] Furthermore, based on the conveyor belt's moving speed and the identification results of the extracted video frames, the methods for inferring the identification results of the unextracted video frames include:

[0012] Obtain the recognition result of the extracted video frame before the unextracted video frame, translate the luggage frame position in the recognition result of the extracted video frame by a corresponding translation distance in the direction of the conveyor belt movement, and use the translation result as the recognition result of the unextracted video frame.

[0013] The corresponding translation distance is obtained by the moving speed of the conveyor belt.

[0014] Furthermore, the process of processing video frames and detecting objects in the video stream includes object detection algorithm recognition for determining the location of the luggage frame and the number on the luggage frame, OCR algorithm recognition for recognizing the luggage frame number, and streaming processing steps; different processing steps are executed using different separate threads; the streaming refers to sending video frames for real-time playback.

[0015] The beneficial effects of the above technical solution are as follows: different processing steps are executed using different separate threads. Compared with the sequential execution of each processing step using a single thread, this can improve the parallelism of the program, make full use of CPU and GPU resources, increase the throughput of data processing, and improve processing efficiency.

[0016] Furthermore, the streaming process retrieves a video frame from the processing result queue of the previous processing step at a set time interval and sends it for real-time playback. If the number of video frames in the processing result queue exceeds a set threshold, the set time interval between two frames during streaming is reduced to allow for faster playback of the video frames.

[0017] The beneficial effects of the above technical solution are: when it is determined that there may be a backlog of video frames (i.e., when the number of video frames in the processing result queue exceeds the set threshold), the video frames are played at double speed in a timely manner, which can ensure the synchronization of the video frames after processing and target detection when the luggage frame passes through the security inspection machine.

[0018] Furthermore, if there is no video frame data in the processing result queue after the set time interval, the video frame is retrieved and streamed immediately after it is added to the queue.

[0019] The beneficial effects of the above technical solution are: to minimize the impact of stuttering in upstream processing steps on the frequency of streaming frames and to improve the real-time performance of video frames after playback processing.

[0020] Furthermore, the luggage frame number includes a check code; the characters in the obtained luggage frame number image are split and recombined to generate a new, different number text image that does not conform to the check code rules corresponding to the luggage frame number. The generated new number text image is added to the training set of the OCR algorithm recognition model to train the OCR algorithm recognition model. The trained OCR algorithm recognition model is used to perform the OCR algorithm recognition processing steps to identify the luggage frame number in the video frame.

[0021] The beneficial effects of the above technical solution are: it can solve the problem that OCR recognition records the results as a "fixed template", which leads to the output of previously recognized luggage box numbers in the event of recognition errors, causing the verification code to become invalid, thereby enhancing the robustness and accuracy of the algorithm.

[0022] Furthermore, the methods for security checks on luggage frames in video frames include:

[0023] Obtain the recognition result corresponding to the current video frame being played in real time, and display the luggage frame in the current video frame in the form of a first type of graphic.

[0024] The luggage frame in the current video frame is selected by the mouse cursor in the first type of graphic form, and the luggage frame is subjected to security inspection. If the inspection result is suspicious, the suspicious area of ​​the luggage frame is determined according to the coordinates of the mouse cursor. Luggage frames that have not been inspected are displayed in the first type of graphic form, luggage frames that are safe are displayed in the second type of graphic form, and luggage frames that are suspicious are displayed in the third type of graphic form.

[0025] The beneficial effects of the above technical solution are: it optimizes the visualization of the object to be judged during security checks, making the operation process during security checks more intuitive.

[0026] Furthermore, based on the recognition results corresponding to the video frames in the video stream and the baggage frame security inspection judgment results, the methods for obtaining the security inspection recognition judgment result information include:

[0027] Obtain the operation information corresponding to the security check, including the coordinates of the mouse cursor, the judgment result, the operation time, and the video frame ID;

[0028] Based on the coordinates of the mouse cursor corresponding to the operation and the video frame ID, the recognition result corresponding to the video frame is obtained, thereby determining which luggage compartment the mouse cursor is located in. If it is determined that the mouse cursor is not in the area corresponding to any luggage compartment, an error is determined. If it is determined that the mouse cursor is in the area corresponding to a luggage compartment, the number of that luggage compartment is obtained.

[0029] Convert the video frame corresponding to the video frame ID into an image, and if the luggage frame is deemed suspicious, mark the suspicious area of ​​the luggage frame in the image corresponding to the video frame.

[0030] The information is assembled based on the luggage frame number, the luggage frame security inspection result, the coordinates of the suspicious area of ​​the luggage frame when the result is suspicious, the video frame ID, the image corresponding to the video frame, and the operation time to obtain the security inspection and identification result information of the luggage frame in the video frame.

[0031] The beneficial effects of the above technical solution are: binding the security check identification and judgment result information, the identification result and the security check operation, making it easier to obtain more complete security check related information corresponding to the luggage frame, which is conducive to realizing the functions of luggage frame information identification, luggage tracking and security check information backtracking.

[0032] Furthermore, if during the security check of the luggage frame in the video frame, it is detected that the luggage frame number has appeared within a set time, then the screen is rewound, and the graphic form corresponding to the luggage frame is displayed according to the previous security check result of the luggage frame number; if the luggage frame is checked again, the latest check result is used as the luggage frame security check result.

[0033] The beneficial effects of the above technical solution are: it ensures that the image rewind function can be realized without affecting the recognition results of the luggage frame number and the acquisition of the security inspection judgment results.

[0034] The present invention also provides a security inspection and identification system, including a processor, which is used to execute program instructions to implement the security inspection and identification method described above.

[0035] This security inspection and identification system can achieve the same beneficial effects as the security inspection and identification methods described above. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the overall process of the security inspection and identification method in an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram illustrating the principle of the ASI-YOLOV5 algorithm used in the target detection algorithm identification in the embodiment of the security inspection and identification method of the present invention;

[0038] Figure 3 This is a flowchart illustrating the process of using an OCR algorithm to identify luggage frame numbers in an embodiment of the security inspection and identification method of the present invention.

[0039] Figure 4 This is a schematic diagram illustrating the principle of OCR algorithm recognition in an embodiment of the security inspection and identification method of the present invention;

[0040] Figure 5 This is a flowchart illustrating the process of obtaining a security check result for a luggage frame by performing a security check on a luggage frame in a video frame, as described in an embodiment of the security check identification and determination method of the present invention.

[0041] Figure 6 This is a flowchart illustrating the process of obtaining security inspection and identification result information in an embodiment of the security inspection and identification method of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0043] Examples of Security Inspection and Identification Methods

[0044] This embodiment provides a technical solution for a security inspection and identification method, referring to... Figure 1 It includes the following steps:

[0045] 1) Acquire the video stream corresponding to the security check in real time, process the video frames in the video stream and perform target detection to obtain the corresponding recognition results; among which, the recognition results include the location and number of the luggage frame in the identified video frame;

[0046] The processed and target-detected video frames are played in real time to perform security checks on the luggage frames and obtain the luggage frame security check results.

[0047] In this embodiment, the video stream corresponding to the security check is the X-ray video stream of the security inspection machine. The X-ray video stream is primarily read using a video capture card. The location of the luggage compartment and its number is determined by a target detection algorithm, and the luggage compartment number is identified using an OCR algorithm. Specifically, the methods for processing video frames and performing target detection in the video stream to obtain the corresponding recognition results include the following:

[0048] In the target detection algorithm recognition process of this embodiment, the ASI-YOLOv5 algorithm for airport security scenarios is used (its network structure is as follows). Figure 2 The steps shown are to implement target detection and recognition (i.e., the processing steps of target detection algorithm recognition).

[0049] The overall structure of the ASI-YOLOV5 algorithm network consists of a Backbone, a Neck, and a Head. The Backbone is mainly responsible for extracting relevant feature information from the image input to the network. The Neck is used to further process the features extracted by the Backbone to improve the network performance. The Head is used to implement the final classification and localization functions.

[0050] The backbone mainly consists of the C3, Focus, and SPP modules. The neck uses a combination of FPN and PANet. FPN fuses information from high and low layers of the network, using high-level semantic information to improve the performance of low layers. PANet adds a bottom-up path structure, directly passing low-level information upwards, which can better utilize low-level information. The model head is mainly used for the final detection part. This head part includes four branches. The input of each branch is the feature map obtained by downsampling the input image by 4, 8, 16, and 32 times, respectively. A detection result vector is generated at each scale to detect different targets.

[0051] The ASI-YOLOv5 algorithm is a multi-scale network. It extracts and downsamples features through a backbone network, and then uses a network structure combining FPN and PANet to predict the target on feature maps at multiple different scales. The downsampling factors for these feature maps are 8, 16, and 32, respectively. In the data scenario of this embodiment, feature prediction is performed based on multi-scale detection, improving the feature extraction capability for shallow features and obtaining more fine-grained shallow feature information while preserving deep semantic information.

[0052] For the video stream generated during the security check process, the object detection algorithm reads the video frames one by one into memory and performs data preprocessing operations according to the data characteristics. The preprocessed video frame images are then input into the trained ASI-YOLOV5 model for object detection, obtaining the location and category information of the luggage racks and their numbers (in this embodiment, the metal number tags located on the luggage racks). The output results of the ASI-YOLOV5 model are then post-processed, using the non-maximum suppression (NMS) algorithm to remove overlapping bounding boxes and low-confidence predictions. The processed output data of the ASI-YOLOV5 model is then returned, which is used to draw and display the luggage racks and their numbers in the video frame, and also serves as input for OCR text recognition.

[0053] In the OCR algorithm recognition process (i.e., the processing steps of OCR algorithm recognition) of this embodiment, the OCR algorithm used is the MMT-OCR algorithm for metal tag text recognition in airport security scenarios. This algorithm is based on the deep learning convolutional recurrent neural network structure and is used for end-to-end recognition of text sequences of variable length. This algorithm does not require explicit text segmentation, but transforms text recognition into a sequence learning problem. Although the input image scale and text length are different, after passing through DCNN and RNN, and after a certain CTC translation transcription in the output stage, the entire text image can be recognized, transforming text recognition into a time-dependent sequence learning problem, i.e., image sequence recognition. In the prediction process of the OCR algorithm, the features of the input text image containing text are first extracted using a standard CNN network, and then the feature vectors are fused using BLSTM to extract the context features of the character sequence. Then, the probability distribution of each column of features is obtained, and finally, the text sequence recognized based on the text image containing text is obtained through CTC prediction.

[0054] This embodiment also utilizes BLSTM and CTC to learn the contextual relationships in text images, effectively improving the accuracy of text recognition and making the model more robust; that is, the sequence modeling part of the OCR algorithm adopts bidirectional LSTM, which has the following advantages: (1) text is a sequence, and the CNN model that extracts features can only see the image features of the nearby frames, while bidirectional LSTM can fuse features from further away, so that the model can see the complete whole word; (2) LSTM can be concatenated with CNN using the Back-Propagation Through Time (BPTT) method for training; (3) LSTM can handle inputs of arbitrary length; the transcription part of the OCR algorithm uses CTC, which has different processing methods during training and prediction. The alignment of CTC is based on the rule of "first merging all repeated characters, then deleting all whitespace characters".

[0055] In this embodiment, refer to Figure 3 During manufacturing and installation, the luggage frame number is ensured to have the top of the text facing inwards towards the inside of the frame. Each rectangular luggage frame has two metal tags along its edge, for a total of eight tags per frame. All tags have identical content and the text always faces inwards. The first three digits of the text on the metal tags represent the luggage frame number, and the last three digits are a verification digit. The OCR-recognized text is verified using a validation algorithm to determine its accuracy. If five or more tags pass the verification and have identical text, the text is considered the luggage frame number; otherwise, the luggage frame number is considered not recognized.

[0056] The principle of the OCR algorithm in this embodiment is as follows: Figure 4 As shown, the entire network structure consists of three parts, from bottom to top: Convolutional layer: using a deep CNN to extract features from the input image and obtain a feature map; Recurrent layer: using a bidirectional RNN (BLSTM) to predict the feature sequence, learning each feature vector in the sequence, and outputting the predicted label (true value) distribution; Transcription layer: using CTC loss to transform a series of label distributions obtained from the recurrent layer into the final label sequence.

[0057] In this embodiment, the luggage frame number used is 6 digits, with each digit consisting of 32 characters: numbers and uppercase letters. The first 3 digits are an auto-incrementing sequence number, and the last 3 digits are a check digit, calculated from the first 3 digits. Specifically, the luggage frame number has 32 characters per digit, which can be converted to and from a 5-bit binary number. Both the auto-incrementing sequence number and the check digit can be converted to 15-bit binary. Using the CRC15 algorithm, 15-bit binary input can be used to output a 15-bit binary check digit. The output binary check digit is converted into 3 characters, which are the last 3 digits of the number. As long as the polynomial of the CRC algorithm is chosen appropriately, this method can achieve no check digit conflicts, meaning that the check digit for each number is unique, and the linear correlation between the check digit and the first 3 digits of the number is weak, which is beneficial for detecting recognition errors in the OCR recognition results. Multiple redundant metal tags with the luggage frame number are set on each luggage frame, which can achieve high redundancy in number recognition and mutual verification of multiple number OCR results, thereby improving the recognition accuracy.

[0058] Because commonly used deep learning-based OCR algorithms use RNNs, when the training set of numbered text is small, they tend to "memorize" the complete number as a series of words. This leads to a high probability of outputting "memorized words" in case of recognition errors, rendering the checksum invalid. Therefore, in this embodiment, the characters in the acquired luggage frame number image are split and recombined to generate new, different numbered text images that do not conform to the checksum rules corresponding to the luggage frame number. These new numbered text images are added to the training set of the OCR algorithm recognition model to train the model. The trained OCR algorithm recognition model is then used to perform the OCR recognition processing steps to identify the luggage frame number in the video frame. This solves the problem of OCR recognition recording the result as a "fixed template," thereby enhancing the robustness and accuracy of the algorithm. Specifically, when training the OCR algorithm recognition model, the characters in the existing luggage frame number image are split and recombined to generate a large number of different numbered text images that do not conform to the checksum rules, which are then used for training. This solves the problem of OCR recognition results "memorizing" the number as a word.

[0059] The methods for processing video frames in a video stream include frame extraction algorithms:

[0060] Video frames are extracted from the video stream at set intervals. These extracted frames are then processed and subjected to target detection to obtain recognition results. Furthermore, based on the conveyor belt's speed and the recognition results of the extracted video frames, the recognition results of the unextracted video frames are calculated. Specifically, the methods for calculating the recognition results of the unextracted video frames include:

[0061] The process involves obtaining the recognition results of the extracted video frames before the unextracted video frames are obtained. The luggage frame position in the recognition results of the extracted video frames is then shifted a corresponding distance in the direction of the conveyor belt's movement. This shifted result is used as the recognition result for the unextracted video frame. The corresponding shift distance is determined by the conveyor belt's speed. For example, when processing and detecting targets in a video stream, 10 frames can be processed per second, and the corresponding recognition results for each frame are output. Since the frame rate of the acquired video stream and the output real-time playback video frames is 30 frames per second, one frame out of every three is taken for processing and target detection. That is, frames 1, 4, and 7 are used for recognition; for example, frame 2 has no recognition result. Therefore, the luggage frame position and luggage frame number in the recognition result of frame 1 are shifted to the right (i.e., in the direction of the conveyor belt's movement) by a certain number of pixels, and this is considered the recognition result for frame 2. The same process is applied to frame 3, and the same logic applies to calculating the recognition results for other unextracted video frames. In this embodiment, the distance the recognition result is shifted to the right (i.e., the corresponding shift distance mentioned above) = conveyor belt speed * interval time between the 1st and 2nd frames. Since the conveyor belt speed is generally uniform, the conveyor belt speed can be monitored in real time based on the value of the luggage frame movement distance / interval time in the 1st and 4th frames. This frame extraction processing step allows the algorithm to obtain the recognition results of all video frames by extracting frames and calculating the remaining frames when the algorithm's data throughput does not reach the planned video stream frame rate. This reduces the requirements on algorithm processing efficiency and total processing volume when processing video frames and detecting targets in the video stream, while also avoiding processing lag.

[0062] In this embodiment, security checks are primarily conducted by relevant personnel, such as security inspectors, through a human-computer interaction interface. Therefore, referring to... Figure 5 The method of playing video frames in real time to perform security checks on luggage frames and obtaining the security check results includes the following:

[0063] The HDMI signal is converted into a video stream using a video capture card. This video stream is then played on the computer screen's human-computer interaction interface, enabling real-time playback of video frames. Simultaneously, each video frame in the video stream is converted into an image, using a millisecond-level timestamp as the video frame ID. When processing and detecting targets in this video frame, the corresponding image and video frame ID are input to obtain the recognition result for that video frame.

[0064] The recognition result corresponding to the current video frame being played in real time is obtained, and the luggage frame in the current video frame is displayed in the form of a first type of graphic. In this embodiment, the first type of graphic is a yellow bounding box rectangle (hereinafter referred to as the bounding box rectangle). According to the position of the recognized luggage frame, the luggage frame to be recognized is displayed on the human-computer interaction interface in the form of a bounding box (in this embodiment, the yellow bounding box rectangle is drawn on the screen of the human-computer interaction interface), and the number of the recognized luggage frame is drawn above the bounding box.

[0065] The user selects a luggage frame in the current video frame using the mouse cursor (in the first type of graphic format) for security inspection. If the inspection result is suspicious, the suspicious area of ​​the luggage frame is determined based on the coordinates of the mouse cursor. Luggage frames that have not undergone security inspection are displayed in the first type of graphic format, luggage frames that are deemed safe are displayed in the second type of graphic format, and luggage frames that are deemed suspicious are displayed in the third type of graphic format. In this embodiment, if the user (e.g., a security officer) moves the mouse over a luggage box, the border of the luggage box will be thickened as a highlight. The user uses the left and right mouse buttons or presses a keyboard key to indicate that the luggage frame corresponding to the highlighted luggage box is being inspected. Different mouse buttons and keyboard keys are used to indicate whether the luggage box is safe or suspicious. If the inspection result is suspicious, the coordinates of the mouse cursor are considered the suspicious area of ​​the luggage frame. Specifically, luggage boxes that have not undergone security inspection are displayed in yellow, luggage boxes deemed safe are displayed in green, and luggage boxes deemed suspicious are displayed in red.

[0066] Additionally, if security personnel need to re-evaluate a recently passed baggage rack, they can use the security scanner to rewind the video. Therefore, if, during the security check of a baggage rack in a video frame, a baggage rack number is detected to have appeared within a set time frame, a rewind is performed, and the corresponding graphic of the baggage rack is displayed based on the previous security check result for that baggage rack number. If the baggage rack is checked again, the latest result is used. Typically, this set time frame is the maximum rewind time; in this embodiment, it is a maximum of 30 seconds. If a baggage rack number is detected to have appeared within 30 seconds during a security check, it is considered that a video rewind has caused the baggage rack, which had left the screen, to re-enter the frame. The previous check result for that baggage rack number is then retrieved, and the bounding box color (green or red) is determined on the interface according to the previous check result. If the user checks the baggage rack again, the latest check result is used for display.

[0067] 2) Based on the recognition results corresponding to the video frames in the video stream and the baggage frame security inspection judgment results, obtain the security inspection recognition judgment result information; specifically, refer to... Figure 6The methods for obtaining security check and identification results include:

[0068] Obtain the operation information corresponding to the security check, including the coordinates of the mouse cursor, the judgment result, the operation time, and the video frame ID. In this embodiment, obtaining the operation information corresponding to the security check means obtaining the operation information of the security inspector in the human-computer interaction, including the screen coordinates of the mouse cursor (used to determine which luggage compartment to operate), the judgment result, the operation time, and the video frame ID corresponding to the page.

[0069] Based on the coordinates of the mouse cursor corresponding to the operation and the video frame ID, the recognition result corresponding to the video frame is obtained, thereby determining which luggage frame the mouse cursor is located in. If it is determined that the mouse cursor is not in the area corresponding to any luggage frame, an error is detected, an error message is displayed, and the user is notified. If it is determined that the mouse cursor is in the area corresponding to a luggage frame, the number of that luggage frame is obtained. In this embodiment, the area corresponding to the luggage frame is the area range of the luggage frame displayed in the form of a bounding box. Determining which luggage frame the mouse cursor is located in is equivalent to determining which luggage frame the mouse cursor is located in the bounding box.

[0070] The video frame corresponding to the video frame ID is converted into an image, and if the luggage frame is determined to be suspicious, the suspicious area of ​​the luggage frame is marked in the image corresponding to the video frame; in this embodiment, a rectangle is drawn at the coordinates of the mouse cursor in the image to mark the suspicious area.

[0071] The system assembles the following information: luggage frame number, luggage frame security inspection result, coordinates of the suspicious area of ​​the luggage frame if the result is suspicious (this information is only available for luggage frames deemed suspicious), video frame ID, the image corresponding to the video frame, and operation time. This information is then sent to the front-end database.

[0072] In summary, the overall process of steps 1)-2) above includes the following steps: Step 1), which involves processing video frames and detecting targets in the video stream. These steps include target detection algorithm recognition for determining the location of the luggage frame and its number, OCR algorithm recognition for recognizing the luggage frame number, and streaming (sending video frames to the playback front-end for real-time playback). In this embodiment, the streaming process also involves taking a video frame from the processing result queue of the previous processing step at a set time interval and sending it for real-time playback. If there is no video frame data in the processing result queue after the set time interval, the process waits for a video frame to be added to the queue before immediately taking the video frame out and streaming it. If the number of video frames in the processing result queue exceeds a set threshold, the set time interval between two frames during streaming is reduced to speed up the playback of the video frames, i.e., a speed-up playback method when data is backlogged. For example, under normal circumstances, the last processing step, "pushing", takes one output video frame from the processing result queue of the previous processing step every 33.3ms (that is, the set time interval between two frames during the push is 33.3ms) and pushes it to the front-end player, so the frame rate is 30 frames per second.

[0073] If, due to delays in upstream processing, the streaming step attempts to retrieve a video frame from the queue but there is no video frame data in the queue, it will wait for one video frame to be added to the queue before immediately retrieving that video frame and streaming it (at this time, the front-end player will experience some stuttering); then it will wait 33.3ms before attempting to retrieve the next frame.

[0074] If the upstream step has low throughput for a period of time, causing delays in the generation of some resulting video frames, and then resumes normal throughput to process the latest input video frames in real time, a backlog will form in the queue before the push stream step. Without additional processing, the backlog will only increase, not decrease, leading to a growing delay between the output video and the input video. Therefore, a speed-up playback strategy is adopted: when the number of video frames in the queue exceeds a set threshold (e.g., more than 10 frames), the waiting time between two frames during push stream is increased to 33.3ms. The set time interval is reduced, for example to 30ms, which corresponds to about 1.1 times the playback speed. This allows the front-end player to temporarily increase the frame rate and play at double speed to reduce backlogged data and reduce the latency of the system's output video stream. Thus, the algorithm's frame extraction processing can prevent the display of processing results from being delayed due to excessive data volume. The method of double-speed playback when data is backlogged ensures the synchronization of video display when the baggage box passes through the X-ray machine (security inspection machine). The combination of the two can improve the real-time processing of video frames in the video stream and target detection, thereby adapting to the 24 / 7 uninterrupted use at the airport.

[0075] In this embodiment, different processing steps (such as object detection algorithm recognition, OCR algorithm recognition, and streaming processing steps) are executed using different separate threads. After each step is completed, the result data is placed in its output queue for the input queue of the next processing step to read. Compared with single-threaded execution, this can improve the parallelism of the program, make full use of CPU and GPU resources, increase the throughput of data processing, and improve processing efficiency. Furthermore, this embodiment also uses a producer-consumer pattern to decouple different steps in scheduling and execution, which is beneficial for realizing the above-mentioned "algorithm frame extraction processing" and "speed playback when data is backlogged".

[0076] Specifically, the "producer-consumer pattern" is a model and a method for parallel processing. It separates task execution from task scheduling, reducing coupling between producers and consumers. Specifically, after a producer generates data, it puts the data into a queue; the consumer attempts to retrieve data from the queue and then executes its task. If the existing data in the queue is consumed before the producer has had time to generate new data, the queue is empty, and the consumer will block and wait (using a blocking queue achieves blocking waiting without polling or spinning) until new data enters the queue, at which point the consumer immediately retrieves the data and executes its task. If the queue is full, the producer will be blocked when inserting data into the queue until the queue is no longer full. The "producer-consumer pattern" decouples producers and consumers, allowing each to focus on their own task processing without the producer needing to worry about calling the consumer. It also enables low-cost collaboration between the two types of tasks, allowing one to wait for the other when the throughput of the two tasks is mismatched, and smoothing out peaks and valleys when task throughput fluctuates. Furthermore, the use of blocking queues ensures that blocking waits do not consume CPU resources, resulting in high efficiency and avoiding waste.

[0077] In this real-time example, the producer-consumer pattern can be used between object detection algorithm recognition and OCR algorithm recognition. For instance, object detection algorithm recognition acts as the producer, and OCR algorithm recognition acts as the consumer. OCR algorithm recognition relies on the "position of the luggage frame number (such as the position of the metal tag)" output by object detection algorithm recognition to perform OCR algorithm recognition. There is only one producer and one consumer thread. When the throughput of the object detection algorithm recognition step is low, or when the OCR algorithm recognition step has no data to process, it will block while retrieving data from the queue, implementing a blocking wait. When the object detection algorithm recognition step outputs data, the OCR algorithm recognition step can execute immediately. If the throughput of the OCR algorithm recognition step is low for a period of time, the tasks to be processed will be temporarily stored in the blocking queue. The operation of the object detection algorithm recognition step will not be affected when the queue is not full (the queue length is set very large here, and the queue is almost never full).

[0078] Security Inspection and Identification System Implementation Examples

[0079] This embodiment provides a technical solution for a security inspection and identification system. The security inspection and identification system includes a processor, which executes program instructions to implement the security inspection and identification method as described in the above embodiment.

[0080] Since the specific working method and principle of the security inspection and identification system have been described in detail in the above-mentioned security inspection and identification method embodiments, they will not be repeated here.

[0081] This invention has the following characteristics:

[0082] 1) Only a portion of video frames are extracted from the video stream for corresponding processing and target detection to obtain recognition results. Based on the conveyor belt's moving speed and the recognition results of the extracted video frames, the recognition results of the unextracted video frames are inferred. Thus, through frame extraction recognition and the inference of the remaining frames, only a small number of actual frames need to be recognized to obtain the recognition results of all video frames. This reduces the requirements on algorithm processing efficiency and total processing volume when processing video frames and detecting targets in the video stream, while avoiding processing lag.

[0083] 2) In the streaming process, a video frame is taken from the processing result queue of the previous processing step at a set time interval and sent for real-time playback. If the number of video frames in the processing result queue exceeds the set threshold, the set time interval between two frames during streaming is reduced to play the video frames at double speed, ensuring the synchronization of the video frames after processing and target detection when the luggage box passes through the security inspection machine.

[0084] 3) The characters in the obtained luggage frame number image are split and recombined to generate new, different numbered text images that do not conform to the check code rules corresponding to the luggage frame number. The generated new numbered text images are added to the training set of the OCR algorithm recognition model to train the OCR algorithm recognition model. The trained OCR algorithm recognition model is then used to perform the OCR algorithm recognition processing steps to identify the luggage frame number in the video frame. This solves the problem of OCR recognition recording the result as a "fixed template", thereby enhancing the robustness and accuracy of the algorithm.

[0085] It should be understood that the above-described specific embodiments of the present invention are merely illustrative or explanatory of the principles of the present invention, and do not constitute a limitation thereof.

Claims

1. A security inspection and identification method, characterized in that, Includes the following steps: 1) Real-time acquisition of the video stream corresponding to the security check, processing and target detection of the video frames in the video stream, including target detection algorithm recognition for determining the position of the luggage rack and the number on the luggage rack, OCR algorithm recognition for recognizing the luggage rack number, and streaming processing steps; streaming refers to sending video frames for real-time playback; obtaining the corresponding recognition results; the recognition results include the recognized luggage rack position and number in the video frame; The processed and target-detected video frames are played in real time, and the recognition results corresponding to the current video frame are obtained. The luggage frame in the current video frame is displayed in the form of a first-class graphic. Select the luggage frame in the current video frame in the first type of graphic form using the mouse cursor, and then perform a security check on the luggage frame; If the result is suspicious, the suspicious area of ​​the luggage frame is determined based on the coordinates of the mouse cursor. Luggage frames that have not undergone security checks are displayed in the first type of graphic form, luggage frames that are deemed safe are displayed in the second type of graphic form, and luggage frames that are deemed suspicious are displayed in the third type of graphic form, so as to obtain the luggage frame security check results; 2) Obtain the operation information corresponding to the security check, including the coordinates of the mouse cursor, the judgment result, the operation time, and the video frame ID; Based on the coordinates of the mouse cursor corresponding to the operation and the video frame ID, the recognition result corresponding to the video frame is obtained, thereby determining which luggage compartment the mouse cursor is located in. If it is determined that the mouse cursor is not in the area corresponding to any luggage compartment, an error is determined. If it is determined that the mouse cursor is in the area corresponding to a luggage compartment, the number of that luggage compartment is obtained. Convert the video frame corresponding to the video frame ID into an image, and if the luggage frame is deemed suspicious, mark the suspicious area of ​​the luggage frame in the image corresponding to the video frame. The information is assembled based on the luggage frame number, the luggage frame security inspection result, the coordinates of the suspicious area of ​​the luggage frame when the result is suspicious, the video frame ID, the image corresponding to the video frame, and the operation time to obtain the security inspection and identification result information of the luggage frame in the video frame. The method for processing video frames in the video stream includes: extracting video frames from the video stream at set intervals, performing corresponding processing and target detection on the extracted video frames to obtain recognition results; and calculating the recognition results of the unextracted video frames based on the moving speed of the conveyor belt and the recognition results of the extracted video frames.

2. The security inspection and identification method according to claim 1, characterized in that, Based on the conveyor belt's moving speed and the identification results of the extracted video frames, the methods for inferring the identification results of the unextracted video frames include: Obtain the recognition result of the extracted video frame before the unextracted video frame, translate the luggage frame position in the recognition result of the extracted video frame by a corresponding translation distance in the direction of the conveyor belt movement, and use the translation result as the recognition result of the unextracted video frame. The corresponding translation distance is obtained by the moving speed of the conveyor belt.

3. The security inspection and identification method according to claim 1 or 2, characterized in that, Different processing steps are executed using separate threads.

4. The security inspection and identification method according to claim 1 or 2, characterized in that, The streaming process retrieves a video frame from the processing result queue of the previous processing step at a set time interval and sends it for real-time playback. If the number of video frames in the processing result queue exceeds a set threshold, the set time interval between two frames during streaming is reduced to allow for faster playback of the video frames.

5. The security inspection and identification method according to claim 4, characterized in that, If there are no video frame data in the processing result queue after the set time interval, the video frame will be retrieved and streamed immediately after it is added to the queue.

6. The security inspection and identification method according to claim 1 or 2, characterized in that, The luggage frame number includes a check code; the characters in the obtained luggage frame number image are split and recombined to generate a new, different number text image that does not conform to the check code rules corresponding to the luggage frame number. The generated new number text image is added to the training set of the OCR algorithm recognition model to train the OCR algorithm recognition model. The trained OCR algorithm recognition model is used to perform the OCR algorithm recognition processing steps to identify the luggage frame number in the video frame.

7. The security inspection and identification method according to claim 1 or 2, characterized in that, In the target detection algorithm recognition process, the ASI-YOLOV5 algorithm, which is designed for airport security scenarios, is used to implement the target detection algorithm recognition processing steps.

8. The security inspection and identification method according to claim 1 or 2, characterized in that, In the OCR algorithm recognition process, the OCR algorithm used is the MMT-OCR algorithm for metal tag text recognition, which is designed for airport security scenarios.

9. The security inspection and identification method according to claim 1 or 2, characterized in that, If, during the security check of a baggage box in a video frame, the baggage box number in the video frame is detected to have appeared within a set time, then the screen is rewound, and the graphic form corresponding to the baggage box is displayed based on the previous security check result of the baggage box number. If the luggage frame is subjected to security inspection again, the latest inspection result will be used as the security inspection result for the luggage frame.

10. A security inspection and identification system, characterized in that, Includes a processor, which executes program instructions to implement the security inspection and identification method as described in any one of claims 1-9.

Images