Multi-modal human body safety inspection method and device

Abstract

The invention discloses a multi-modal human body security check method and device, and solves the problem of high leak detection risk of single-dimensional security check. A multi-modal human body safety inspection method comprises the following steps: acquiring a human body dangerous article detection result of a to-be-detected person, and scoring different regions of a human body; acquiring state information of the to-be-tested person, and identifying whether the state information is abnormal; determining a first set threshold, and adjusting the first set threshold according to the abnormity of the state information to obtain a second set threshold; and comparing the score with the second set threshold value, and marking an area of which the score is not less than the second set threshold value. According to the invention, defects of millimeter wave image-based single-mode detection are made up, and more clues and information are provided for dangerous article detection and decision making, so that the risk of missing detection of dangerous articles is further reduced, and the accuracy and reliability of a security check system are improved.

Description

Technical Field

[0001] The present application relates to the technical field of human body safety inspection equipment, and in particular to a multimodal human body safety inspection method and device. Background Art

[0002] Millimeter waves are invisible light waves with a frequency range of 30GHz to 300GHz. With their unique penetrating ability, they have become an important tool in the field of modern human body security inspections. Existing millimeter-wave-based human body security inspection equipment uses the characteristics of millimeter waves that can penetrate common clothing materials to detect dangerous items hidden under clothing. This technology is not only suitable for dangerous items made of metal, but also for dangerous items made of non-metallic materials. At the same time, since millimeter waves cannot penetrate human skin and are safe and harmless to the human body, dangerous goods detection equipment based on millimeter-wave human body imaging is widely favored by the human body security inspection market and has been successfully used in airports, high-speed rail stations and other places. However, due to the problems that the placement and angle of some dangerous items in the human body are too hidden or the size of the dangerous items is too small, there is still a certain risk of missed detection when simply using millimeter-wave images for human body security inspections, which leads to public safety hazards. Summary of the invention

[0003] The embodiments of the present application provide a multi-modal human body security inspection method and device, which solves the problem of high risk of missed inspections in single-dimensional security inspections.

[0004] In a first aspect, an embodiment of the present application provides a multimodal human body security inspection method, comprising the steps of: Obtain the human body dangerous goods detection results of the person to be tested, and score different areas of the human body; the score is proportional to the possibility of the presence of dangerous goods in the area; Acquire status information of the person to be tested, and identify whether the status information is abnormal; the abnormality is identified by a training model; Determine a first set threshold value, and adjust the first set threshold value according to the abnormality of the state information to obtain a second set threshold value; The score is compared with the second set threshold, and regions having a score not less than the second set threshold are marked.

[0005] In one of the embodiments, in response to the existence of an abnormality in the status information, the set threshold is lowered; in response to the existence of no abnormality in the status information, the set threshold is increased.

[0006] In one embodiment, the state information includes expression, voice and behavior; The data used to train the model include: static images, videos, and audio; The static image is used for training expression recognition; The video is used for training behavior recognition; The audio is used to train speech recognition.

[0007] In one of the embodiments, a human body image marked by a threshold filtering module is displayed; Regions marked by the threshold filtering module highlight expression.

[0008] In one embodiment, the negative index adjustment strategy is used to adjust the set threshold downward, which specifically includes the steps of: Determine the number of abnormal conditions and the magnitude of downward adjustments; The product of the number of abnormal states and the downward adjustment amplitude of a single abnormality is used as the input of the negative exponential function to calculate the scale of the downward adjustment of the first set threshold, thereby determining the second set threshold.

[0009] In one of the embodiments, a linear adjustment strategy is used to adjust the set threshold upward.

[0010] In the second aspect, the embodiments of the present application also provide a multimodal human body safety inspection device, which is used to implement the multimodal human body safety inspection method described in any one of the embodiments of the first aspect, and includes: a preliminary detection module for dangerous goods, a threshold filtering module, an acquisition module, and an abnormality recognition module. The dangerous goods detection module is used to perform dangerous goods detection on the person to be tested and score different areas of the human body; the score is proportional to the possibility of the presence of dangerous goods in the area. The threshold filtering module is used to mark the area where the score exceeds the set threshold. The acquisition module is used to collect the status information of the person to be tested and send it to the abnormality recognition module. The abnormality recognition module is used to identify the status information of the person to be tested and send instructions to the threshold filtering module to adjust the set threshold of the person to be tested.

[0011] Furthermore, the acquisition module includes a visible light acquisition unit and a voice acquisition unit. The visible light acquisition unit is used to acquire the expression and behavior of the person to be tested. The voice acquisition unit is used to acquire the voice of the person to be tested.

[0012] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the first aspect.

[0013] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in any embodiment of the first aspect is implemented.

[0014] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: This application makes up for the shortcomings of single-mode detection based on millimeter-wave images, provides more clues and information for dangerous goods detection and decision-making, thereby further reducing the risk of missed detection of dangerous goods and improving the accuracy and reliability of the security inspection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flow chart of a multimodal human body safety inspection method is provided for an embodiment of the present application; Figure 2 Provide a multimodal model training and deployment process for the embodiments of the present application; Figure 3 A structural diagram of a multimodal human body safety inspection device is also provided for an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0017] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0018] Figure 1 A multi-modal human body safety inspection method flow chart is provided for the embodiment of the present application, comprising the steps of: Step 110, obtaining the human body dangerous goods detection result of the person to be tested, and scoring different areas of the human body; the score is proportional to the possibility of the presence of dangerous goods in the area; The commonly used dangerous goods inspection process is completed through the millimeter wave human body imaging security inspection equipment. The human body dangerous goods detection results of the person to be tested are obtained, and the human body dangerous goods detection results divide the human body into several areas; The regions may be distinguished according to the structural parts of the human body, for example, the head, forearms, upper arms, chest trunk, abdominal trunk, hip trunk, calves, thighs, hands and feet, etc.

[0019] Furthermore, the regions can be used to make detailed distinctions between structures. For example, the forearm can be divided into the medial and lateral sides.

[0020] Preferably, the body of the person to be tested is subdivided into grids, each grid being a region, and the granularity of the grid is smaller than the structural parts of the human body.

[0021] When collecting data, the grid where the person to be tested hides the items is used as an area, and emphasis is placed on placing the items in different locations to ensure that the simulation area is relatively comprehensive.

[0022] It can also be divided according to the shape of the bulge under the clothes of the person to be tested. For example, if the left trouser pocket of the person to be tested has a bulge, it can be divided into a separate part.

[0023] Millimeter wave human images are used to identify dangerous items that may be carried by the human body. When collecting millimeter wave human images, models of different body shapes are needed to adapt to people of different body shapes. For items carried with you, dangerous items of different sizes and types are used, such as knives, guns, explosives, etc. of different sizes that can be hidden. For the location of the carried items, different parts of the human body are traversed, such as the upper and lower arms, chest, abdomen, back, side of the body, upper and lower legs, etc.

[0024] The input of the dangerous goods detection model is a two-dimensional millimeter wave image. Its function is to locate and classify dangerous goods in the millimeter wave image. The model performs a typical target detection task. Common target detection models can be used as reference, such as the detection models of the Faster RCNN series, YOLO series, and DETR series.

[0025] Different areas are scored using the dangerous goods detection model. The greater the probability that dangerous goods may exist in the area, the higher the score.

[0026] Step 120, obtaining the status information of the person to be tested, and identifying whether the status information is abnormal; the abnormality is identified by the training model; The model is trained based on the acquired status information of the person to be tested, and the trained model is used to identify whether the status information is abnormal.

[0027] In one embodiment, the state information includes expression, voice and behavior; The data used to train the model include: static images, videos, and audio; like Figure 2 As shown, in addition to the antenna array for millimeter wave data collection, a multi-modal perception hardware system is built in the security gate by installing multiple types of sensors, cameras, speakers, microphones and other related hardware and drivers.

[0028] The camera is used to collect frontal images and behavioral videos of the detected persons for expression recognition and behavior analysis.

[0029] The loudspeaker and microphone are used for pre-made voice playback and voice data collection respectively.

[0030] This application uses multimodal training data, including static images for expression recognition, videos for behavior recognition, voice for speech recognition, and millimeter wave images for dangerous goods detection. It should be noted that the process of training data in the present application occurs before using the model to identify state information, for example, before starting hazardous material detection, and is not further limited here.

[0031] The data used are introduced below one by one: The static image is used for training expression recognition; The expression recognition model is trained with a static image, wherein the static image is a frontal image of a human body.

[0032] For example, since the expression recognition task of the security gate system has commonalities with the general expression recognition task, the static image data for training the expression recognition model can use a public expression recognition database as part of the training data, such as the JAFFE dataset, the RAF (Real-world Affective Faces) dataset, the EmotioNet dataset, etc.

[0033] For security inspection gates, some facial expression-related human frontal images can be collected and annotated to expand the sample. For facial expression types, common classification methods can be used, such as happiness, anger, sadness, surprise, neutrality, disgust, etc.

[0034] Based on the expression recognition model, the system can distinguish the current expression type of the person being tested.

[0035] For example, expression types can be divided into positive expressions and negative expressions, with positive expressions such as happiness and neutrality considered to be reasonable expressions, while negative expressions such as sadness, surprise, disgust, and anger are considered abnormal.

[0036] If the person under test is detected to show a negative expression, it is considered that an abnormality has occurred.

[0037] The number of abnormalities can also be increased according to the length of time a negative expression is maintained. For example, a time threshold is set. If the person under test maintains a negative expression for more than the set time threshold, it is considered that one additional abnormality has occurred. If the person maintains a negative expression for more than twice the set time threshold, it is considered that two additional abnormalities have occurred, and so on.

[0038] It should be noted that the frontal photo of the person to be tested taken by the camera contains other parts of the human body and the background in addition to the face, and has not been cropped. Therefore, the expression recognition here is actually a composite task of face detection and expression classification based on the face. The face target detection scheme can be used to find the target. The face target detection scheme is used to locate the face and classify the target face. Commonly used convolutional neural networks based on deep learning can be selected as expression recognition models, such as the detection networks of the Faster RCNN series, YOLO series, and DETR series.

[0039] The video is used for training behavior recognition; The behavior recognition model is trained with videos of people's behavior at the security gate.

[0040] For example, the behavior recognition model collects and annotates video data by taking the person to be tested entering the security gate as the starting point of video collection and the person to be tested performing millimeter wave antenna array collection as the end point of video collection. Training is used to identify common abnormal behavior manifestations. For example, 1) frequent hand movements, such as clenching fists or shaking them constantly; 2) muscle tension and body stiffness; 3) frequent licking of lips and swallowing of saliva; 4) frequent hand movements, such as clenching fists or shaking them constantly; 5) uncoordinated or too fast steps when walking, and unbalanced or swaying body when walking.

[0041] For another example, the input of the behavior recognition model is a video, and the model needs to determine whether there is any abnormality in the behavior of the person being detected in the video.

[0042] It should be noted that videos are different from static images. In addition to the spatial information of images, they also contain rich temporal information. In view of this feature of videos, convolutional networks and long short-term memory networks can be used in combination, using convolutional neural networks (CNN) and long short-term memory (LSTM) networks to process spatial information and time series information respectively, so as to perform behavior recognition. Alternatively, three-dimensional convolution operators can be used for spatiotemporal information perception. The behavior recognition models based on three-dimensional convolution operators that can be used include C3D models, I3D models, etc.

[0043] The audio is used to train speech recognition; The speech recognition model is trained with audio recordings of people speaking at security gates.

[0044] The speech recognition model consists of two parts: speech feature extraction and speech feature classification. In order to simplify the process of feature extraction and classifier training and make the speech model more generalizable, a deep feature extractor based on a neural network can be used to train the classifier end-to-end. In view of the time series characteristics of speech, commonly used deep neural network structures include recurrent neural networks (RNNs), long short-term memory networks, etc.

[0045] Voice data is used for emotion analysis of the test subjects. Since there are many public data sets for emotion analysis based on voice, public data sets can be used as part of the training and verification data. When collecting voice data, different genders and age groups should be included. Different languages ​​should be collected according to the needs of the application scenario.

[0046] The speech recognition model can identify whether the person being tested is nervous based on the audio.

[0047] For example, a question-and-answer mechanism is used for voice collection, with questions set in advance, such as "Do you carry any illegal items?" and "Do you feel nervous or uncomfortable?" The emotions of the person being tested are judged based on their answers, and those who are nervous are considered abnormal.

[0048] After the training model is completed, the model is output and deployed: Convert each model into a portable format, such as the pt format under the Pytorch framework, the cross-framework ONNX format, and the pb format under the Tensorflow framework, and then deploy it to the recognition system for reasoning.

[0049] Step 130: determine a first set threshold, and adjust the first set threshold according to the abnormality of the state information to obtain a second set threshold; In one of the embodiments, in response to the existence of an abnormality in the status information, the set threshold is lowered; in response to the existence of no abnormality in the status information, the set threshold is increased.

[0050] The set threshold directly affects the detection result of dangerous objects in the millimeter wave image.

[0051] In one embodiment, determining the first set threshold and training the object detection model is also performed in the following manner: Figure 2 The model shown is performed during training.

[0052] In one embodiment, if Figure 2 As shown, before the model is output and deployed, it is necessary to determine the set threshold by detecting the threshold control parameters.

[0053] For example, when no dangerous goods are detected based on millimeter wave images, the threshold set for the main road can be lowered according to the degree of abnormal results in expression recognition, behavior recognition, and emotion recognition, increasing the probability of dangerous goods being detected, thereby reducing the risk of missed detection. When dangerous goods are detected based on millimeter wave images, if the results of expression recognition, behavior recognition, and emotion recognition are normal, the threshold set can also be appropriately increased to reduce the false alarm rate of dangerous goods and ensure the accuracy of detection.

[0054] In this embodiment, the human body dangerous goods detection result is compared with the first set threshold first, and the part larger than the first set threshold is considered to be risky. If a millimeter wave image is compared with the first set threshold and at least one part exceeding the first set threshold is found, it is considered that the millimeter wave image has detected dangerous goods. Otherwise, it is considered that the millimeter wave image has not detected dangerous goods.

[0055] It should be noted that the first threshold of the present application may have been determined before step 110 and is a basic threshold, which is subsequently adjusted according to the status information.

[0056] Step 140: Compare the score with the second set threshold, and mark the parts whose scores are not less than the second set threshold.

[0057] In one of the embodiments, a human body image marked by a threshold filtering module is displayed; The sites marked by the threshold filtering module are highlighted.

[0058] For example, at the data level, parts with scores not less than a second set threshold are marked, and at the actual level, there is a display screen on the host computer that can display the detection results of the millimeter wave image and frame the marked parts with a circular or square frame.

[0059] Adjust the set threshold according to the abnormality. The specific process is as follows: In one embodiment, downward adjustment of the set threshold adopts an exponential adjustment strategy, which specifically includes the steps of: Determine the number of abnormal conditions and the magnitude of downward adjustments; The product of the number of abnormal states and the downward adjustment amplitude of a single abnormality is used as the input of the negative exponential function to calculate the scale of the downward adjustment of the first set threshold, thereby determining the second set threshold.

[0060] To control the rate of decline, an exponential decline strategy can be used to adjust the threshold downward.

[0061] For example, suppose the threshold is adjusted downward by , the number of abnormal identifications in the visible light branch and the voice branch is used as the threshold downward adjustment degree, and the number of abnormalities is set to , whose value is between 0 and 3, and the downward adjustment range is , adjust the threshold to the original value times.

[0062] It should be noted that the value of the number of anomalies k is between 0 and 3, which means that in the collection and identification of three different signals in this application, each abnormality is counted as one abnormality, that is, no matter how many abnormalities there are in one path, it is counted as one abnormality. In this way, the abnormality level is divided into four levels, namely, no abnormality, one abnormality, two abnormalities and three abnormalities.

[0063] In one of the embodiments, a linear adjustment strategy is used to adjust the set threshold upward.

[0064] For example, when the trunk road detects dangerous goods and the two branches do not find any abnormalities, the threshold can be adjusted upward. The upward adjustment range is , adjust the second threshold to (1+ ) times.

[0065] Alternatively, the first threshold + as the second threshold.

[0066] For another example, when a dangerous object is detected on the main road and no abnormality is found on the two branches, the threshold remains unchanged. When the facial expression recognition detects that the person has a positive attitude, the set threshold is adjusted upward.

[0067] For another example, when the number of persons to be tested is greater than the number threshold, in order to improve the detection efficiency, the set threshold can be adjusted upward.

[0068] The threshold value may be adjusted upward or downward according to specific work requirements and is not further limited here.

[0069] Figure 3 The present application also provides a multimodal human body security inspection device structure diagram for implementing the multimodal human body security inspection method described in any one of the embodiments of the first aspect, comprising: Dangerous goods preliminary detection module 210, threshold filtering module 220, collection module 230 and abnormality identification module 240.

[0070] The dangerous goods detection module is used to perform dangerous goods detection on the person to be tested and score different parts of the human body; the score is proportional to the possibility of the presence of dangerous goods in the part.

[0071] For example, the device described in the present application includes three groups of different signal acquisition and recognition modules, which specifically include a main channel based on millimeter wave images, a visible light branch based on visible light images and videos, and a voice branch based on voice.

[0072] The millimeter wave-based trunk route is a dangerous goods inspection process commonly used by existing millimeter wave human imaging security inspection equipment, which includes a millimeter wave signal acquisition and imaging module 200, a dangerous goods preliminary detection module and a threshold filtering module, and ultimately outputs the detection results of dangerous goods that may be hidden in the human body.

[0073] The threshold filtering module is used to mark the parts where the scores exceed the set threshold.

[0074] The acquisition module is used to collect the status information of the person to be tested and send it to the abnormality recognition module.

[0075] Furthermore, the acquisition module includes a visible light acquisition unit 230-1 and a voice acquisition unit 230-2. The visible light acquisition unit is used to acquire the expression and behavior of the person to be tested. The voice acquisition unit is used to acquire the voice of the person to be tested.

[0076] For example, the millimeter wave signal acquisition and imaging module specifically includes an antenna array for millimeter wave data acquisition, and the acquisition module specifically includes related hardware and drivers such as cameras, speakers, microphones, etc., to build a multimodal perception hardware system in the security gate. The camera is used to collect frontal images and behavioral videos of the inspected person for expression recognition and behavior analysis. The speaker and microphone are used for pre-made voice playback and voice data acquisition respectively.

[0077] The abnormality identification module is used to identify the status information of the person to be tested and send instructions to the threshold filtering module to adjust the set threshold of the person to be tested.

[0078] Furthermore, the abnormality recognition module is provided with a visible light abnormality recognition unit and a voice abnormality recognition unit corresponding to the visible light collection unit and the voice collection unit respectively.

[0079] For example, the visible light branch includes a visible light collection unit and a visible light anomaly recognition unit 240-1. The voice branch includes a voice collection unit and a voice anomaly recognition unit 240-2. The visible light branch and the voice branch are used to improve the detection results of dangerous goods in the millimeter wave-based trunk road, and provide multi-dimensional human safety inspection indicators to make human safety inspection more reliable and comprehensive.

[0080] The visible light branch mainly uses the collected visible light images and videos to perform facial recognition and behavior analysis on the detected person, and to determine whether there are abnormal expressions or behaviors. When the detected person enters the security gate, if he / she carries dangerous items, he / she is likely to have an overly nervous expression or abnormal behavior, such as frequent hand movements, uncoordinated or fast steps when walking. Through the detailed analysis of these expressions and behaviors, rich information can be mined to assist human safety inspection.

[0081] The voice branch mainly analyzes the emotional state of the person being tested from the voice aspect. When collecting voice information, a question-and-answer mode is used to ask simple pre-set questions for the person being tested to answer. Emotion recognition is performed from the voice of the person being tested's answers, such as using tone, volume, speaking speed, pauses, etc. to evaluate the emotions of the person being tested. If the person being tested is found to be too nervous or panicky, it can be used as one of the bases for further investigation. In addition, the pre-set questions raised by the voice branch increase the possibility of abnormal expressions or behaviors of the person being tested who is carrying dangerous goods, thereby increasing the chances of the visible light branch capturing abnormal performance of the person being tested and enhancing the overall detection capability of the system.

[0082] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] Therefore, the present application also proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any embodiment of the present application is implemented.

[0084] Furthermore, the present application also proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any embodiment of the present application when executing the computer program.

[0085] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0086] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0088] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0089] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0090] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A multimodal human body security inspection method, characterized in that: Contains steps: Obtain the human body dangerous goods detection results of the person to be tested, and score different areas of the human body; the score is proportional to the possibility of the presence of dangerous goods in the area; Obtain the status information of the person to be tested and identify whether there is any abnormality in the status information; The anomaly is identified by a training model; Determine a first set threshold value, and adjust the first set threshold value according to the abnormality of the state information to obtain a second set threshold value; The score is compared with the second set threshold, and regions having a score not less than the second set threshold are marked.

2. The multimodal human body security inspection method according to claim 1, characterized in that: In response to the state information being abnormal, the set threshold is lowered; in response to the state information not being abnormal, the set threshold is increased.

3. The multimodal human body security inspection method according to claim 1, characterized in that: The state information includes expressions, voices and behaviors; The data used to train the model include: static images, videos, and audio; The static image is used for training expression recognition; The video is used for training behavior recognition; The audio is used to train speech recognition.

4. The multimodal human body security inspection method according to claim 1, characterized in that: Display the human body image after being marked by the threshold filtering module; Regions marked by the threshold filtering module highlight expression.

5. The multimodal human body security inspection method according to claim 1, characterized in that: The negative exponential adjustment strategy is used to adjust the threshold downward, which specifically includes the following steps: Determine the number of abnormal conditions and the magnitude of downward adjustments; The product of the number of abnormal states and the downward adjustment amplitude of a single abnormality is used as the input of the negative exponential function to calculate the scale of the downward adjustment of the first set threshold, thereby determining the second set threshold.

6. The multimodal human body security inspection method according to claim 1, characterized in that: A linear adjustment strategy is used to adjust the threshold upward.

7. A multimodal human body security inspection device, used to implement the multimodal human body security inspection method according to any one of claims 1 to 6, characterized in that: Contains: Dangerous goods preliminary detection module, threshold filtering module, acquisition module and anomaly recognition module; The dangerous goods detection module is used to detect dangerous goods on the person to be tested and score different areas of the human body; the score is proportional to the possibility of the presence of dangerous goods in the area; The threshold filtering module is used to mark the area where the score exceeds a set threshold; The acquisition module is used to collect the status information of the person to be tested and send it to the abnormality recognition module; The abnormality identification module is used to identify the status information of the person to be tested and send instructions to the threshold filtering module to adjust the set threshold of the person to be tested.

8. The multi-modal human body security inspection device according to claim 1, characterized in that: The acquisition module includes a visible light acquisition unit and a voice acquisition unit; The visible light collection unit is used to collect the expression and behavior of the person to be tested; The voice collection unit is used to collect the voice of the person to be tested.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

Images