A method and device for detecting a cabin emergency exit attendant safety introduction specification

Abstract

The present application belongs to the technical field of audio and human posture recognition, and particularly relates to a cabin emergency exit attendant safety introduction specification detection method and device, which is based on an attendant specification manual to determine the behavior specification specific rules and standard safety introduction terms of the safety introduction of the attendant to the passengers at the emergency exit, to perform safety introduction voice keyword detection, human recognition and posture detection of the attendant and the passengers on the audio and video data collected at the emergency exit, and finally based on the rules at the emergency exit, to determine whether the safety introduction and guidance of the attendant conforms to the specification by using the context-related grammar in the syntax model on the recognition and detection results. The present application can verify whether the safety guidance of the attendant is in specification in the cabin of the airplane, reduces the loss and flight delay caused by the opening of the emergency exit due to the knowledge blind area of the passengers, and the specification operation at the necessary time can improve the survival rate of the cabin personnel, and provides strong support for the safety work of the airplane.

Description

Technical Field

[0001] This invention belongs to the field of audio and human posture recognition technology, and in particular relates to a method and device for detecting the safety introduction procedures for cabin emergency exit flight attendants. Background Technology

[0002] Emergency exits in civil aircraft cabins can be accidentally opened by passengers due to a lack of knowledge, which can disrupt flight order and endanger flight safety. Inadequate safety briefings by flight attendants may also lead to passengers in emergency exit seats not understanding the instructions, resulting in safety hazards.

[0003] However, with the continuous advancement of technologies such as deep learning, machine vision, audio and image processing, and syntactic pattern recognition, based on emergency exit rules, these key technologies can be used to infer the standardization of flight attendants' safety briefings for passengers in emergency exit seats, and ultimately solve the above problems. Summary of the Invention

[0004] To address the aforementioned technical problems in the existing technology, this invention proposes a method and device for detecting cabin emergency exit crew safety briefing procedures, the specific technical solution of which is as follows:

[0005] A method for testing a cabin emergency exit safety briefing procedure for flight attendants includes the following steps:

[0006] Step 1: According to the flight attendant manual, determine the specific rules and standard safety introduction terminology for flight attendants at the emergency exits when introducing safety to passengers. Install a high-definition monitoring camera with an audio sensor on the top of the aircraft in the middle of the emergency exit to collect audio and video data in real time after passengers board and before takeoff.

[0007] Step 2: Preprocess the collected audio and video data to form a dataset of audio and video recordings of flight attendants giving safety briefings to passengers at the emergency exits of the aircraft;

[0008] Step 3: Train the audio dataset collected multiple times at the emergency exit of the aircraft to obtain a safety introduction voice keyword detection algorithm that meets the recognition rate requirements;

[0009] Step 4: The collected video dataset is uniformly sampled using a method of extracting one frame per second to obtain a total of T frames. The image data is used for training to obtain a human body recognition and posture detection algorithm for flight attendants and passengers at emergency exits that meets the recognition rate requirements.

[0010] Step 5: Using the safety introduction voice keyword detection algorithm from Step 3 and the human body recognition and posture detection algorithm from Step 4, audio text information is identified at the emergency exits in the aircraft cabin, and the posture information of flight attendants and passengers is detected.

[0011] Step 6: Based on the rules at the emergency exit, the identification and detection results from Step 5 are used to infer, using context-sensitive grammar in the syntactic model, whether the entire process of the flight attendant's safety introduction and guidance complies with the regulations.

[0012] Furthermore, step 2 specifically involves: dividing the video dataset into video categories with and without safety introductions, cleaning and labeling the video category data to form a training, verification, and testing dataset for flight attendants' safety introduction videos; processing standard safety introduction terms through a language model to generate safety introduction keyword reference templates, and then cleaning and labeling the collected audio data to form a training, verification, and testing dataset for flight attendants' safety introduction voice.

[0013] Furthermore, step 3 specifically includes the following sub-steps:

[0014] Step 3.1: For the audio data at the emergency exit, extract the MFCC feature parameters: First, preprocess the audio signal, including pre-emphasis, windowing, and framing; then perform FFT transformation on each frame to convert the data from the time domain to the frequency domain, and calculate the spectral energy. Pass the calculated spectral energy spectrum of each frame through a Mel filter and calculate the energy in that Mel filter. The calculation formula is as follows: , 0≤m≤M, where i represents the i-th frame and k represents the k-th spectral line in the frequency domain; finally, by taking the logarithm of the energy in the Mel filter and calculating the DCT, the MFCC characteristic parameters can be obtained, expressed as: , where n is the spectral line after DCT;

[0015] Step 3.2: Process MFCC feature parameters using an acoustic model and pronunciation dictionary, convert the publicly available Chinese speech training database into a spectrogram to extract features, and use 1427 pinyin characters composed of tones and vowels as the modeling units of the acoustic model; process the pinyin sequence corresponding to the audio signal to generate a pinyin index sequence, and then use the spectrogram and the pinyin index sequence as input data to train the acoustic model;

[0016] Step 3.3: During the decoding process, the acoustic model processes the MFCC feature parameters to generate a pinyin index sequence. After finding the corresponding pinyin sequence in the pronunciation dictionary, the recognized text is output.

[0017] Step 3.4: When the identified text contains passenger information broadcast over the radio, the instruction to perform video detection at the emergency exit is initiated.

[0018] Step 3.5: Set the threshold for terminology keyword matching, and match the text identified by the audio at the emergency exit with the safety introduction keyword reference template.

[0019] Furthermore, step 4 specifically includes the following sub-steps:

[0020] Step 4.1: Using video collected from the emergency exit area of ​​the aircraft, divide the video dataset into training, validation, and test sets in a 6:2:2 ratio; then, extract T frames from the collected video dataset using a method of extracting one frame per second. I =[ [Data annotation of target bounding boxes and flight attendant attribute features and skeletons for flight attendants and passengers;]

[0021] Step 4.2: Design a flight attendant and passenger recognition and posture detection network;

[0022] Step 4.3: Label the training data of the flight attendants' uniforms, hairstyles, badges, scarves, and the human body frames and skeletons of the flight attendants and passengers, input them into the flight attendant and passenger recognition and posture detection network for training, and test them to obtain a human body recognition and posture detection algorithm for flight attendants and passengers at emergency exits that meets the recognition rate.

[0023] Step 4.4: When the speech recognition system detects the passenger information broadcast, the posture detection algorithm for the flight attendants and passengers at the emergency exit begins.

[0024] Furthermore, step 4.2 specifically includes the following sub-steps:

[0025] Step 4.2.1: The T-frame video frames obtained by uniform sampling are cropped into images with a resolution of 224×224 and transmitted forward in parallel to the crew member and passenger recognition and posture detection network.

[0026] Step 4.2.2: In the flight attendant and passenger identification network, the Faster-RCNN algorithm is used. T-frame images are processed by the CNN network to extract attribute features of the flight attendant's uniform, hairstyle, badge, and scarf, generating 16... The feature map of 16; In the RPN structure, a 3×3 window is used to scan the image to obtain candidate boxes. After the feature map and candidate boxes are input into the ROI pooling layer, the softmax is used to classify the human body in the image in the fully connected layer. The bounding box regression generates detection boxes with flight attendant and passenger identifiers. Then, the multi-scale features shared by the hourglass network are used to regress the human bounding boxes on the input image. The fusion of these two feature sources can better predict the human bounding boxes. After identifying the flight attendant and passenger, the video area is divided into left and right regions along the aisle center line, and the duration of the flight attendant's stay in the left and right regions is calculated.

[0027] Step 4.2.3: In the pose detection network, it is first necessary to distinguish between flight attendants and passengers. The bounding box regression parameters obtained from the human body recognition network and the T-frame images are input into the hourglass network. The generated human key point heatmap is represented as a third-order tensor (W×H×). M * ), M * This represents the total number of heatmaps, i.e., the key points of the human body. Each key point generates a heatmap, with W and H being the width and height of the heatmap, respectively. A total of 16 key points are extracted, including: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left waist, right waist, left knee, right knee, left ankle, and right ankle. Each individual in each frame receives 17 key point heatmaps, with a 2D pose size of 17×56×56. The key point heatmaps for each frame are extracted. M ; M The calculation formula is: , () represents the pixel coordinates. Represents the coordinates of the K key point; The confidence level is represented by the scale. Multi-scale supervision is added during the deconvolution process, which is reduced by 1 / 8, 1 / 4, and 1 / 2 respectively, and then enlarged by 8, 4, and 2 times respectively before being concatenated with the output of the deconvolution to obtain the human posture detection output. After detecting the posture of the flight attendant and the passenger, the angle between the flight attendant's arm skeleton and body skeleton and the tilt angle of the body skeleton are further detected.

[0028] Furthermore, step 5 specifically includes the following sub-steps:

[0029] Step 5.1: Using the safety introduction voice keyword detection algorithm from Step 3, output the recognized text in real time. If the text information of the passenger instructions broadcast in the cabin is recognized, then start the human body recognition and posture detection algorithm for the flight attendants and passengers at the emergency exit.

[0030] Step 5.2: Determine whether the video content output by the SOTFMAX layer of the human body recognition and posture detection algorithm for flight attendants and passengers at the emergency exit belongs to the category of flight attendants providing safety instructions to passengers, and set a confidence threshold of [value missing]. If the output softmax value is less than the threshold, the output video contains no safety briefing for passengers by the flight attendant; if the output softmax value is greater than the threshold, the syntax is used to infer whether the safety briefing for passengers by the flight attendant fully complies with the regulations.

[0031] Furthermore, step 6 specifically includes the following sub-steps:

[0032] Step 6.1: The following rules apply to emergency exits: (1) Passengers must be seated at both emergency exit positions; (2) Flight attendants must conduct safety briefings at designated locations; (3) Flight attendants must tilt their bodies at a certain angle and raise their arms during the safety briefing; (4) Each passenger at the emergency exit positions in the left and right areas must be briefed once; (5) The content of the flight attendant's safety briefing voice must conform to the regulations; Design a method to meet the above emergency exit rules: (1) Passengers must be detected seated at both emergency exit positions; (2) Flight attendants must be detected in the emergency exit area; (3) During the flight attendant's safety briefing, the overall tilt angle of the body frame at time t1 must be greater than α, and the angle between the arm and body at time t2 must be greater than β; (4) The flight attendant's dwell time in the left and right areas must be greater than time t3 respectively; (5) The matching rate between the keywords of the flight attendant's voice and the template must be greater than the threshold.

[0033] Step 6.2: Let the entire process of the flight attendant's safety introduction be grammar G, G = {Vn, Vt, P, S}, where Vn is a non-terminal symbol, represented by uppercase letters, and Vn = {A, B, C, D, E, F, G}, where A indicates the start of inferring whether the flight attendant's safety introduction and guidance actions are standardized; B indicates the start of inferring whether the flight attendant's safety introduction voice is standardized; C indicates that there are passengers seated at the emergency exit; D indicates that there is a flight attendant at the emergency exit; E indicates that during the flight attendant's safety introduction, at time t1, the overall tilt angle of the flight attendant's body skeleton is greater than α, and at time t2, the angle between the arm and body is greater than β, indicating that the flight attendant's posture is standardized; F indicates that the flight attendant frequently stops in the left and right areas respectively. The time is greater than t3, which conforms to the one-person-one-introduction rule; G indicates that the matching rate between the keywords in the flight attendant's voice and the template must be greater than the threshold, and the flight attendant's voice introduction is standardized; Vt is the terminator, Vt={c,d,e,f,g,T}, where c indicates that no passenger is seated at the emergency exit; d indicates that the flight attendant's position is abnormal; e indicates that the flight attendant's posture is not standard; f indicates that the flight attendant's safety introduction does not conform to the one-person-one-introduction rule; g indicates that the flight attendant's voice introduction is not standard; T indicates that all the above syntaxes conform to the rules, and the entire process of the flight attendant's safety introduction to passengers is standardized; except for T, all other terminators output "xx is not standard"; P is the production rule of the syntax; S is the start symbol, indicating the start of syntactic inference;

[0034] Step 6.3: Begin the syntactic inference process. In the inference process for the flight attendant's safety introduction and guidance, there are a total of 5 production rules, P = {SAc, SACd, SACDe, SACDEf, SACDEF}. Here, 1 indicates continuing the inference forward, and 0 indicates the appearance of a terminator. Therefore, production rule SAc: S→SA→SAc, indicates that no passenger is seated at the emergency exit, and outputs an alarm indicating no passenger is seated at the emergency exit. Production rule SACd: S→SA→SAC→SACd, indicates that the flight attendant's position is abnormal, and outputs an alarm indicating abnormal flight attendant position. Normal alarm; Generation formula SACDe: S→SA→SAC→SACD→SACDe, indicating that the flight attendant's posture is not in accordance with the standard, outputting a flight attendant posture abnormality alarm; Generation formula SACDEf: S→SA→SAC→SACD→SACDE→SACDEf, indicating that the flight attendant's safety introduction does not conform to the one-person-one-introduction rule, outputting a flight attendant does not conform to the one-person-one-introduction rule alarm; Generation formula SACDEF: S→SA→SAC→SACD→SACDE→SACDEF, indicating that the flight attendant's body and movements during the passenger safety introduction are all in accordance with the standard;

[0035] Step 6.4: In the flight attendant safety introduction voice standard inference process, there are a total of 2 production rules, P={SBg, SBG}; Production rule SBg: S→SB→SBg, which means that the flight attendant's voice introduction is not standard and outputs a flight attendant voice introduction not standard alarm; Production rule SBG: S→SB→SBG, which means that the flight attendant's voice introduction is standard.

[0036] Step 6.5: After completing the deduction process of the flight attendant's safety introduction and guidance action and voice specifications, if both specifications are met, the production formula is P={SABCDEFT}; Production formula SABCDEFT: S→SA→SAB→SABC→SABCD→SABCDE→SABCDEF→SABCDEFT, indicating that the emergency exit flight attendant safety introduction process is fully compliant.

[0037] A cabin emergency exit crew safety briefing procedure testing device includes one or more processors for implementing the cabin emergency exit crew safety briefing procedure testing method.

[0038] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned method for testing cabin emergency exit crew safety briefing procedures.

[0039] Beneficial effects:

[0040] This method can verify whether flight attendants' safety guidance is standardized within the aircraft cabin, reducing losses and flight delays caused by passengers accidentally opening emergency exits due to a lack of knowledge. Standardized operation when necessary can improve the survival rate of passengers on board, providing strong support for aircraft safety work. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the cabin emergency exit flight attendant safety briefing specification testing method of the present invention;

[0042] Figure 2 This is a detailed illustrated flowchart of the cabin emergency exit safety briefing procedure test according to an embodiment of the present invention;

[0043] Figure 3 This is a structural diagram of the flight attendant and passenger recognition and posture detection algorithm of the present invention;

[0044] Figure 4 This is a schematic diagram of the safety description syntax inference of the present invention;

[0045] Figure 5 This is a schematic diagram of a cabin emergency exit flight attendant safety briefing procedure testing device according to an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0047] like Figure 1 and Figure 2 As shown, this invention provides a method for detecting the standardization of cabin emergency exit safety introductions by flight attendants. It utilizes deep learning, posture detection, image recognition, and speech recognition to learn and process keywords from the flight attendant's posture and speech during the standardized guidance of passengers at the emergency exit. Combined with syntactic pattern recognition, it infers whether the entire safety introduction and guidance process is standardized. The main steps of the flight attendant safety introduction standardization detection are: Detection Step 1: Audio detection at the emergency exit; Detection Step 2: Identification of flight attendants and passengers at the emergency exit; Detection Step 3: Posture detection of flight attendants and passengers at the emergency exit.

[0048] More specifically, it includes the following steps:

[0049] Step 1: Based on the flight attendant manual, determine the specific rules and standard safety terminology for flight attendants to provide safety briefings to passengers at the emergency exits. Install a high-definition monitoring camera with an audio sensor on the top of the aircraft in the middle of the emergency exit to collect audio and video data in real time after passengers board and before takeoff.

[0050] Step 2: Preprocess the collected audio and video data to form a dataset of voice and video of flight attendants giving safety instructions to passengers.

[0051] Specifically, the video dataset is divided into categories with and without safety introductions. The video category data is cleaned and labeled to form a training, verification, and testing dataset for flight attendants' safety introduction videos. Standard safety introduction terms are processed through a language model to generate safety introduction keyword reference templates. The collected audio data is then cleaned and labeled to form a training, verification, and testing dataset for flight attendants' safety introduction voice.

[0052] Step 3: Based on the safety introduction keyword reference template obtained in Step 2, train the audio data collected multiple times at the aircraft emergency exits to obtain a safety introduction audio keyword detection algorithm that meets the recognition rate requirements. This includes the following sub-steps:

[0053] Step 3.1: For the audio data from the emergency exit, MFCC feature parameters need to be extracted. First, the speech signal is preprocessed, including pre-emphasis, windowing, and framing. Then, each frame is subjected to FFT transformation to convert the data from the time domain to the frequency domain, and its spectral energy is calculated. The calculated spectral energy spectrum of each frame is then passed through a Mel filter, and the energy in that Mel filter is calculated using the following formula: , 0≤m≤M, where i represents the i-th frame and k represents the k-th spectral line in the frequency domain. Finally, the DCT is calculated after taking the logarithm of the energy in the Mel filter, and the MFCC parameters can be obtained, as shown in the following formula. , where n is the spectral line after DCT.

[0054] Step 3.2: The acoustic model and pronunciation dictionary are used to process the MFCC feature parameters. The publicly available Chinese speech training database is converted into a spectrogram for feature extraction. 1427 pinyin syllables composed of tones and vowels are used as the modeling units of the acoustic model. The pinyin sequences corresponding to the audio signals are processed to generate pinyin index sequences. Then, the spectrogram and pinyin index sequences are used as input data to train the acoustic model.

[0055] Step 3.3: During the decoding process, the acoustic model processes the MFCC feature parameters to generate the pinyin index sequence. After finding the corresponding pinyin sequence in the pronunciation dictionary, the recognized text is output.

[0056] Step 3.4: In civil aviation regulations, the broadcast of passenger information indicates that all passengers are seated. If the recognized text contains the broadcast passenger information, an instruction to begin video detection of emergency exits is conveyed.

[0057] Step 3.5: Set the threshold for terminology keyword matching, match the text identified from the emergency exit audio with the safety introduction keyword reference template, and use the matching result for step 6 syntactic inference to determine whether the flight attendant's safety introduction at the emergency exit is standardized.

[0058] Step 4: The collected video dataset is uniformly sampled using a method of extracting one frame per second to obtain a total of T frames. The image data is used for training to obtain a human body recognition and posture detection algorithm for flight attendants and passengers at emergency exits that meets the recognition rate requirements. This step specifically includes the following sub-steps:

[0059] Step 4.1: Preliminary Preparation of Training and Testing Data: Videos collected from monitoring the emergency exit area of ​​the aircraft were divided into training, validation, and testing sets in a 6:2:2 ratio. The video dataset from Step 2 was then processed to obtain T frames using a method of extracting one frame per second. I =[ [Data annotation of target bounding boxes and crew attribute features and skeletons for flight attendants and passengers.]

[0060] Step 4.2: Design a flight attendant and passenger recognition and posture detection network.

[0061] Step 4.2.1: The T-frame video frames obtained by uniform sampling are cropped into images with a resolution of 224×224 and transmitted forward in parallel to the crew and passenger recognition and posture detection network.

[0062] Step 4.2.2: As Figure 3 As shown, the flight attendant and passenger recognition network employs the Faster-RCNN algorithm. T frames of images are processed by a CNN network to extract attribute features such as uniforms, hairstyles, badges, and scarves of the flight attendants, generating 16×16 feature maps. In the RPN structure, a 3×3 window is used to scan the image to obtain candidate boxes. After the feature maps and candidate boxes are input into the ROI pooling layer, softmax is used in the fully connected layer to classify the human figures in the image. Bounding box regression generates detection boxes with flight attendant and passenger identifiers. Then, the multi-scale features shared by the hourglass network are used to regress the human bounding boxes on the input image. Fusion of these two feature sources allows for better prediction of human bounding boxes. After identifying flight attendants and passengers, the video area is divided into left and right regions along the aisle centerline. The duration of the flight attendant's stay in the left and right regions is calculated, providing information for step 6, syntactic inference, to determine whether the flight attendant's safety presentation at the emergency exit was performed correctly.

[0063] Step 4.2.3: In the pose detection network, to achieve the different pose detection tasks for flight attendants and passengers, it is first necessary to distinguish between flight attendants and passengers. The bounding box regression parameters obtained from the human body recognition network and the T-frame images are input into the hourglass network. The generated human key point heatmap is represented as a third-order tensor (W×H×). M * ), M * This represents the total number of heatmaps, i.e., the key points of the human body. Each key point generates one heatmap. A total of 16 key points are extracted, including: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left waist, right waist, left knee, right knee, left ankle, and right ankle. Each individual in each frame receives 17 keypoint heatmaps. The 2D pose size is represented as 17×56×56. The keypoint heatmap M for each frame is extracted; the formula for calculating M is: , () represents the pixel coordinates. Represents the coordinates of the K key point; The confidence level is represented by a multi-scale supervision method, unlike ordinary hourglass networks. This method incorporates multi-scale supervision during deconvolution, scaling down the data by 1 / 8, 1 / 4, and 1 / 2 respectively to improve the detail of the keypoint heatmap. These heatmaps are then magnified by 8, 4, and 2 times respectively before being concatenated with the deconvolution output, resulting in a more accurate human pose detection output. After detecting the poses of the flight attendant and passengers, the method further detects the angle between the flight attendant's arm and body skeletons, as well as the body skeleton's tilt angle, for use in step 6, which uses syntactic inference to determine whether the flight attendant's safety introduction process is fully compliant with regulations.

[0064] Step 4.3: Model training and testing: Input the training data of the flight attendants' uniforms, hairstyles, badges, scarves, and the human body frames and skeletons of the flight attendants and passengers into the flight attendant and passenger recognition and posture detection network for training, and test it to obtain a human body recognition and posture detection algorithm for flight attendants and passengers at emergency exits that meets the recognition rate.

[0065] Step 4.4: Model Start Time: To improve the utilization of computing resources, wait for the speech recognition to read the passenger information broadcast, and then start the posture detection algorithm for the flight attendants and passengers at the emergency exit.

[0066] Step 5: Using the safety introduction voice keyword detection algorithm from Step 3 and the human body recognition and posture detection algorithm from Step 4, identify audio-text information at the emergency exits in the aircraft cabin and detect the posture information of flight attendants and passengers. This includes the following sub-steps:

[0067] Step 5.1: Using the template matching-based safety introduction voice keyword detection algorithm from Step 4, output the recognized text in real time. If the text information of the passenger instructions broadcast in the cabin is recognized, then start the human body recognition and posture detection algorithm for flight attendants and passengers at the emergency exit.

[0068] Step 5.2: The SOTFMAX layer of the human body recognition and posture detection algorithm for flight attendants and passengers at the emergency exit outputs whether the video content belongs to the category of flight attendants giving safety instructions to passengers, and sets the confidence threshold to [value missing]. If the output softmax value is less than the threshold, then the video will be output as having no safety briefing content from the flight attendant to the passenger; if the output softmax value is greater than the threshold, then in step 6, the syntax will be used to infer whether the safety briefing content from the flight attendant to the passenger fully complies with the specifications.

[0069] Step 6: Based on the rules at emergency exits, the identification and detection results from Step 5 are used to infer, using context-sensitive grammar in the syntactic model, whether the entire process of the flight attendant's safety introduction and guidance complies with regulations. This specifically includes the following sub-steps:

[0070] Step 6.1: The following rules apply to emergency exits: (1) Passengers must be seated at both emergency exit positions; (2) Flight attendants must conduct safety briefings at designated locations; (3) Flight attendants must tilt their bodies at a certain angle and raise their arms during the safety briefing; (4) Each passenger at the emergency exit positions in the left and right areas must be briefed once; (5) The content of the flight attendant's safety briefing voice must conform to the regulations; Design a method to meet the above emergency exit rules: (1) Passengers must be detected seated at both emergency exit positions; (2) Flight attendants must be detected in the emergency exit area; (3) During the flight attendant's safety briefing, the overall tilt angle of the body frame at time t1 must be greater than α, and the angle between the arm and body at time t2 must be greater than β; (4) The flight attendant's dwell time in the left and right areas must be greater than time t3 respectively; (5) The matching rate between the keywords of the flight attendant's voice and the template must be greater than the threshold.

[0071] Step 6.2: Let the entire process of the flight attendant's safety introduction be grammar G, G = {Vn, Vt, P, S}, where Vn is a non-terminal symbol, represented by uppercase letters, and Vn = {A, B, C, D, E, F, G}, where A indicates the start of inferring whether the flight attendant's safety introduction and guidance actions are standardized; B indicates the start of inferring whether the flight attendant's safety introduction voice is standardized; C indicates that there are passengers seated at the emergency exit; D indicates that there is a flight attendant at the emergency exit; E indicates that during the flight attendant's safety introduction, at time t1, the overall tilt angle of the flight attendant's body skeleton is greater than α, and at time t2, the angle between the arm and body is greater than β, indicating that the flight attendant's posture is standardized; F indicates that the flight attendant frequently stops in the left and right areas respectively. The time is greater than t3, which conforms to the one-person-one-introduction rule; G indicates that the matching rate between the keywords in the flight attendant's voice and the template must be greater than the threshold, and the flight attendant's voice introduction is standardized; Vt is the terminator, Vt={c,d,e,f,g,T}, where c indicates that no passenger is seated at the emergency exit; d indicates that the flight attendant's position is abnormal; e indicates that the flight attendant's posture is not standard; f indicates that the flight attendant's safety introduction does not conform to the one-person-one-introduction rule; g indicates that the flight attendant's voice introduction is not standard; T indicates that all the above syntaxes conform to the rules, and the entire process of the flight attendant's safety introduction to passengers is standardized; except for T, all other terminators output "xx is not standard"; P is the production rule of the syntax; S is the start symbol, indicating the start of syntactic inference;

[0072] Step 6.3: Begin the syntactic inference process. In the inference process for the flight attendant's safety introduction and guidance, there are a total of 5 production rules, P = {SAc, SACd, SACDe, SACDEf, SACDEF}. Here, 1 indicates continuing the inference forward, and 0 indicates the appearance of a terminator. Therefore, production rule SAc: S→SA→SAc, indicates that no passenger is seated at the emergency exit, and outputs an alarm indicating no passenger is seated at the emergency exit. Production rule SACd: S→SA→SAC→SACd, indicates that the flight attendant's position is abnormal, and outputs an alarm indicating abnormal flight attendant position. Normal alarm; Generation formula SACDe: S→SA→SAC→SACD→SACDe, indicating that the flight attendant's posture is not in accordance with the standard, outputting a flight attendant posture abnormality alarm; Generation formula SACDEf: S→SA→SAC→SACD→SACDE→SACDEf, indicating that the flight attendant's safety introduction does not conform to the one-person-one-introduction rule, outputting a flight attendant does not conform to the one-person-one-introduction rule alarm; Generation formula SACDEF: S→SA→SAC→SACD→SACDE→SACDEF, indicating that the flight attendant's body and movements during the passenger safety introduction are all in accordance with the standard;

[0073] Step 6.4: In the flight attendant safety introduction voice standard inference process, there are a total of 2 production rules, P={SBg, SBG}; Production rule SBg: S→SB→SBg, which means that the flight attendant's voice introduction is not standard and outputs a flight attendant voice introduction not standard alarm; Production rule SBG: S→SB→SBG, which means that the flight attendant's voice introduction is standard.

[0074] Step 6.5: After completing the deduction process of the flight attendant's safety introduction and guidance action and voice specifications, if both specifications are met, the production formula is P={SABCDEFT}; Production formula SABCDEFT: S→SA→SAB→SABC→SABCD→SABCDE→SABCDEF→SABCDEFT, indicating that the emergency exit flight attendant safety introduction process is fully compliant.

[0075] Corresponding to the aforementioned embodiment of the cabin emergency exit flight attendant safety introduction standard testing method, the present invention also provides an embodiment of the cabin emergency exit flight attendant safety introduction standard testing device.

[0076] See Figure 5 The present invention provides a cabin emergency exit flight attendant safety introduction procedure testing device, which includes one or more processors for implementing a cabin emergency exit flight attendant safety introduction procedure testing method in the above embodiments.

[0077] An embodiment of the cabin emergency exit flight attendant safety briefing procedure detection device of the present invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 5 The diagram shown is a hardware structure diagram of any data processing-capable device used for detecting the cabin emergency exit flight attendant safety briefing procedures according to the present invention. (Except for...) Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0078] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0079] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0080] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a cabin emergency exit flight attendant safety briefing procedure testing method described in the above embodiments.

[0081] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0082] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method of detecting a cabin emergency exit attendant safety introduction specification, characterized by, The method comprises the following steps: Step 1: According to the behavior specification rules and standard safety introduction terms of the safety introduction of the flight attendant to the passengers at the emergency exit in the flight attendant specification manual, a high-definition monitoring camera with an audio sensor is installed at the top of the middle of the emergency exit to collect audio and video data after the passengers board the plane and before the plane takes off; Step 2: The collected audio and video data are preprocessed to form an audio and video data set of the safety introduction of the flight attendant to the passengers at the emergency exit of the plane; Step 3: The audio data sets collected at the emergency exit of the plane are trained to obtain a safety introduction voice keyword detection algorithm meeting the recognition rate requirement; Step 4: The video data set is uniformly sampled by using a 1s extraction one frame method to obtain T frames of pictures, and the picture data is trained to obtain a human body recognition and posture detection algorithm of the flight attendant and the passengers at the emergency exit meeting the recognition rate requirement; Step 5: The safety introduction voice keyword detection algorithm of step 3 and the human body recognition and posture detection algorithm of step 4 are used to recognize the audio text information and detect the posture information of the flight attendant and the passengers at the emergency exit in the passenger cabin of the plane; Step 6: Based on the rules at the emergency exit, the recognition and detection results of step 5 are inferred whether the whole process of the safety introduction and guidance of the flight attendant conforms to the specification by using the context-related grammar in the syntax model; The step 6 specifically comprises the following sub-steps: Step 6.1: The emergency exit has the following rules: (1) the passengers must sit at the positions of the left and right emergency exits; (2) the flight attendant needs to introduce safety at the specified position; (3) the flight attendant needs to have a certain inclination angle and arm lifting action during the safety introduction; (4) the flight attendant needs to introduce the safety to one passenger at the left and right emergency exit positions respectively; (5) the voice content of the safety introduction of the flight attendant needs to conform to the specification; the methods meeting the above emergency exit rules are designed as follows: (1) the passengers need to be detected to sit at the positions of the left and right emergency exits; (2) the flight attendant needs to be detected at the emergency exit area; (3) the whole body skeleton of the flight attendant needs to have an inclination angle greater than a for t1 time and the angle between the arm and the body needs to be greater than b for t2 time during the safety introduction; (4) the flight attendant needs to stay at the left and right areas for more than t3 time respectively; (5) the matching rate of the keywords of the voice of the flight attendant and the template needs to be greater than a threshold value; Step 6.2: Set the safety introduction process of the crew as the grammar G, G = {Vn, Vt, P, S}, where Vn is a non-terminal symbol represented by a capital letter, Vn = {A, B, C, D, E, F, G}, where A represents the start of the inference of whether the action of the crew safety introduction and guidance is standard; B represents the start of the inference of whether the speech of the crew safety introduction is standard; C represents that there is a passenger sitting at the emergency exit location; D represents that there is a crew member at the emergency exit; E represents that the crew member's body posture is standard during the safety introduction process, with the overall inclination angle of the body skeleton being greater than α for t1 time, and the angle between the arm and the body being greater than β for t2 time; F represents that the crew member stays in the left and right areas for more than t3 time respectively, in line with the one-introduction rule; G represents that the matching rate of the key words of the crew speech and the template needs to be greater than the threshold, and the crew speech introduction is standard; Vt is a terminal symbol, Vt = {c, d, e, f, g, T}, where c represents that there is no passenger sitting at the emergency exit location; d represents that the crew member's position is abnormal; e represents that the crew member's posture is not standard; f represents that the crew member's safety introduction does not meet the one-introduction rule; g represents that the crew member's speech introduction is not standard; T represents that all the above grammars meet the rules, and the crew's entire process of safety introduction to passengers is standard; all the other terminal symbols output "xx is not standard" except T; P is the production of the grammar; S is the start symbol, representing the start of the grammar inference; Step 6.3: Start entering the grammar inference process, in the action standard inference process of the crew safety introduction and guidance, there are a total of 5 productions, P = {SAc, SACd, SACDe, SACDEf, SACDEF}; where 1 represents continuing to infer forward, 0 represents that a terminal symbol appears later, then the production SAc: S→SA→SAc, represents that there is no passenger sitting at the emergency exit location, and outputs an emergency exit location no passenger sitting alarm; the production SACd: S→SA→SAC→SACd, represents that the crew member's position is abnormal, and outputs a crew member position abnormal alarm; the production SACDe: S→SA→SAC→SACD→SACDe, represents that the crew member's posture does not meet the standard, and outputs a crew member posture abnormal alarm; the production SACDEf: S→SA→SAC→SACD→SACDE→SACDEf, represents that the crew member's safety introduction does not meet the one-introduction rule, and outputs a crew member not meeting the one-introduction rule alarm; the production SACDEF: S→SA→SAC→SACD→SACDE→SACDEF, represents that the crew member's entire body and action of safety introduction to passengers is standard; Step 6.4: In the speech standard inference process of the crew safety introduction, there are a total of 2 productions, P = {SBg, SBG}; the production SBg: S→SB→SBg, represents that the crew member's speech introduction is not standard, and outputs a crew member speech introduction not standard alarm; the production SBG: S→SB→SBG, represents that the crew member's speech introduction is standard. Step 6.5: After the action specification and the voice specification inference process of the safety introduction and guidance of the attendant are completed, if the specifications are met at the same time, the production is P = {SABCDEFT}; the production SABCDEFT: S→SA→SAB→SABC→SABCD→SABCDE→SABCDEF→SABCDEFT, indicating that the emergency exit attendant safety introduction process is all specifications.

2. A method of detecting a cabin emergency exit purser safety presentation specification as claimed in claim 1, characterized in that, The step 2 is specifically: dividing the video data set into video categories with and without safety introduction, and cleaning and labeling the video category data to form the training, verification and test data set of the safety introduction video of the attendant to the passenger; After the standard safety introduction terms are processed by the language model to generate the safety introduction keyword reference template, the collected audio data is cleaned and labeled to form the training, verification and test data set of the safety introduction voice of the attendant.

3. A method of detecting a cabin emergency exit purser safety presentation specification as claimed in claim 2, characterized in that, The step 3 specifically includes the following sub-steps: Step 3.1: For the audio data at the emergency exit, extract the MFCC feature parameters: first, pre-process the audio signal, which includes pre-emphasis, windowing, and framing steps; then, perform FFT transformation on each frame of signal, convert the data from time domain to frequency domain, and calculate the spectral line energy. The calculated energy spectrum of each frame is filtered through a Mel filter to calculate the energy in the Mel filter, and the formula is as follows: , 0≤m≤M, where i represents the i-th frame, and k represents the k-th spectral line in the frequency domain; finally, take the logarithm of the energy in the Mel filter to calculate the DCT, i.e., the MFCC feature parameters, and the expression is as follows: , where n is the DCT spectrum line. Step 3.2: Use the acoustic model and pronunciation dictionary to process the MFCC feature parameters, convert the public Chinese voice training database into a spectrogram to extract features, and use 1427 kinds of pinyin composed of tones and vowels as modeling units of the acoustic model; process the pinyin sequence corresponding to the audio signal to generate the index sequence of the pinyin, and then input the spectrogram and the index sequence of the pinyin into the acoustic model for training; Step 3.3: In the decoding process, the acoustic model processes the MFCC feature parameters to generate a pinyin index sequence, and after searching for the corresponding pinyin sequence in the pronunciation dictionary, the recognized text is output; Step 3.4: When it is judged that the recognized text contains the passenger information broadcasted by the broadcast, the instruction for detecting the video at the emergency exit is started to be conveyed; Step 3.5: Set the threshold value of the term keyword matching, and match the text recognized by the audio at the emergency exit with the safety introduction keyword reference template.

4. A method of detecting a cabin emergency exit purser safety presentation specification according to claim 3, characterized in that, The step 4 specifically includes the following sub-steps: Step 4.1: divide the video dataset into training set, validation set and test set according to 6:2:2 by collecting the video through monitoring the area of the emergency exit of the airplane; divide the collected video dataset into T frame data by adopting the method of extracting one frame every 1s I [ I 1, ...I T ]annotating the data of target frame and the attribute features and skeleton of the flight attendant to the flight attendant and the passenger; Step 4.2: Design the attendant and passenger recognition and posture detection network; Step 4.3: Label the training data of the uniform, hairstyle, badge, scarf of the attendant, and the human body frame and skeleton of the attendant and passenger, input them into the attendant and passenger recognition and posture detection network for training, and test them to obtain the human body recognition and posture detection algorithm of the attendant and passenger at the emergency exit that meets the recognition rate; Step 4.4: When the passenger information broadcasted by the broadcast is recognized by the voice, the posture detection algorithm of the attendant and passenger at the emergency exit is started to be detected.

5. A method of detecting a cabin emergency exit purser safety presentation specification as claimed in claim 4, characterized in that, The step 4.2 specifically includes the following sub-steps: Step 4.2.1: The T-frame video frames obtained by uniform sampling are cropped into pictures with a resolution size of 224x224, which are transmitted in parallel to the attendant and passenger recognition and posture detection network; Step 4.2.2: In the flight attendant and passenger identification network, the Faster-RCNN algorithm is used, and the T-frame picture is extracted by the CNN network to obtain the attribute features of the flight attendant's uniform, hairstyle, chest badge and scarf, to generate a 16x16 feature map; in the RPN structure, a 3x3 size window is used to scan the image to obtain a candidate frame, the feature map and the candidate frame are input into the ROI pooling layer, and then the human body in the picture is classified by the softmax in the full connection layer, and the detection frame with the flight attendant and passenger identification is generated by the boundary frame regression, and the multi-scale features shared by the hourglass network are used to regress the human boundary frame on the input image, and the two feature sources are fused to better predict the human boundary frame; after identifying the flight attendant and passenger, the video area is divided into left and right areas along the center line of the aisle, and the time length of the flight attendant staying in the left and right areas is calculated; Step 4.2.3: In the pose detection network, first, the flight attendants and passengers need to be distinguished. The bounding box regression parameters obtained in the human body recognition network and the T-frame picture are jointly input into the hourglass network. The generated human key point heat map is represented as a three-order tensor (WxHx M * M * is the total number of heat maps, that is, the key points of the human body, and each key point generates a heat map. W and H are the width and height of the heat map, respectively. A total of 16 key points are extracted, including nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left waist, right waist, left knee, right knee, left ankle, and right ankle. Each individual in each frame of picture obtains 17 key point heat maps, and the two-dimensional pose size is represented as 17x56x56. The key point heat map of each frame of picture is extracted M M The calculation formula is as follows: (i,j) represents the pixel point coordinates, c k represents the confidence. During the deconvolution process, multi-scale supervision is added. The picture is reduced by 1 / 8, 1 / 4, and 1 / 2, respectively, and is enlarged by 8, 4, and 2 times, respectively, and then is spliced with the deconvolution output to obtain the human body pose detection output. After detecting the poses of the flight attendants and passengers, the angle between the arm skeleton and the body skeleton of the flight attendants and the inclination angle of the body skeleton are further detected.​​ 6. A method of detecting a cabin emergency exit purser safety presentation specification as claimed in claim 5, characterized in that, The step 5 specifically includes the following sub-steps: Step 5.1: Real-time output of the recognized text by the safety introduction voice keyword detection algorithm of step 3, if the passenger information text broadcast by the cabin broadcast is recognized, the human body identification and posture detection algorithm of the flight attendant and passenger at the emergency exit is started; Step 5.2: Human body recognition and posture detection algorithm of the crew member and the passenger at the emergency exit sotfmax layer outputs whether the video content belongs to the category of the crew member's safety introduction to the passenger, and sets the confidence threshold value to If the output softmax value is less than the threshold value, the video is output without the crew member's safety introduction content to the passenger; if the output softmax value is greater than the threshold value, it is inferred whether the crew member's safety introduction content to the passenger is all in line with the specification by syntax.

7. A cabin emergency exit attendant safety presentation specification detection device, characterized by, The one or more processors are configured to implement the cabin emergency exit flight attendant safety introduction specification detection method of any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, A program is stored thereon, and the program is executed by the processor to implement the cabin emergency exit flight attendant safety introduction specification detection method of any one of claims 1 to 6.

Images