Airborne entertainment screen control method, airborne electronic equipment and control system

Abstract

The invention discloses an airborne entertainment screen control method, airborne electronic equipment and a control system. The method comprises the following steps: inputting a collected passenger gesture image into a pre-trained gesture recognition model for recognition to obtain a corresponding control instruction; according to the monitored pressure data and / or passenger cabin broadcast voice data, whether the aircraft is in a stable state currently is judged, and if yes, the recognized control instruction is sent to a passenger cabin entertainment display screen for corresponding operation. Otherwise, discarding the identified control instruction and sending an early warning instruction to a passenger cabin entertainment display screen for airplane bumping and pausing entertainment control prompt. According to the method, the gesture action of the passenger is captured, the collected image is input into the model, the corresponding control instruction is obtained through recognition, and the control instruction is sent to the passenger cabin entertainment display screen for corresponding operation only in the stable flight state of the airplane in combination with an anti-jittering and jolting protection mechanism, so that false triggering caused by jittering and jolting is avoided, and the safety of the airplane is improved. And the control precision and reliability are improved.

Description

Technical Field

[0001] The present application relates to the field of aviation airborne technology, and in particular to an airborne entertainment screen control method, airborne electronic equipment and control system. Background Art

[0002] In order to provide passengers with higher quality services, the cabin will provide an onboard entertainment system for passengers to kill time during the long flight. Especially for relatively spacious areas such as first class and business class, a large high-definition entertainment screen is generally installed, and the screen is far away from the passenger seat. The entertainment screen cannot be controlled by ordinary cabin touch screens, and a remote controller must be equipped to control the entertainment screen.

[0003] For the control of the above entertainment screens, wired or wireless (for example, based on infrared or WIFI) remote controls are generally used. For wireless remote controls, due to interference from WIFI and infrared signals, they cannot be used during certain periods of the flight, resulting in passengers being unable to control the entertainment screen; for wired remote controls, wiring from the screen to the passenger position is required, increasing deployment and operation and maintenance costs. In addition, due to the jitter of the aircraft during flight, the remote control is generally fixed in a certain position in the cabin, which not only affects the passenger's operating experience, but also causes inconvenience in disassembly and maintenance.

[0004] In addition, there are many technologies that use gesture recognition technology to control electronic equipment. However, most of them are for controlling electronic equipment in a static environment on the ground. During the flight, the aircraft often shakes and bumps due to different flight phases, airflow, weather and other factors. If the in-flight entertainment display screen is controlled only by recognizing passengers' gestures, the control accuracy is not high and it is easy to cause misjudgment. Summary of the invention

[0005] In order to solve the problems existing in the existing control of in-flight entertainment display screens using wired or wireless remote controls, the present application proposes an in-flight entertainment screen control method, in-flight electronic equipment and control system. The present application captures passenger gestures and inputs the collected images into a pre-trained model to identify corresponding control instructions. In combination with an anti-shake and turbulence protection mechanism, control instructions are sent to the cabin entertainment display screen for corresponding operations only when the aircraft is in a stable flight state, thereby avoiding false triggering caused by shake and turbulence and improving control accuracy and reliability.

[0006] In the first aspect, the present application is implemented through the following technical solutions:

[0007] A method for controlling an in-flight entertainment screen, the control method comprising:

[0008] Input the collected passenger gesture images into the pre-trained gesture recognition model to obtain corresponding control instructions;

[0009] Based on the monitored pressure data and / or cabin broadcast voice data, it is determined whether the aircraft is currently in a stable state. If so, the recognized control command is sent to the cabin entertainment display screen for corresponding operation. Otherwise, the recognized control command is discarded and a warning command is sent to the cabin entertainment display screen to prompt aircraft turbulence and pause entertainment control.

[0010] In some implementations, determining whether the aircraft is currently in a stable state based on the monitored pressure data and / or cabin broadcast voice data specifically includes:

[0011] The pressure data is obtained through the pressure sensor installed at the bottom of the image acquisition device. When the pressure data exceeds the threshold, it is determined that the aircraft is in an unsteady state, otherwise the aircraft is in a stable state;

[0012] And / or, the voice broadcast of the flight attendant is detected and recognized by the voice sensor, and when the voice broadcast of the flight attendant about the turbulence of the aircraft is monitored, it is determined that the aircraft is in an unsteady state, otherwise the aircraft is in a stable state.

[0013] In some implementations, the control instructions include: screen switch instructions, operation control instructions, and program play instructions.

[0014] In some embodiments, before the step of inputting the collected passenger gesture image into a pre-trained gesture recognition model to identify and obtain the corresponding control instruction, the step further includes:

[0015] The collected passenger gesture image is compensated using the monitored pressure data, and the compensation process specifically includes:

[0016] While collecting each frame of image data, the pressure value at the corresponding moment is collected synchronously. When the pressure value is not 0, an image pixel offset is calculated using the collected pressure value.

[0017] The collected image data is compensated using the image pixel offset so that each frame of image data after processing reaches a stable state.

[0018] In a second aspect, the present application proposes an airborne electronic device, the airborne electronic device comprising:

[0019] An image recognition module, which inputs the collected passenger gesture image into a pre-trained gesture recognition model to identify and obtain corresponding control instructions;

[0020] A state determination module, which determines whether the aircraft is currently in a stable state based on the monitored pressure data and / or cabin broadcast voice data. If so, the control communication module sends the identified control command to the cabin entertainment display screen for corresponding operation; otherwise, the identified control command is discarded and a warning command is sent to the cabin entertainment display screen through the communication module to prompt aircraft turbulence and pause entertainment control;

[0021] And, a communication module, the communication module is responsible for completing the communication interaction with the cabin entertainment display screen.

[0022] In some embodiments, the image recognition module further includes:

[0023] An image compensation unit, wherein the image compensation unit uses the monitored pressure data to perform compensation processing on the collected passenger gesture image, wherein the compensation processing process specifically includes:

[0024] While collecting each frame of image data, the pressure value at the corresponding moment is collected synchronously. When the pressure value is not 0, an image pixel offset is calculated using the collected pressure value.

[0025] The collected image data is compensated by using the image pixel offset, so that each frame of image data after processing reaches a stable state;

[0026] And, a recognition unit, wherein the recognition unit inputs the compensated passenger gesture image into a pre-trained gesture recognition model to recognize and obtain corresponding control instructions.

[0027] In a third aspect, the present application proposes an in-flight entertainment screen control system, the control system comprising:

[0028] A data acquisition device, the data acquisition device is used to collect passenger gesture images, pressure data and cabin broadcast voice data;

[0029] The above-mentioned airborne electronic equipment is communicatively connected with the data acquisition equipment;

[0030] And, a cabin entertainment display screen, wherein the cabin entertainment display screen is communicatively connected with the onboard electronic equipment.

[0031] In some embodiments, the data acquisition device includes an image acquisition device, a pressure sensor, and a voice sensor;

[0032] Wherein, the image acquisition device is used to acquire the passenger's gesture image and upload it to the onboard electronic device;

[0033] The pressure sensor is used to collect pressure data representing the aircraft jitter and upload it to the onboard electronic equipment;

[0034] The voice sensor is used to collect cabin broadcast voice data of the crew and upload it to the onboard electronic equipment.

[0035] In some embodiments, the image acquisition device is installed directly above the in-flight entertainment display screen and directly facing the passenger seat; and the pressure sensor is installed at the bottom of the image acquisition device.

[0036] In some embodiments, the cabin entertainment display screen includes a device control module;

[0037] The device control module receives the operation control instruction and completes the corresponding screen operation control function according to the instruction;

[0038] The device control module receives the screen switch instruction and performs the corresponding screen on / off operation according to the instruction;

[0039] The device control module receives a program playing instruction and performs the corresponding program playing operation according to the instruction;

[0040] When the device control module receives an error instruction, it will issue an identification failure prompt according to the instruction;

[0041] When the equipment control module receives the warning instruction, it will perform corresponding aircraft turbulence and pause entertainment control prompts according to the instruction.

[0042] The present application proposes an in-flight entertainment screen control method, in-flight electronic equipment and control system, which captures passenger gesture image data, obtains corresponding control instructions based on the passenger gesture image data combined with deep learning model recognition, and combines an anti-shake protection mechanism to ensure that the corresponding cabin entertainment display screen operation control is performed only when the aircraft is in a stable flight state, thereby reducing false triggering caused by aircraft shaking and turbulence, and improving control accuracy and reliability. In addition, the present application also uses pressure data to compensate for the impact of shaking on the collected image, further improving the accuracy and reliability of control. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the embodiments of the present application, constitute a part of the present application, and do not constitute a limitation on the embodiments of the present application. In the drawings:

[0044] Figure 1 A flow chart of a control method provided in an embodiment of the present application;

[0045] Figure 2 A block diagram of the airborne electronic equipment provided in the embodiment of the present application;

[0046] Figure 3A control system architecture diagram provided for an embodiment of the present application;

[0047] Figure 4 This is a schematic diagram of the installation of the data acquisition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with examples and drawings. The illustrative implementation scheme of the present application and its description are only used to explain the present application and are not intended to limit the present application.

[0049] Example:

[0050] The existing methods of controlling the in-flight entertainment screen using wired or wireless remote controllers have problems such as inconvenient operation and poor user experience, while the technology of controlling electronic devices using gesture recognition technology cannot meet the requirements of in-flight entertainment screen control. In view of this, this embodiment proposes an in-flight entertainment screen control method, which introduces a cabin entertainment anti-shake control mechanism to ensure the reliability of control.

[0051] like Figure 1 As shown, the control method proposed in this embodiment includes the following steps:

[0052] Step 100: collect passenger gesture images and input them into a pre-trained gesture recognition model to obtain corresponding control instructions.

[0053] Among them, the gesture recognition model can adopt existing neural network structures such as deep neural networks, use gesture images and corresponding instruction labels to build a training data set to train the deep neural network, and thus obtain a gesture recognition model.

[0054] In addition, in order to adapt to the dynamic environment of the aircraft flight process, gesture images in the corresponding flight environment can be simulated to construct a training data set to train the deep neural network to improve the recognition accuracy.

[0055] At the same time, in order to ensure the accuracy and reliability of control command recognition when the aircraft is shaking but not turbulent, this embodiment uses pressure data collected from the pressure sensor to compensate the image data; the data of the pressure sensor represents the jitter parameters on the aircraft, and the jitter parameters will directly affect the quality of image acquisition and the recognition results of gestures. The pressure sensor and the image acquisition device (such as a camera) are synchronized by a high-precision clock device to ensure that the clock sources of the two devices are completely synchronized. While collecting each frame of image data, a pressure value at the corresponding moment is synchronously collected. When the pressure value is not 0, it means that the aircraft is shaking at this time. Then, an image pixel offset is calculated using the collected pressure value, and then the collected image is compensated using this offset so that each frame of the image after processing reaches the effect of a stable state, and then the processed image is input into the deep learning model for recognition processing.

[0056] Control instructions mainly include three categories: (1) screen switch instructions; (2) operation control instructions; (3) program play instructions; and (4) error instructions.

[0057] Step 200, based on the monitored pressure data and / or cabin broadcast voice data, determine whether the aircraft is currently in a stable state. If so, send the identified control command to the cabin entertainment display screen for corresponding operation; otherwise, discard the control command and send a warning command to the cabin entertainment display screen to prompt aircraft turbulence and pause entertainment control.

[0058] In order to prevent the aircraft from shaking and bumping during flight, which may cause false triggering of image recognition control, a pressure sensor is installed at the bottom of the image acquisition device (such as a camera) in this embodiment. When the aircraft shakes violently, the pressure sensor will generate a pressure value. The stability of the aircraft is determined by detecting the pressure value of the pressure sensor. During the flight, in order to ensure the safety of passengers, when the aircraft is violently bumped, the cabin will stop entertainment activities; therefore, when the pressure value generated by the pressure sensor reaches a certain threshold, indicating that the aircraft is currently in an unstable state, the cabin entertainment display screen will be controlled to display a prompt indicating that the aircraft is bumping and suspending entertainment control. The control instructions generated at this time will be discarded, avoiding the situation where the control is falsely triggered due to the shaking and bumping of the aircraft.

[0059] Optionally, the present embodiment may also adopt voice recognition technology to determine the stability of the aircraft by monitoring and identifying the voice broadcasts of cabin crew members. When the aircraft is experiencing severe turbulence, it will be affected by the voice broadcasts of aircraft turbulence by the crew members. When the voice is detected, the cabin entertainment display screen will be controlled to display prompts indicating aircraft turbulence and pausing entertainment controls. The control instructions generated at this time will be lost, avoiding the situation where control errors due to aircraft jitter are lost.

[0060] It should be noted that this embodiment can provide reminders of aircraft turbulence and suspension of entertainment control by monitoring pressure exceeding a certain threshold, or by monitoring the flight attendant's voice broadcast of aircraft turbulence, or can provide reminders of aircraft turbulence and suspension of entertainment control by synchronously monitoring pressure and detecting the flight attendant's voice broadcast of aircraft turbulence.

[0061] Furthermore, the method proposed in this embodiment also includes:

[0062] The collected passenger gesture images, cabin broadcast voice data and pressure data are verified to further ensure the reliability of the data.

[0063] For the collected passenger gesture images, this embodiment uses conventional calibration methods such as motion trajectory direction analysis in N consecutive frames of images and image data smoothing, and also combines discrete instructions such as fasten seat belts FSB and prohibit mobile phones No Mobile from the airborne avionics system for processing; when fasten seat belts FSB or prohibit mobile phones No Mobile is discretely triggered, this embodiment will analyze the motion trajectory and direction of the received image data. If it is found that the passenger has fastened the seat belt or operated the mobile phone, the gesture instruction generated at this time will be discarded. At the same time, this embodiment will also analyze the passenger's face and eyes. When a gesture instruction is recognized but the passenger's face or eyes are not facing the in-flight entertainment screen, it is considered a false trigger and the instruction will be discarded.

[0064] For cabin broadcast voice data, the onboard electronic equipment of this embodiment will be connected to the cabin broadcast intercom system through a discrete line. When the crew makes a cabin broadcast, the discrete line will be triggered. This embodiment will only collect and recognize broadcast voice data when the broadcast discrete line is triggered. At the same time, this embodiment also suppresses non-human audio through a filter algorithm for the collected voice data, retaining only the human voice, greatly reducing the possibility of false touches.

[0065] For the pressure data of the pressure sensor, this embodiment performs continuity judgment on the collected data and processes it only when there are at least two consecutive pressure points; and stipulates a pressure threshold range, and discards data that greatly deviates from the threshold range.

[0066] This embodiment uses gesture recognition technology in combination with an anti-shake mechanism to achieve reliable control of the in-flight entertainment display screen and reduce the occurrence of false triggering.

[0067] This embodiment also proposes an airborne electronic device, such as Figure 2 As shown, the airborne electronic equipment proposed in this embodiment includes:

[0068] The image recognition module inputs the collected passenger gesture images into a pre-trained gesture recognition model to identify and obtain corresponding control instructions.

[0069] A state identification module determines whether the aircraft is currently in a stable state based on the monitored pressure data and / or cabin broadcast voice data. If so, the control communication module sends the identified control command to the cabin entertainment display screen for corresponding operation; otherwise, the control command is discarded and the control communication module outputs a warning command to the cabin entertainment display screen to prompt aircraft turbulence and pause entertainment control.

[0070] And, a communication module, which is responsible for completing communication interaction with the cabin entertainment display screen and sending control instructions or warning instructions to the cabin entertainment display screen under the control of the state judgment module.

[0071] Furthermore, the airborne electronic equipment also includes:

[0072] The data verification module is used to verify the collected data to ensure the reliability of the data. The collected data includes cabin broadcast voice data, passenger gesture image data and pressure data. The specific verification process is as described in the above method and will not be repeated here.

[0073] Furthermore, the image recognition module also includes:

[0074] An image compensation unit is used to compensate the passenger's gesture image data using the pressure data collected by the pressure sensor. The specific compensation process is as described in the above method and will not be described in detail here.

[0075] The recognition unit inputs the compensated passenger gesture image into a pre-trained gesture recognition model to obtain corresponding control instructions.

[0076] This embodiment also proposes an in-flight entertainment screen control system, such as Figure 3 As shown, the system proposed in this embodiment includes:

[0077] The data acquisition device includes an image acquisition device, a pressure sensor and a voice sensor; wherein the image acquisition device is used to acquire passenger gesture images and upload them to the onboard electronic equipment, the pressure sensor is used to acquire pressure data representing the aircraft jitter and upload them to the onboard electronic equipment, and the voice sensor is used to acquire cabin broadcast voice data of the crew and upload them to the onboard electronic equipment.

[0078] Optionally, the image acquisition device may be a camera but is not limited to a camera, which may be installed directly above the in-flight entertainment display screen and directly facing the passenger seat; the pressure sensor may be installed at the bottom of the image acquisition device, such as Figure 4 As shown; the voice sensor is installed in the cabin.

[0079] The above-mentioned airborne electronic equipment interacts with the data acquisition device and is mainly responsible for processing and identifying the collected data uploaded by the data acquisition device, and outputting control instructions or warning instructions to the cabin entertainment display screen.

[0080] And, the cabin entertainment display screen includes a device control module, which is connected to the onboard electronic equipment in communication, and is used to perform corresponding actions according to the control instructions or warning instructions issued by the onboard electronic equipment. For example, if an operation control instruction is received, the corresponding screen operation control function is completed according to the instruction; if a screen switch instruction is received, the corresponding screen on / off operation is completed according to the instruction; if a program playback instruction is received, the corresponding program playback operation is completed according to the instruction; if a warning instruction is received, the corresponding aircraft turbulence and pause entertainment control prompts are performed according to the instruction; if an error instruction is received, the recognition failure prompt is performed according to the instruction.

[0081] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0082] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0085] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for controlling an in-flight entertainment screen, characterized in that: The control method comprises: Input the collected passenger gesture images into the pre-trained gesture recognition model to obtain corresponding control instructions; Based on the monitored pressure data and / or cabin broadcast voice data, it is determined whether the aircraft is currently in a stable state. If so, the recognized control command is sent to the cabin entertainment display screen for corresponding operation. Otherwise, the recognized control command is discarded and a warning command is sent to the cabin entertainment display screen to prompt aircraft turbulence and pause entertainment control.

2. The in-flight entertainment screen control method according to claim 1, characterized in that: The step of determining whether the aircraft is currently in a stable state based on the monitored pressure data and / or cabin broadcast voice data specifically includes: The pressure data is obtained through the pressure sensor installed at the bottom of the image acquisition device. When the pressure data exceeds the threshold, it is determined that the aircraft is in an unsteady state, otherwise the aircraft is in a stable state; And / or, the voice broadcast of the flight attendant is detected and recognized by the voice sensor, and when the voice broadcast of the flight attendant about the turbulence of the aircraft is monitored, it is determined that the aircraft is in an unsteady state, otherwise the aircraft is in a stable state.

3. The in-flight entertainment screen control method according to claim 1 or 2, characterized in that: The control instructions include: screen switch instructions, operation control instructions and program play instructions.

4. The in-flight entertainment screen control method according to claim 1 or 2, characterized in that: Before the step of inputting the collected passenger gesture image into the pre-trained gesture recognition model to identify and obtain the corresponding control instruction, the step further includes: The collected passenger gesture image is compensated using the monitored pressure data, and the compensation process specifically includes: While collecting each frame of image data, the pressure value at the corresponding moment is collected synchronously. When the pressure value is not 0, an image pixel offset is calculated using the collected pressure value. The collected image data is compensated using the image pixel offset so that each frame of image data after processing reaches a stable state.

5. An airborne electronic device, characterized in that: The onboard electronic equipment includes: An image recognition module, which inputs the collected passenger gesture image into a pre-trained gesture recognition model to identify and obtain corresponding control instructions; A state determination module, which determines whether the aircraft is currently in a stable state based on the monitored pressure data and / or cabin broadcast voice data. If so, the control communication module sends the identified control command to the cabin entertainment display screen for corresponding operation; otherwise, the identified control command is discarded and a warning command is sent to the cabin entertainment display screen through the communication module to prompt aircraft turbulence and pause entertainment control; And, a communication module, the communication module is responsible for completing the communication interaction with the cabin entertainment display screen.

6. An airborne electronic device according to claim 5, characterized in that: The image recognition module also includes: An image compensation unit, wherein the image compensation unit uses the monitored pressure data to perform compensation processing on the collected passenger gesture image, wherein the compensation processing process specifically includes: While collecting each frame of image data, the pressure value at the corresponding moment is collected synchronously. When the pressure value is not 0, an image pixel offset is calculated using the collected pressure value. The collected image data is compensated by using the image pixel offset, so that each frame of image data after processing reaches a stable state; And, a recognition unit, wherein the recognition unit inputs the compensated passenger gesture image into a pre-trained gesture recognition model to recognize and obtain corresponding control instructions.

7. An in-flight entertainment screen control system, characterized in that: The control system comprises: A data acquisition device, the data acquisition device is used to collect passenger gesture images, pressure data and cabin broadcast voice data; The airborne electronic device as claimed in claim 5 or 6, wherein the airborne electronic device is communicatively connected to the data acquisition device; And, a cabin entertainment display screen, wherein the cabin entertainment display screen is communicatively connected with the onboard electronic equipment.

8. The in-flight entertainment screen control system according to claim 7, characterized in that: The data acquisition device includes an image acquisition device, a pressure sensor and a voice sensor; Wherein, the image acquisition device is used to acquire the passenger's gesture image and upload it to the onboard electronic device; The pressure sensor is used to collect pressure data representing the aircraft jitter and upload it to the onboard electronic equipment; The voice sensor is used to collect cabin broadcast voice data of the crew and upload it to the onboard electronic equipment.

9. The in-flight entertainment screen control system according to claim 7, characterized in that: The image acquisition device is installed just above the in-flight entertainment display screen and facing the passenger seat; the pressure sensor is installed at the bottom of the image acquisition device.

10. The in-flight entertainment screen control system according to claim 7, characterized in that: The cabin entertainment display screen includes a device control module; The device control module receives the operation control instruction and completes the corresponding screen operation control function according to the instruction; The device control module receives the screen switch instruction and performs the corresponding screen on / off operation according to the instruction; The device control module receives a program playing instruction and performs the corresponding program playing operation according to the instruction; When the device control module receives an error instruction, it will issue an identification failure prompt according to the instruction; When the equipment control module receives the warning instruction, it will perform corresponding aircraft turbulence and pause entertainment control prompts according to the instruction.

Images