Intelligent porthole system based on TOF infrared perception and use method thereof

Abstract

The invention discloses an intelligent porthole system based on TOF infrared sensing and a using method thereof, and relates to the technical field of intelligent homes. In order to overcome the defects that a cruise ship cannot see outdoors, no external natural light exists, an intelligent porthole system cannot flexibly change picture content along with sight movement, and picture change is not matched with a human eye perception angle, an embedded industrial personal computer processes data from a 3D depth camera in real time; the TOF infrared sensor module obtains the sight line of a user and irradiates the eyes of the user through infrared light; the 3D depth sensor captures 3D coordinates of eyes and surrounding feature information; the LED display module displays dynamic natural landscape content according to the sight direction and the fixation point of the user; the offline voice control module adopts an embedded voice recognition system, and a user starts, switches scenes and / or adjusts display parameters through voice commands; the power management module provides power support and power supply management. The device is mainly used for simulating the outdoor natural landscape of the inner cabin of the cruise ship.

Description

Technical Field

[0001] The present invention relates to the technical field of smart home, and particularly to an intelligent porthole system based on TOF infrared sensing and a method for using the same. Background Art

[0002] With the development of modern technology, since there are usually no external windows or natural landscapes in the inner cabins of cruise ships, the user experience may be somewhat limited. Although there are some intelligent porthole systems in the prior art that can simulate outdoor natural landscapes, the viewing angles displayed by the existing intelligent porthole systems cannot dynamically follow the movement of the passengers' heads and cannot change accordingly with the changes in the passengers' line of sight, which easily leads to a mismatch between the picture and the angle perceived by the human eye.

[0003] Therefore, there is a need for an intelligent porthole system based on TOF infrared sensing and a method for using the same that can simulate real pictures, have a high degree of simulation, a high degree of matching with the angle perceived by the human eye, and can change the picture in real time following the line of sight. Summary of the Invention

[0004] In order to solve the defects that existing cruise ships do not have windows to see the outdoors, do not have external natural light, existing intelligent porthole systems cannot flexibly change the picture content following the line of sight, and the picture change does not match the angle perceived by the human eye, the present invention provides an intelligent porthole system based on TOF infrared sensing and a method for using the same that can simulate real pictures, have a high degree of simulation, a high degree of matching with the angle perceived by the human eye, and can change the picture in real time following the line of sight.

[0005] An intelligent porthole system based on TOF infrared sensing according to the present invention includes an embedded industrial computer, a TOF infrared sensor module, a 3D depth sensor, an LED display module, an offline voice control module, and a power management module; The embedded industrial computer is used for real-time processing of data from a 3D depth camera; The TOF infrared sensor module is used for obtaining the user's line of sight by irradiating the user's eyes with infrared light; The 3D depth sensor is used for capturing the 3D coordinates of the eyes and the characteristic information around; The LED display module is used for displaying dynamic natural landscape content according to the user's line of sight direction and fixation point; The offline voice control module adopts an embedded voice recognition system, and the user starts, switches scenes, and / or adjusts display parameters through voice commands; The power management module is used for providing power support and power supply management.

[0006] Further: The TOF infrared sensor emits infrared light towards the user's eye area through the emission module. After the infrared light encounters the eyeball and its surrounding structures, it is reflected. The TOF infrared sensor receives the reflected infrared light signal, calculates the time difference between the emission and the reflection of the light based on the return time of the light, calculates the three-dimensional depth information of the reflection point, and forms the 3D coordinate data of the user's eyes and the surrounding area.

[0007] Further: An eye movement tracking algorithm is set on the embedded industrial computer to identify the pupil position of the customer and calculate the line of sight direction based on the position and angle of the user's head.

[0008] Further: The eye movement tracking algorithm uses image processing technology to identify the user's eye features and reduces noise interference through Kalman filtering or optical flow method.

[0009] Further: The user's eye features include pupil position, eyelid position, eyeball movement trajectory, and fixation point.

[0010] Further: The offline voice control module uses an embedded offline speech recognition engine. The user inputs the user's voice command through the microphone and executes power-on, power-off, scene switching, and display parameter adjustment by parsing the instruction.

[0011] A usage method of the intelligent porthole system based on TOF infrared sensing according to the present invention includes the following steps: S1. Receive the instruction to start the intelligent porthole, and the embedded industrial computer starts initialization; S2. The intelligent porthole system adjusts the display content of the screen in real time according to the user's eye movement information; S3. When receiving the instruction to close the intelligent porthole, the embedded industrial computer stops data collection and turns off the LED display screen; the system enters the standby state and waits for the next startup.

[0012] Further: In S2, the intelligent porthole system adjusts the display content of the screen in real time according to the user's eye movement information, including the following steps: S21. Capture the depth information of the user's eye area in real time and transmit the 3D data to the embedded industrial computer; S22. The embedded industrial computer identifies the pupil position of the user and calculates the line of sight direction based on the position and angle of the user's head; S23. Start dynamically updating the display content according to the user's line of sight direction to ensure that the direction the user is looking at is synchronized with the screen display content; S24. According to the user's eye movement information, the LED display module renders a virtual natural landscape in real time, and the user can switch different scenes through voice commands; S25. The eye movement information of the user is continuously tracked, and the display content of the screen is continuously adjusted according to the user's line of sight direction.

[0013] The beneficial effects of the present invention are as follows: The present invention can display the external real scene in real time and dynamically adjust the perspective of the external scene according to the change of the user's perspective. The embedded industrial control computer described in the present invention has high processing power and reliability, can run for a long time and process a large amount of data from a 3D depth camera in real time, and can smoothly process a large amount of data.

[0014] The TOF infrared sensor module obtains the user's line of sight by irradiating the user's eyes with infrared light, using a depth sensor to capture the 3D coordinates of the eyes and the characteristic information around them, and capturing the line of sight information through infrared light to accurately track the user's line of sight, respond in real time, make the user feel natural, and the interaction is more intuitive.

[0015] The LED display module is used to display dynamic natural landscape content according to the user's line of sight direction and fixation point. The LED display module displays dynamic content according to the line of sight, which can bring an immersive experience, make the scene more real, make the user feel on the scene, and enhance the visual enjoyment.

[0016] The offline voice control module adopts an embedded voice recognition system, and the user can perform operations such as starting, switching scenes, and adjusting display parameters through voice commands. The advantage of the offline voice control module is that it does not require a network, has a fast response, protects privacy, is convenient for user operation, and is particularly useful for voice control when the hands cannot be used.

[0017] The power management module is used to provide power support and power supply management. The power management module can ensure stable power supply, extend the battery life, manage energy consumption, and be environmentally friendly and energy-saving. Description of the Drawings

[0018] Figure 1 It is the overall hardware composition diagram of the intelligent porthole system; Figure 2 It is the overall hardware structure block diagram of the intelligent porthole system; Figure 3 It is the working principle diagram of the TOF infrared sensor module; Figure 4 It is the eye movement tracking scene schematic diagram of the TOF infrared sensor module; Figure 5 It is the functional working flow chart of the intelligent porthole system; Figure 6 It is the real-time rendering of the virtual natural landscape diagram by the LED display module. Detailed Embodiment

[0019] The following are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. The following embodiments are only used to explain the present invention and cannot be construed as a limitation of the present invention. The protection scope of the present invention should be subject to the protection scope of the claims. The embodiments of the present invention are described in detail below. For the convenience of describing the present invention and simplifying the description, the technical terms used in the specification of the present invention should be interpreted in a broad sense, including but not limited to the conventional substitution schemes not mentioned in this application, and including both direct implementation methods and indirect implementation methods.

[0020] Embodiment 1 Combined with Figures 1-6 To illustrate this embodiment, an intelligent porthole system based on TOF infrared sensing disclosed in this embodiment includes an embedded industrial computer, a TOF infrared sensor module, a 3D depth sensor, an LED display module, an offline voice control module, and a power management module; The embedded industrial computer has high processing power and reliability, can run for a long time and process a large amount of data from a 3D depth camera in real time. The embedded industrial computer selects the industrial computer of the Jetson Xavier NX model of NVDIA company, supports multiple interfaces such as USB 3.1, PCIe, HDMI, and MIPI CSI, has the ability to connect and run stably for a long time, can process a large amount of data from the TOF infrared sensor and other modules in real time, and ensures the efficient operation of the system. It can be seamlessly connected to the TOF infrared sensor module, the LED display module, the offline voice control module, etc., and perform operations such as obtaining eye movement information through the TOF infrared sensor module and analyzing it in real time, updating the display content of the LED display module according to the eye movement information, and switching scenarios or adjusting parameters according to the instructions received by the offline voice control module.

[0021] By running a real-time operating system (such as Linux or RTOS), the embedded industrial computer can realize operations such as obtaining eye movement information through the TOF infrared sensor module and analyzing it in real time, updating the display content of the LED display module according to the eye movement information, and switching scenarios or adjusting parameters according to the instructions received by the offline voice control module. As Figure 6 Shown is the LED display module rendering a virtual natural landscape map in real time.

[0022] The TOF infrared sensor module obtains the technical solution of the user's line of sight by irradiating the user's eyes with infrared light and using a depth sensor to capture the 3D coordinates of the eyes and the characteristic information around them.

[0023] The TOF infrared sensor emits infrared light towards the user's eye area through the emission module. After the infrared light encounters the eyeball and its surrounding structures, it is reflected. The TOF infrared sensor receives the reflected infrared light signal, calculates the time difference between the emission and the reflection of the light based on the return time of the light (i.e., the time of flight), calculates the three-dimensional depth information of the reflection point, and forms the 3D coordinate data of the user's eyes and the surrounding area. Extract the key features of the eyes (including the spatial position of the eyeball, pupil position, etc.) from the three-dimensional data captured by the TOF infrared sensor. At the same time, the TOF infrared sensor or other supporting sensing devices detect the pose information of the user's head (such as the deflection angle, tilt, etc.), and combine the head position and pose to further optimize the spatial position data of the eyeball.

[0024] An eye movement tracking algorithm is set on the embedded industrial computer. The eye movement tracking algorithm uses image processing technology to identify the user's eye features, including the pupil and eyelid positions, and reduces noise interference through technologies such as Kalman Filter or Optical Flow to optimize the prediction accuracy and stability of the line of sight direction.

[0025] TOF (Time-of-Flight) technology is a technology that obtains three-dimensional depth information by emitting infrared light and measuring the light reflection time. It has the characteristics of high precision and real-time performance. In the intelligent porthole system described in this embodiment, TOF technology can accurately capture the user's eye movement information, including pupil position, eyeball movement trajectory, fixation point, etc., providing key support for realizing natural interaction and dynamic content display.

[0026] The TOF infrared sensor module meets the following basic performance requirements: 1. The measurement range of the TOF sensor should be between 0.2 meters and 3 meters; 2. The depth resolution of the sensor needs to reach millimeter-level accuracy; 3. The sensor should have a horizontal viewing angle of at least 70° and a vertical viewing angle of 50°; 4. The frame rate of the TOF sensor should be 30 frames per second (fps) or higher; 5. The TOF sensor should be capable of operating in the range of 0 to 100,000 lux; 6. The operating power consumption of the TOF sensor should not exceed 1.5W, and it supports a low-power operation mode; 7. The TOF sensor needs to have an IP65 or higher protection level, and should also have the ability to withstand high temperatures and low temperatures. The recommended operating temperature range is between -20°C and 60°C; 8. The sensor should have I2C, SPI, UART or USB interfaces; 9. The size of the sensor module does not exceed 50x50x25 mm, ensuring that it can be easily embedded into the intelligent porthole system.

[0027] The LED display module is used to display dynamic natural landscape content according to the user's line of sight direction and fixation point. The LED display module adjusts the display content through dynamic rendering technology, and can switch and adjust the landscape picture in real time according to the user's fixation point, making it synchronous with the user's line of sight direction.

[0028] The LED display module includes an LED display screen, and the performance indicators of the display screen are as follows: 1. The resolution is at least 4K UHD (3840x2160); 2. The screen diagonal size is between 40 - 50 inches; 3. The brightness is 600 nits or above; 4. The contrast ratio is at least 5000:1; 5. The viewing angle is 170° or higher; 6. Adopt a low-power design, and the working power does not exceed 100W; 7. IP65 waterproof and dustproof rating; 8. The thickness of the LED screen does not exceed 50 mm for embedding into the cabin wall; 9. The refresh rate is 60Hz or higher.

[0029] The offline voice control module adopts an embedded offline voice recognition engine. Users can input user voice commands through the microphone and perform operations such as power on, power off, scene switching, and display parameter adjustment by parsing the instructions. The offline voice control module operates through localized voice recognition technology, avoiding the upload of user data to the cloud and ensuring user privacy and security.

[0030] The offline voice control module meets the following basic performance: 1. Support automatic switching between main and secondary power supplies, and automatic management of adapters and batteries; 2. The power input interface is DC5521, supporting a wide voltage input of 8V - 28V, with built-in overvoltage, overcurrent, and undervoltage protection functions; 3. Support the opening and closing of 1 mechanical relay, and can connect various types of loads, including AC and DC loads; 4. Provide an industrial computer power management control interface, and the relay node can connect a variety of external devices; 5. The module has overvoltage, overcurrent, and undervoltage protection designs, can work safely in different power environments, is suitable for use in complex cruise ship environments, and ensures long-term stable operation.

[0031] The power management module is used to provide power support and power supply management. The power management module meets the following basic performance requirements: 1. Support wide voltage input (8V - 28V), and the output voltage is stably output at 12V, ensuring a stable power supply for modules such as LED displays and TOF sensors; 2. Integrates an emergency circuit breaker module, which can immediately cut off the power when detecting system short - circuit or current overload, preventing equipment damage and fire risks; 3. Support input over - voltage protection, input under - voltage protection, over - current protection and short - circuit protection. The maximum output current can support 10A, meeting the power supply requirements of high - power devices; 4. Comply with RoHS3 and REACH standards, the moisture sensitivity level (MSL) is 1, and the operating temperature range is from - 20°C to + 60°C, and can operate stably in various environments; 5. The size does not exceed 100x50x25 mm; 6. Meet the ECCN:EAR99 and HTSUS 8542.39.0001 standards, and comply with the relevant safety requirements for international transportation and cruise applications.

[0032] Embodiment 2 Combined with Embodiment 1 and Figure 5 This embodiment describes the usage method of the intelligent porthole system based on TOF infrared sensing disclosed in this embodiment, including the following steps: S1. When starting the porthole by user voice command or button control, the embedded industrial computer starts to initialize all hardware modules, including starting the TOF infrared sensor module, LED display and voice recognition module. The system enters the standby mode, ready to capture the user's eye movement information.

[0033] S2. The TOF infrared sensor module starts to capture the depth information of the user's eye area in real time and transmits the 3D data to the embedded industrial computer. The eye-tracking algorithm on the embedded industrial computer will identify the position of the user's pupils and calculate the line of sight direction based on the position and angle of the user's head. At this time, the system will start to dynamically update the display content according to the user's line of sight direction to ensure that the direction the user is looking at is synchronized with the content displayed on the screen. According to the user's eye movement information, the LED display screen renders a virtual natural landscape in real time, and the user can switch different scenes through voice commands. The user's eye movement information is continuously tracked, and the system continuously adjusts the display content of the screen according to the user's line of sight direction. Among them, the selection of the display scene is realized through button control / voice control, and then the infrared image processing is obtained by capturing real-time data for dynamic tracking; the TOF infrared sensor module first performs face recognition, then extracts eye features, then performs pupil positioning, and finally calculates the line of sight direction, calculates the viewpoint interaction angle according to the obtained user feedback, and finally updates the LED screen to achieve the effect of simulating human eye observation.

[0034] S3. When the user turns off the intelligent porthole system through voice commands or button control, the embedded industrial computer will stop data collection and turn off the LED display screen. The system enters the standby state, waiting for the next startup.

Claims

1. The intelligent porthole system based on TOF infrared sensing is characterized by: Including embedded industrial computers, TOF infrared sensor modules, 3D depth sensors, LED display modules, offline voice control modules and power management modules; The embedded industrial computer is used to process data from the 3D depth camera in real time; The TOF infrared sensor module is used to obtain the user's line of sight by irradiating the user's eyes with infrared light; The 3D depth sensor is used to capture the 3D coordinates of the eyes and surrounding feature information; The LED display module is used to display dynamic natural landscape content according to the user's line of sight and gaze point; The offline voice control module uses an embedded voice recognition system, and the user starts, switches scenes and / or adjusts display parameters through voice commands; The power management module is used to provide power support and power supply management.

2. The intelligent porthole system based on TOF infrared sensing according to claim 1 is characterized in that: The TOF infrared sensor transmits infrared light to the user's eye area through the transmitting module. The infrared light is reflected after encountering the eyeball and its surrounding structures. The TOF infrared sensor receives the reflected infrared light signal and calculates the time difference from the emission to the reflection based on the return time of the light, calculates the three-dimensional depth information of the reflection point, and forms 3D coordinate data of the user's eyes and surrounding areas.

3. The intelligent porthole system based on TOF infrared sensing according to claim 1 is characterized in that: The embedded industrial computer is provided with an eye tracking algorithm for identifying the pupil position of the customer and calculating the sight direction according to the position and angle of the user's head.

4. The intelligent porthole system based on TOF infrared sensing according to claim 3 is characterized in that: The eye tracking algorithm uses image processing technology to identify the user's eye features and reduces noise interference through Kalman filtering or optical flow method.

5. The intelligent porthole system based on TOF infrared sensing according to claim 4 is characterized in that: The user's eye features include pupil position, eyelid position, eye movement trajectory and gaze point.

6. The intelligent porthole system based on TOF infrared sensing according to claim 1 is characterized in that: The offline voice control module adopts an embedded offline voice recognition engine. The user inputs user voice commands through a microphone, and the module performs power on, power off, scene switching and display parameter adjustment by parsing the commands.

7. A method for using the smart porthole system based on TOF infrared sensing according to any one of claims 1 to 6, characterized in that: The steps include: S1. Receive the command to start the intelligent porthole, and the embedded industrial computer starts initialization; S2, the intelligent porthole system adjusts the display content of the screen in real time according to the user's eye movement information; S3. When receiving the command to close the smart porthole, the embedded industrial computer will stop data collection and turn off the LED display; the system will enter standby mode and wait for the next startup.

8. The method for using the smart porthole system based on TOF infrared sensing according to claim 7, characterized in that: In S2, the smart porthole system adjusts the display content of the screen in real time according to the user's eye movement information, including the following steps: S21, capturing the depth information of the user's eye area in real time, and transmitting the 3D data to the embedded industrial computer; S22, the embedded industrial computer identifies the user's pupil position and calculates the sight direction according to the position and angle of the user's head; S23, dynamically updating the displayed content according to the user's gaze direction, ensuring that the direction the user is looking at is synchronized with the screen display content; S24, based on the user's eye movement information, the LED display module renders a virtual natural landscape in real time, and the user can switch between different scenes through voice commands; S25. The user's eye movement information is continuously tracked, and the screen display content is continuously adjusted according to the user's line of sight.

Images