A multi-modal interactive LED display
By integrating sensors such as cameras, voice modules, gesture and touch modules into LED displays and adopting a distributed control system, the collaborative processing of multimodal data is achieved, solving the problem of insufficient interactive capabilities of LED displays in artificial intelligence applications and improving the level of intelligence and data transmission stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Utility models(China)
- Current Assignee / Owner
- SHENZHEN HAOYUN TECHNOLOGY CO LTD
- Filing Date
- 2025-07-30
- Publication Date
- 2026-06-26
AI Technical Summary
When combined with artificial intelligence technology, existing LED displays lack multimodal interaction capabilities and struggle to effectively integrate various interaction modalities such as vision, voice, touch, and gestures. This results in insufficient intelligent interaction capabilities, compatibility issues, and unstable data processing.
Design a multimodal interactive LED display screen that uses multiple sensors such as cameras, voice modules, gestures, and touch modules to collect data, and processes and displays the data through a distributed control system of sending and receiving cards. It integrates FPGA main control chip, MCU chip, etc., supports multimodal signal input and output, and realizes collaborative processing of multiple interaction modes.
It enhances the intelligence level of LED displays, enabling a more natural and efficient human-computer interaction experience. It solves problems such as insufficient compatibility, space and wiring limitations, multimodal data processing and transmission delay, ensuring the stability and accuracy of data transmission.
Smart Images
Figure CN224417282U_ABST
Abstract
Description
Technical Field
[0001] This utility model relates to the field of LED display technology, and specifically to a multimodal interactive LED display. Background Technology
[0002] LED displays are one of the most important carriers of human-computer interaction. Artificial intelligence (AI) technology plays an increasingly important role in the display industry. By using these technologies, display devices can better understand users' behavior, emotions, and intentions, thereby providing a more personalized and intelligent experience. However, for LED displays to integrate AI, they need to possess multimodal interaction capabilities. This involves integrating multiple interaction modalities such as vision, voice, touch, and gestures to build an intelligent perception and interaction system based on the display. As the core carrier of human-computer interaction, the integration of multimodal recognition technology to enhance the intelligent interaction capabilities of LED displays has widespread demand in exhibition halls, reception areas, and other fields. Utility Model Content
[0003] The purpose of this invention is to provide a multimodal interactive LED display screen to solve the problems mentioned above.
[0004] The objective of this utility model can be achieved through the following technical solutions:
[0005] A multimodal interactive LED display screen is characterized in that it includes a plurality of LED light boards, which are fixedly mounted on an LED cabinet made of die-cast aluminum. The LED cabinet is provided with reinforcing plates and reinforcing ribs, and the LED cabinet is provided with a locking structure with magnetic attraction function.
[0006] Several receiver cards are installed on the back of the LED light panel, and a camera is installed above the LED display screen. The LED display screen is fixed on the metal bracket by a mounting bracket.
[0007] The LED display screen uses cameras, voice modules, gestures, touch modules, and several sensors to collect multimodal interactive data, which is then transmitted to the display screen control system for calculation and processing, and finally sent to the display screen for display presentation.
[0008] As a further embodiment of this utility model: the display screen control system includes a sending card and several receiving cards, and realizes signal processing, distribution and display control through a high-speed data transmission link. The sending card sends data to each receiving card, and then distributes it to each driver chip. The sending card is connected to the receiving card on each LED display screen cabinet through a DVI, HDMI or DP interface; after data processing, it is then sent to the display screen for display.
[0009] As a further embodiment of this utility model: the sending card connects to the host or server, and integrates an input interface, an FPGA main control chip, an MCU and a sensor, supporting multi-modal signal input;
[0010] The transmitting card receives data from the camera, voice module, various sensors, host and server, and sends the data to the receiving card via DVI, HDMI or DP. The receiving card transmits data to the driver chip via FPD-Link or SPI protocol to drive the LED module pixels to emit light.
[0011] As a further embodiment of this utility model: the sending card integrates an FPGA main control chip, an MCU chip, a storage chip, an input / output interface chip, a power supply chip, and a clock module.
[0012] As a further embodiment of this utility model: the FPGA main control chip is responsible for high-speed data processing, realizing pixel-level signal processing, supporting multi-channel parallel data operation, meeting the real-time requirements of high-resolution and high-refresh-rate displays, and being compatible with multiple input interface protocols such as HDMI, DP, PCIe, and USB.
[0013] The MCU chip is used to control sensors and human-machine interaction commands. The FPGA main control chip and the MCU chip are connected via an SPI interface.
[0014] As a further embodiment of this utility model: the FPGA main control chip is connected to the camera via the MIPI-CSI protocol, and the camera is located in the center above the display screen for face recognition and motion recognition;
[0015] The system communicates with the voice module via the I2S protocol. The voice module consists of an audio digital signal processing chip, a power amplifier chip, an audio conversion chip, a speaker, and a microphone, and is used for voice recognition. It is connected to Ethernet via a PHY chip for network communication and to a temperature sensor and power module via an I2C bus. The temperature sensor is used to monitor the operating temperature of the chip and the LED module.
[0016] As a further embodiment of this utility model: the host or server includes a storage unit and a computing platform for storing, computing, transmitting and processing uploaded data, and communicating with the sending card through a PCIe, HDMI or DP interface.
[0017] The receiver card and driver chip are integrated inside the LED display screen. The receiver card is used to receive data from the display screen control card, and the LED constant current driver chip is used to drive the LED unit board.
[0018] As a further embodiment of this invention: the sending card sends data to the receiving card via a DVI, HDMI, or DP interface. The receiving cards are deployed in a distributed manner, and each receiving card controls a portion of the LED display module, integrating an FPGA main control chip, a driver chip, a sensor, and a power module. Data parsing and driving are achieved through FPD-Link / SPI.
[0019] As a further embodiment of this utility model: the receiver card consists of an FPGA main control chip, an MCU chip, a storage module, a clock module, a grayscale control chip, a horizontal and vertical driving chip, a data parsing interface, and a driving interface;
[0020] The receiver card integrates infrared and pressure sensors for touch and gesture sensing, and integrates temperature and brightness sensors for temperature monitoring and brightness adjustment, which can dynamically adjust display parameters, such as automatically reducing brightness according to temperature.
[0021] As a further solution of this utility model: several receiving cards are cascaded through the RS485 communication protocol to achieve data transmission, and the failure of a single receiving card does not affect the whole system.
[0022] The beneficial effects of this utility model are:
[0023] By integrating multiple interaction modalities such as vision, voice, touch, and gesture, a control system for LED displays that supports multimodal interaction is realized. Its core objective is to enhance the intelligence level of the display through the collaborative processing and fusion of multi-source data, achieving a more natural and efficient human-computer interaction experience. Through the integrated design of the LED display, multimodal data input / output, display control system, and communication solutions, problems such as insufficient compatibility, space and wiring limitations, multimodal data processing, environmental interference, and transmission delays encountered when implementing multimodal interaction on LED displays are addressed, thereby improving the intelligent interaction capabilities of the LED display and ensuring the stability and accuracy of data transmission. Attached Figure Description
[0024] The present invention will be further described below with reference to the accompanying drawings.
[0025] Figure 1 This is a diagram of the multimodal display screen control system in this utility model;
[0026] Figure 2 This is a schematic diagram of the display screen structure in this utility model;
[0027] Figure 3 This is a connection diagram of the display screen control card system architecture in this utility model.
[0028] In the diagram: 1. LED light board; 2. LED cabinet; 21. Reinforcing plate; 22. Reinforcing rib; 23. Locking structure; 3. Receiver card; 4. Camera; 5. Mounting bracket. Detailed Implementation
[0029] The technical solutions of the present utility model will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present utility model, and not all embodiments. Based on the embodiments of the present utility model, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present utility model.
[0030] Please see Figures 1-3 As shown, this utility model is a multimodal interactive LED display screen. It uses a camera 4, a voice module, a gesture, a touch module, and several sensors to collect multimodal interactive data, transmit it to the display screen control system for calculation and processing, and finally send it to the display screen for display presentation.
[0031] The display screen control system includes a sending card and several receiving cards 3. The sending card sends data to each receiving card 3, which then distributes it to each driver chip. The sending card is connected to the receiving cards 3 on each LED display screen cabinet via DVI / HDMI / DP interfaces. Figure 3 As shown; alternatively, video source data can be sent directly from the cloud platform or storage device to the display control system, processed, and then sent to the display screen for display.
[0032] Furthermore, modeling data (such as digital humans) can be created based on 3D modeling tools (such as Blender), built into the display control card, and combined with neural network rendering technology (such as Epic's MetaHuman) to achieve realistic expressions and movements, such as... Figure 1 As shown.
[0033] Multimodal interaction integrates various sensors, camera modules, cross-modal gesture information acquisition devices, and displays, and integrates them onto an LED display screen through a control system. Figure 1 As shown, it enables immersive interactive functions such as gesture recognition, voice recognition, motion capture, and touch control.
[0034] like Figure 3 As shown, the structure consists of multiple LED light panels 1 fixed on multiple die-cast aluminum LED cabinets 2. Reinforcing plates 21 and reinforcing ribs 22 are installed on the LED cabinets 2 to ensure structural strength and flatness. A magnetic locking structure 23 is provided to ensure the flatness and security of the cabinet splicing and the ease of installation on a metal bracket (not shown in the figure). The multiple LED light panels 1 are encapsulated using COB (chip on board) and potting compound to form a highly protective LED display screen.
[0035] Multiple receiver cards 3 are set on the back of the LED light board 1 to receive data from the transmitting card and multimodal interaction data, and to collect data from temperature, brightness, infrared and pressure sensors.
[0036] A camera 4 is integrated above the LED display screen for functions such as face recognition and motion recognition. The entire LED display screen is mounted on a metal bracket via a magnetic structure and secured by a mounting bracket 5. The receiver card 3 integrates an FPGA (Field-Programmable Gate Array) main control chip, an MCU chip, a grayscale control chip, a horizontal and vertical drive chip, and temperature, brightness, infrared, and pressure sensors. The FPGA chip receives and parses incoming video and audio data packets, extracts pixel data, and maps it to the LED matrix coordinates.
[0037] The MCU chip controls each sensor and issues control commands, assisting in the management of input / output interfaces and local storage. The grayscale control chip converts the digital signals output by the FPGA into PWM grayscale signals, achieving 256 levels (8 bits) or higher grayscale display effects. The row and column driver chips expand the pixel data into a row and column matrix, driving the on / off state of the corresponding LED beads, including scan drive or static drive modes.
[0038] Temperature sensors are used to monitor the display temperature, while brightness sensors are used for intelligent adjustment of the display brightness. Infrared and pressure sensors are used together for touch and gesture interaction on the display. Liveness tracking is achieved by combining dynamic 3D feature analysis based on video acquisition and infrared feature point tracking. This is accomplished using a flexible array sensor that can recognize electromyographic signals of different gestures. The system consists of an array sensor module composed of infrared and pressure sensors and a computing module.
[0039] The display control system includes a sending card, multiple receiving cards 3, and multiple driver chips. The system data transmission method is as follows: Figure 3 As shown, this control system is mainly used for distributed control of LED displays. It consists of two core modules: a sending card and a receiving card 3. Signal processing, distribution, and display control are achieved through a high-speed data transmission link. The sending card connects to the host / server and integrates input interfaces, an FPGA main control chip, an MCU, and sensors, supporting multi-modal signal input (video, audio, sensor data). The sending card receives data from camera 4, the audio module, various sensors, the host, and the server, and transmits the data to the receiving card 3 via DVI / HDMI / DP. The receiving card 3 transmits data to the driver chip via FPD-Link or SPI protocol to drive the LED module pixels to emit light.
[0040] The transmitting card integrates an FPGA main control chip, an MCU chip, a memory chip, an input / output interface chip, a power supply chip, and a clock module. The FPGA main control chip is responsible for high-speed data processing, implementing pixel-level signal processing (such as format conversion, scaling, color gamut matching, and multi-screen splicing synchronization), supporting multi-channel parallel data processing, meeting the real-time requirements of high-resolution (e.g., 4K / 8K) and high refresh rate (e.g., 3840Hz) displays, and is compatible with multiple input interface protocols such as HDMI, DP, PCIe, and USB. The MCU chip is used for controlling sensors and human-machine interaction commands. The FPGA main control chip and the MCU chip are connected via an SPI interface.
[0041] The FPGA main control chip connects to camera 4 via the MIPI-CSI protocol. Camera 4 is located at the top center of the display screen and is used for face recognition and motion recognition. It communicates with the voice module via the I2S protocol. The voice module consists of an audio digital signal processing (DSP) chip, a power amplifier chip, an audio conversion chip, a speaker, and a microphone, and is used for voice recognition. A PHY chip connects to Ethernet for network communication, and an I2C bus connects to a temperature sensor and a power module. The temperature sensor monitors the operating temperature of the chip and LED modules. The host or server includes a storage unit and a computing platform for storing, computing, transmitting, and processing uploaded data. It communicates with the sending card via PCIe / HDMI / DP interfaces. Receiver card 3 and the driver chip are integrated inside the LED display screen. Receiver card 3 receives data from the display screen control card, and the LED constant current driver chip drives the LED unit boards. The sending card sends data to receiver card 3 via DVI / HDMI / DP interfaces. Receiver cards 3 are distributed, with each receiver card 3 controlling a portion of the LED display modules. Each receiver card 3 integrates an FPGA main control chip, a driver chip, a sensor, and a power module, and uses FPD-Link / SPI for data parsing and driving.
[0042] Receiver Card 3 consists of an FPGA main control chip, an MCU chip, a storage module, a clock module, a grayscale control chip, a horizontal and vertical drive chip, a data parsing interface, and a drive interface. Infrared and pressure sensors are integrated on Receiver Card 3 for touch and gesture sensing, while temperature and brightness sensors are integrated for temperature monitoring and brightness adjustment, dynamically adjusting display parameters such as automatically reducing brightness based on temperature. Multiple Receiver Cards 3 can be cascaded via the RS485 communication protocol for data transmission, supporting ultra-large-scale LED displays; a single card failure does not affect the entire system.
[0043] The LED display screen achieves multimodal interaction by collecting video, voice, and sensor data from various sources through a voice module, camera 4, and sensor modules. This data is then transmitted to a sending card via an input communication protocol. The sending card calculates and processes the collected data before sending it to each receiving card 3. Each receiving card 3 converts the data format and transmits it to the LED driver chip to drive each pixel of the LED to emit light. This enables immersive interactive functions such as gesture recognition, voice recognition, motion capture, and touch control through the collection, processing, and transmission of multimodal data.
[0044] This invention integrates multiple interaction modalities such as vision, voice, touch, and gesture to realize a control system for an LED display screen that can support multimodal interaction;
[0045] By integrating LED displays, multimodal data input / output, display control systems, and communication solutions, this approach addresses issues such as insufficient compatibility, space and wiring limitations, multimodal data processing, environmental interference, and transmission delays that arise when LED displays achieve multimodal interaction.
[0046] It supports mainstream video interfaces such as HDMI / DVI / PCIe, is compatible with multimedia devices such as cameras and audio devices, adapts to complex scenarios, and its two-way communication link supports remote diagnostics and parameter configuration, reducing operation and maintenance costs.
[0047] The control system achieves efficient control of the LED display screen through a distributed architecture and the integration of multiple technologies, balancing performance and intelligence.
[0048] By integrating multiple interaction modalities such as vision, voice, touch, and gesture, a control system for LED displays that supports multimodal interaction is realized. Its core objective is to enhance the intelligence level of the display through the collaborative processing and fusion of multi-source data, achieving a more natural and efficient human-computer interaction experience. Through the integrated design of the LED display, multimodal data input / output, display control system, and communication solutions, problems such as insufficient compatibility, space and wiring limitations, multimodal data processing, environmental interference, and transmission delays encountered when implementing multimodal interaction on LED displays are addressed, thereby improving the intelligent interaction capabilities of the LED display and ensuring the stability and accuracy of data transmission.
[0049] The above description provides a detailed account of one embodiment of the present invention. However, this description is merely a preferred embodiment and should not be construed as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the claims of the present invention should still fall within the patent coverage of the present invention.
Claims
1. A multimodal interactive LED display screen, characterized in that, It includes several LED light boards (1), several LED light boards (1) are fixedly installed on an LED box (2) made of die-cast aluminum material, the LED box (2) is provided with a reinforcing plate (21) and a reinforcing rib (22), and the LED box (2) is provided with a lock structure (23) with magnetic attraction function. The LED light panel (1) has several receiver cards (3) on its back, and a camera (4) is installed above the LED display screen. The LED display screen is fixed on a metal bracket by a fixing frame (5). The LED display screen uses cameras, voice modules, gestures, touch modules, and several sensors to collect multimodal interactive data, which is then transmitted to the display screen control system for calculation and processing, and finally sent to the display screen for display presentation.
2. The multimodal interactive LED display screen according to claim 1, characterized in that, The display screen control system includes a sending card and several receiving cards (3). Signal processing, distribution and display control are realized through a high-speed data transmission link. The sending card sends data to each receiving card (3) and then distributes it to each driver chip. The sending card is connected to the receiving card (3) on each LED display screen cabinet through DVI, HDMI or DP interface. After data processing, it is then sent to the display screen for display.
3. The multimodal interactive LED display screen according to claim 2, characterized in that, The sending card connects to the host or server, integrating an input interface, FPGA main control chip, MCU, and sensors, and supports multi-modal signal input; The transmitting card receives data from the camera, voice module, various sensors, host and server, and sends the data to the receiving card via DVI, HDMI or DP. The receiving card transmits data to the driver chip via FPD-Link or SPI protocol to drive the LED module pixels to emit light.
4. The multimodal interactive LED display screen according to claim 3, characterized in that, The sending card integrates an FPGA main control chip, an MCU chip, a memory chip, an input / output interface chip, a power supply chip, and a clock module.
5. A multimodal interactive LED display screen according to claim 4, characterized in that, The FPGA main control chip is responsible for high-speed data processing, realizing pixel-level signal processing, supporting multi-channel parallel data operation, meeting the real-time requirements of high-resolution and high-refresh-rate displays, and is compatible with multiple input interface protocols such as HDMI, DP, PCIe and USB. The MCU chip is used to control sensors and human-machine interaction commands. The FPGA main control chip and the MCU chip are connected via an SPI interface.
6. A multimodal interactive LED display screen according to claim 5, characterized in that, The FPGA main control chip is connected to the camera via the MIPI-CSI protocol. The camera is located in the center above the display screen and is used for face recognition and motion recognition. The voice module communicates with the audio digital signal processing (DSP) chip via the I2S protocol. The voice module consists of an audio digital signal processing (DSP) chip, a power amplifier chip, an audio conversion chip, a speaker, and a microphone, and is used for voice recognition. It is connected to Ethernet via a PHY chip for network communication and to a temperature sensor and power module via an I2C bus. The temperature sensor is used to monitor the operating temperature of the sending card chip and the LED module.
7. A multimodal interactive LED display screen according to claim 6, characterized in that, The host or server includes storage units and a computing platform for storing, computing, transmitting and processing uploaded data, and communicates with the sending card via PCIe, HDMI or DP interfaces; The receiver card and driver chip are integrated inside the LED display screen. The receiver card is used to receive data from the display screen control card, and the LED constant current driver chip is used to drive the LED unit board.
8. A multimodal interactive LED display screen according to claim 7, characterized in that, The sending card sends data to the receiving card (3) via DVI, HDMI or DP interface. The receiving cards (3) are deployed in a distributed manner. Each receiving card (3) controls a part of the LED display module and integrates an FPGA main control chip, a driver chip, a sensor and a power module. Data parsing and driving are achieved through FPD-Link / SPI.
9. A multimodal interactive LED display screen according to claim 8, characterized in that, The receiver card (3) consists of an FPGA main control chip, an MCU chip, a storage module, a clock module, a grayscale control chip, a horizontal and vertical drive chip, a data parsing interface, and a drive interface; The receiver card (3) integrates an array sensor module consisting of infrared and pressure sensors for touch and gesture sensing, and integrates temperature and brightness sensors for temperature monitoring and brightness adjustment of the LED display screen. It can dynamically adjust display parameters, such as automatically reducing brightness according to temperature.
10. A multimodal interactive LED display screen according to claim 9, characterized in that, Data transmission is achieved by cascading several receiver cards (3) through the RS485 communication protocol. The driver chip is connected to a single receiver card via FPD-Link / SPI. A failure of a single receiver card does not affect the whole system.