Multi-modal based target perception method, display device, and storage medium

Abstract

The embodiment of the application discloses a target perception method based on multi-modal, a display device and a storage medium. The embodiment of the application obtains the perception result of the target object corresponding to each perception module by analyzing the channel state information scanned by the WiFi module group and the auxiliary perception information collected by the auxiliary perception module, and finally determines the final perception result of the target object according to the perception result corresponding to each perception module. The embodiment of the application can improve the accuracy of target object perception by using at least two modal perception information, solves the technical problem that the single sensor perception method in the prior art is easy to trigger, and improves the user experience. In addition, the embodiment of the application only needs to use the existing multi-modal information of the display device, without the need to increase additional sensors, and can further reduce the cost.

Description

Technical Field

[0001] This application relates to the field of wireless communication technology, and in particular to a multimodal target sensing method, display device, and storage medium. Background Technology

[0002] Currently, to enhance the intelligence of terminal devices, many devices are equipped with human body sensing capabilities. This allows them to provide seamless services, such as screen activation or device-wide interaction, when a user is detected in the target area or when the user approaches, thereby reducing the need for manual operation. In existing technologies, terminal devices typically sense the human body by analyzing data collected from a single sensor. Commonly used sensors include infrared sensors, millimeter-wave radar, and ultrasonic radar.

[0003] However, existing technologies that use a single sensor to sense the human body have a technical problem of being prone to false triggering, resulting in a poor user experience. Summary of the Invention

[0004] This application provides a multimodal target perception method, display device, and storage medium, which solves the technical problem of easy false triggering in the prior art method of perceiving the human body through a single sensor.

[0005] In a first aspect, embodiments of this application provide a multimodal target perception method. The method is applicable to a display device, which includes multiple types of sensing modules. These sensing modules include a WiFi module group and an auxiliary sensing module. The auxiliary sensing module includes at least one of a sound acquisition module, a Bluetooth module, and an ambient light acquisition module. The WiFi module group includes two WiFi modules. The method includes:

[0006] The channel status information scanned by the WiFi module group and the auxiliary sensing information collected by the auxiliary sensing module are obtained.

[0007] The channel state information and the auxiliary sensing information are processed respectively to obtain the sensing information of the target object corresponding to each sensing module;

[0008] The perception result of the target object is determined based on multiple pieces of the perception information.

[0009] This application embodiment obtains channel state information scanned by the WiFi module group on the display device and auxiliary sensing information collected by the auxiliary sensing module. By analyzing the channel state information and auxiliary sensing information separately, it obtains the perception result of the target object corresponding to each sensing module. Finally, based on the perception results corresponding to each sensing module, it determines the final perception result of the target object. This application embodiment uses at least two modalities of sensing information to perceive the target object, which improves the accuracy of target object perception and solves the technical problem of easy false triggering in existing technologies that use a single sensor to perceive the human body, thus improving the user experience. Furthermore, this application embodiment, based on multimodal information, offers superior stability and robustness in target object perception, is applicable to complex usage scenarios, and only requires utilizing the existing multimodal information of the display device without adding additional sensors, further reducing costs.

[0010] The auxiliary sensing module includes a sound acquisition module, a Bluetooth module, and an ambient light acquisition module. The process of processing the channel state information and the auxiliary sensing information to obtain sensing information of the target object corresponding to each sensing module includes:

[0011] The channel state information, the sound information collected by the sound acquisition module, the Bluetooth beacon information scanned by the Bluetooth module, and the light intensity information collected by the ambient light acquisition module are processed respectively to obtain the perception information of the target object corresponding to each of the perception modules.

[0012] As described above, the embodiments of this application improve the accuracy of target object perception in complex environments by collecting information from four modes: channel state information, sound information, Bluetooth beacon information, and light intensity information.

[0013] The process of processing the channel state information, the sound information acquired by the sound acquisition module, the Bluetooth beacon information scanned by the Bluetooth module, and the light intensity information acquired by the ambient light acquisition module to obtain the perception information of the target object corresponding to each perception module includes:

[0014] The channel state information is processed to determine the first location information of the target object in the environment, and the first sensing information corresponding to the WiFi module group is obtained.

[0015] The sound information is processed to determine the second location information of the target object in the environment and the first number of the target object, thereby obtaining the second perception information corresponding to the sound acquisition module;

[0016] The Bluetooth beacon information is processed to determine the second number of target objects in the environment, and to obtain third sensing information corresponding to the Bluetooth module;

[0017] The light intensity information is analyzed and processed to determine the light sensitivity change rate, thereby obtaining the fourth sensing information corresponding to the ambient light acquisition module.

[0018] As described above, the embodiments of this application determine the perception information corresponding to different perception modules, so as to realize the perception of the target object based on the perception information of different perception modules, which significantly improves the perception accuracy, anti-interference ability and scene generalization performance of the target object in complex environments.

[0019] The step of processing the Bluetooth beacon information to determine the second number of target objects in the environment and obtain third sensing information corresponding to the Bluetooth module includes:

[0020] Based on the Bluetooth beacon information, the signal strength of the terminal devices scanned by the Bluetooth module is determined;

[0021] Determine that the signal strength is greater than the preset signal strength and that there is a target terminal device with a duration greater than the preset duration;

[0022] Based on the number of target terminal devices, a second number of target objects in the environment is determined, and third sensing information corresponding to the Bluetooth module is obtained.

[0023] As described above, the embodiments of this application can effectively eliminate short-term interference signals (such as passing equipment) and environmental noise by setting preset signal strength and preset duration, ensuring that only stable target terminal devices are counted, thereby improving the accuracy of target object perception.

[0024] The step of determining the perception result of the target object based on multiple pieces of perception information includes:

[0025] Determine the confidence level corresponding to each of the sensing modules;

[0026] Based on the confidence level, the weight of the sensing information corresponding to each sensing module is determined;

[0027] The perception result of the target object is determined based on the perception information and weight corresponding to each of the perception modules.

[0028] As described above, the embodiments of this application can significantly improve the accuracy, robustness, and scene adaptability of perceiving target objects through confidence assessment, dynamic weight allocation, and multimodal fusion.

[0029] The perception result of the target object also includes whether the target object is within a preset distance, and the method further includes:

[0030] When the screen is on and the target object is within a preset distance, the corresponding menu control is displayed on the display interface;

[0031] When the screen is on and the target object is not within a preset distance, the menu control is hidden in the display interface.

[0032] As described above, in the embodiment of this application, when the display device is in the screen-on state, if the target object is determined to be within a preset distance based on the perception result, the corresponding menu control is displayed on the display interface; if the target object moves out of the preset distance, the menu control is hidden on the display interface. This embodiment of the application simplifies the user's operation steps, improves the intelligence level of the display device, and enhances the user experience.

[0033] The display device includes at least one WiFi module group, with two WiFi modules in each WiFi module group symmetrically arranged in the display device, and the radiation direction of each WiFi module is the same as the orientation of the front of the display device.

[0034] As described above, this embodiment of the application, by symmetrically arranging two WiFi modules of the WiFi module group on the display device, and with the radiation direction of the WiFi modules being the same as the orientation of the front of the display device, allows the electromagnetic wave energy of the WiFi modules to be concentrated and cover the front area of ​​the display device. When a human body is moving directly in front of the display device, the influence of body movements on the WiFi signal reflection path is maximized and captured, significantly enhancing the amplitude and phase change characteristics related to human movement in the channel state information. Simultaneously, it allows multipath signals generated by human activity (such as gestures and movement) to form a stronger superposition or cancellation effect with direct signals, improving the detectability of dynamic changes in channel state information.

[0035] The ambient light acquisition module includes at least one ambient light sensor, each of which is disposed at a different position on the display device, and the orientation of the light incident surface of each ambient light sensor is the same as the orientation of the front of the display device.

[0036] As described above, the embodiments of this application achieve good light intensity detection by placing each ambient light sensor at a different position on the display device, and ensuring that the orientation of the light incident surface of each ambient light sensor is the same as the orientation of the front of the display device.

[0037] Secondly, embodiments of this application provide a display device, which includes a processor, a memory, and various types of sensing modules. The types of sensing modules include a WiFi module group and an auxiliary sensing module. The auxiliary sensing module includes at least one of a sound acquisition module, a Bluetooth module, and an ambient light acquisition module. The WiFi module group includes two WiFi modules.

[0038] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0039] The processor is configured to execute a multimodal target perception method as described in the first aspect, according to instructions in the computer program.

[0040] Thirdly, embodiments of this application provide a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform a multimodal target perception method as described in the first aspect. Attached Figure Description

[0041] Figure 1 A flowchart of a multimodal target perception method provided in this application embodiment.

[0042] Figure 2 A flowchart of another multimodal target perception method provided in this application embodiment.

[0043] Figure 3 This is a front view of a display device provided in an embodiment of this application.

[0044] Figure 4 This is a schematic diagram illustrating the radiation direction of a WiFi module provided in an embodiment of this application.

[0045] Figure 5 This is a schematic diagram showing the location of an ambient light sensor on a display device, as provided in an embodiment of this application.

[0046] Figure 6 This is a schematic diagram showing the location of another ambient light sensor on a display device, as provided in an embodiment of this application.

[0047] Figure 7 This is a schematic diagram illustrating the output of perception results through a deep learning model, as provided in an embodiment of this application.

[0048] Figure 8 This is a schematic diagram of the structure of a display device provided in an embodiment of this application. Detailed Implementation

[0049] The following description and accompanying drawings fully illustrate specific embodiments of this application to enable those skilled in the art to practice them. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of embodiments of this application includes the entire scope of the claims and all available equivalents of the claims. In this document, each embodiment may be referred to individually or collectively by the term "invention," which is merely for convenience and is not intended to automatically limit the scope of the application to any single invention or inventive concept if more than one invention is disclosed. Relational terms such as "first" and "second" are used herein only to distinguish one entity or operation from another, without requiring or implying any actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed. The various embodiments in this document are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the structures, products, etc., disclosed in the embodiments, since they correspond to the disclosed parts, the descriptions are relatively simple; relevant details can be found in the method section.

[0050] Currently, to enhance the intelligence of terminal devices, many devices are equipped with human body sensing capabilities. This allows them to provide seamless services, such as screen activation or device-wide interaction, when a user is detected in the target area or when the user approaches, thereby reducing the need for manual operation. In existing technologies, terminal devices typically sense the human body by analyzing data collected by a single sensor. Commonly used sensors include infrared sensors, millimeter-wave radar, and ultrasonic radar. The principles behind human body sensing using different sensors are as follows:

[0051] (1) Infrared sensor

[0052] Infrared sensors can employ pyroelectric infrared sensors, which detect objects by utilizing infrared radiation emitted by the human body or other objects. Specifically, a pyroelectric infrared sensor contains a pyroelectric element. When infrared light shines on this element, it causes a temperature change, leading to a redistribution of electrical charges and generating an electrical signal. By measuring the changes in this electrical signal, the sensor can detect human presence. To detect infrared radiation, pyroelectric infrared sensors are typically equipped with Fresnel lenses or mirrors. These lenses or mirrors concentrate infrared radiation from a distance, improving the sensor's sensitivity and detection range. While pyroelectric infrared sensors can identify the heat signal radiated by the human body, the presence of a mirror can affect the product's appearance design in practical applications. Furthermore, their detection angle (±60°) and distance (3-5M) are relatively limited.

[0053] (2) Millimeter-wave radar

[0054] Millimeter-wave radar typically detects human movement and presence using electromagnetic waves with frequencies between 30 and 300 GHz. Its basic principle is that the radar transmits millimeter-wave signals to the target area and then receives the reflected waves. Since human movement causes changes in the frequency and phase of the reflected waves—a change known as Doppler shift—the motion information of the human body, such as its position, speed, and respiratory rate, can be analyzed by measuring this Doppler shift.

[0055] (3) Ultrasonic radar

[0056] When ultrasonic radar detects human presence, it primarily does so by measuring the distance to objects within the target area. When a change in distance to an object within the target area is detected, the presence of a human body can be determined. This distance detection is based on measuring the time difference between the transmission and reception of ultrasonic waves.

[0057] However, when using millimeter-wave radar or ultrasonic radar for human detection, neither can distinguish between non-biological movement, making them prone to false triggering. Furthermore, millimeter-wave and ultrasonic radars have strict structural requirements; structural tolerances can affect the consistency of trigger thresholds, necessitating calibration during use. Additionally, millimeter-wave radar is relatively expensive.

[0058] Therefore, to address the technical problem of false triggering in existing technologies that rely on a single sensor to perceive the human body, this application provides a multimodal target perception method. This method uses multiple modal perception information to perceive the target object, improving the accuracy of target object perception and resolving the false triggering issue inherent in existing single-sensor methods, thus enhancing the user experience. Furthermore, the multimodal information-based target object perception method of this application offers superior stability and robustness, making it suitable for complex usage scenarios.

[0059] like Figure 1 As shown, Figure 1 This document provides a flowchart of a multimodal target perception method according to an embodiment of this application. The multimodal target perception method provided in this embodiment is applicable to display devices, which can be televisions, educational blackboards, conference tablets, and smart interactive tablets, etc. A smart interactive tablet integrates a display screen, projector, electronic whiteboard, speakers, conference terminal, camera, and omnidirectional microphone into one intelligent interactive device. Display devices typically have functions such as whiteboard writing, high-definition screen projection, and video conferencing. The display device includes various types of perception modules, where the perception module is a module capable of realizing target object perception. The target object can be set according to actual needs; for example, the target object can be a human body or a pet. This embodiment uses a human body as an example for specific explanation. In this embodiment, the perception module types include WiFi module groups and auxiliary perception modules. The auxiliary perception module includes at least one of a sound acquisition module, a Bluetooth module, and an ambient light acquisition module. The WiFi module group includes two WiFi modules. The WiFi module is used to realize data transmission between the device and the wireless network. The auxiliary sensing module is used to assist in the perception of target objects. Within this module, the sound acquisition module collects sound information, converts it into electrical signals, and outputs digital audio data via analog-to-digital converter (ADC) for speech recognition and ambient sound monitoring. The Bluetooth module supports short-range, low-power communication for data and audio transmission between devices, suitable for smart wearables and near-field communication. The ambient light acquisition module detects ambient light intensity. The types of units included in the auxiliary sensing module can be configured according to actual needs; this embodiment does not impose specific limitations. Furthermore, since display devices generally include WiFi, sound acquisition, Bluetooth, and ambient light acquisition modules, this embodiment can reuse these modules as sensing modules, fully utilizing the existing hardware resources of the display device without adding additional hardware, thus further reducing costs.

[0060] The multimodal target perception method provided in this application includes the following steps:

[0061] Step 101: Obtain the channel status information scanned by the WiFi module group and the auxiliary sensing information collected by the auxiliary sensing module.

[0062] The system acquires channel status information scanned by the WiFi module group, sound information acquired by the sound acquisition module, Bluetooth beacon information scanned by the Bluetooth module, and light intensity information acquired by the ambient light acquisition module.

[0063] The display device needs to simultaneously acquire Channel State Information (CSI) scanned by the WiFi module group and auxiliary sensing information collected by the auxiliary sensing module. The auxiliary sensing information is related to the target object collected by the auxiliary sensing module. Channel State Information (CSI) is a crucial parameter describing the characteristics of the wireless channel, including the amplitude, phase, and signal delay of each subcarrier. Specifically, one WiFi module in the WiFi module group acts as the WiFi signal transmitter (RF-TX), and the other as the WiFi signal receiver (RF-RX). The transmitter module transmits a series of known signals within a specific target frequency range at certain step sizes, while the receiver module receives signals within the same frequency range, recording the received signal strength, phase, and other information. During reception, the receiver module continuously adjusts its receiving frequency to cover the entire target frequency range, acquiring signal characteristics at different frequencies. Based on the changes in the received signal at different frequencies, it analyzes the channel state information, including channel gain, noise level, and interference. Finally, the state information is summarized to form the Channel State Information (CSI).

[0064] When the auxiliary sensing module includes a sound acquisition module, the auxiliary sensing information includes sound information acquired by the sound acquisition module; when the auxiliary sensing module includes a Bluetooth module, the auxiliary sensing information includes Bluetooth beacon information scanned by the Bluetooth module; when the auxiliary sensing module includes an ambient light acquisition module, the auxiliary sensing information includes light intensity information acquired by the ambient light acquisition module. Sound information is a signal including sound waveforms in the environment. Bluetooth beacon information is wireless signal data periodically transmitted by the Bluetooth beacon through the Bluetooth Low Energy Broadcast protocol, used to identify location and support positioning calculations. The Bluetooth beacon information contains the MAC address information of the corresponding device. Light intensity information is a signal including the light intensity in the environment. It is understood that, theoretically, the higher the sampling rate of each sensing module, the better. In one embodiment, the sampling frequency for channel state information is 30Hz-100Hz, the sampling frequency for Bluetooth beacon information is 10Hz-30Hz, the sampling frequency for light intensity information is 5Hz-10Hz, and the sampling frequency for sound information is 44100Hz-48000Hz.

[0065] Step 102: Process the channel state information and auxiliary sensing information respectively to obtain the sensing information of the target object corresponding to each sensing module.

[0066] The display device needs to further process the channel state information and auxiliary sensing information separately to obtain the sensing information of the target object corresponding to each sensing module. The sensing information is used to identify whether a target object is sensed. For example, channel state information is typically used in Wi-Fi communication to measure the channel state of a wireless network. However, channel state information is highly sensitive to environmental changes. Therefore, by analyzing changes in channel state information, the physical environmental changes causing these changes can be inferred, enabling contactless intelligent sensing. For instance, analyzing channel state information can not only sense changes caused by large movements such as walking and running of people or animals in the environment, but also capture subtle movements of people or animals in static environments, such as breathing and chewing. Therefore, by analyzing channel state information, the sensing information of the human body corresponding to the Wi-Fi module group can be obtained. As for auxiliary sensing information, the display device can analyze it to assist in the judgment of target object perception. For example, sound information can be used to analyze the presence of a target object; Bluetooth beacon information can be used to analyze changes in the number of Bluetooth devices in the environment to determine changes in the number of target objects; and light intensity information can be used to sense whether a target object is approaching. In one embodiment, the auxiliary sensing module includes a sound acquisition module, a Bluetooth module, and an ambient light acquisition module, which process channel state information and auxiliary sensing information respectively to obtain sensing information of the target object corresponding to each sensing module, including:

[0067] The channel state information, the sound information collected by the sound acquisition module, the Bluetooth beacon information scanned by the Bluetooth module, and the light intensity information collected by the ambient light acquisition module are processed respectively to obtain the perception information of the target object corresponding to each perception module.

[0068] Specifically, in the process of analyzing auxiliary sensing information, for the sound information collected by the sound acquisition module, human voices and non-human voices can be distinguished. Therefore, the display device can identify the presence of a human body by analyzing whether human voices are present in the sound information, thus obtaining the human body perception information corresponding to the sound acquisition module. For the Bluetooth beacon information scanned by the Bluetooth module, the Received Signal Strength Indication (RSSI) of the Bluetooth beacon information can be analyzed to determine whether there are new terminal devices in the surrounding environment. If so, it is determined that a person has brought the device into the surrounding environment, thus obtaining the human body perception information corresponding to the Bluetooth module. For the light intensity information collected by the ambient light acquisition module, the light sensitivity value can be calculated based on the light intensity, and the rate of change of the light sensitivity value can be used to determine whether a human body is approaching, thus obtaining the human body perception information corresponding to the sound acquisition module.

[0069] Step 103: Determine the perception result of the target object based on multiple perception information.

[0070] After obtaining the perception information corresponding to each perception module, the display device can determine the final perception result of the target object based on the perception information corresponding to each perception module. The perception result is used to characterize the presence, quantity, posture, and position of a human body. For example, different confidence levels can be assigned to the perception results corresponding to different perception modules, and the perception results corresponding to different perception modules can be fused based on DS evidence theory to obtain the final perception result of the human body. Alternatively, different weights can be assigned to different perception information, and the final perception result of the human body can be determined through weighted voting. In another embodiment, the user can pre-train an object perception model. The object perception model is used to implement steps 102-103. After collecting channel state information and auxiliary perception information, the display device preprocesses the channel state information and auxiliary perception information and inputs them into the object perception model, which then outputs the perception result of the target object.

[0071] As described above, this embodiment of the application obtains the channel state information scanned by the WiFi module group on the display device and the auxiliary sensing information collected by the auxiliary sensing module. By analyzing the channel state information and the auxiliary sensing information respectively, the perception result of the target object corresponding to each sensing module is obtained. Finally, based on the perception results corresponding to each sensing module, the final perception result of the target object is determined. This embodiment of the application, by perceiving the target object using at least two modalities of sensing information, can improve the accuracy of target object perception and solve the technical problem of easy false triggering in the prior art's method of perceiving the human body using a single sensor, thus improving the user experience. Furthermore, this embodiment of the application, based on multimodal information perception of the target object, has better stability and robustness, is applicable to complex usage scenarios, and only requires the use of the existing multimodal information of the display device, without the need for additional sensors, further reducing costs.

[0072] This application also provides another target perception method based on multimodal modes, such as... Figure 2 As shown, Figure 2 A flowchart of another multimodal target perception method provided in this application embodiment. Figure 2 The multimodal target perception method shown is a specific embodiment of the above-mentioned multimodal target perception method. In this embodiment, an auxiliary perception module including a sound acquisition module, a Bluetooth module, and an ambient light acquisition module is used as an example for illustration. The multimodal target perception method provided in this embodiment includes the following steps:

[0073] Step 201: Obtain the channel status information scanned by the WiFi module group, the sound information collected by the sound acquisition module, the Bluetooth beacon information scanned by the Bluetooth module, and the light intensity information collected by the ambient light acquisition module.

[0074] Step 202: Process the channel state information to determine the first location information of the target object in the environment and obtain the first sensing information corresponding to the WiFi module group.

[0075] When processing channel state information, the display device can filter and denoise the information, extract joint time-frequency features from the denoised channel state information, and input these features into a pre-constructed LSTM (Long Short-Term Memory) network. After the LSTM network outputs continuous trajectory coordinates, these coordinates are smoothed using Gaussian kernels to construct a gridded spatial probability distribution matrix. A heatmap is then generated through color mapping, and finally, the presence and initial location of a human body are determined using the heatmap. It can be understood that if the channel state information determines that no target object exists in the environment, it is unnecessary to determine the target object's initial location.

[0076] Furthermore, to achieve good sensing results using channel state information, the positions and radiation directions of the two WiFi modules in a WiFi module group need to be set according to design requirements. In one embodiment, the display device includes at least one WiFi module group, with two WiFi modules in each group symmetrically arranged within the display device, and the radiation direction of each WiFi module being the same as the orientation of the front of the display device. The radiation direction of each WiFi module refers to the spatial distribution characteristics of the electromagnetic wave energy of the WiFi module's antenna, which can be quantified using a radiation pattern. The fact that the WiFi module's radiation direction is the same as the orientation of the front of the display device means that the maximum radiation direction (main lobe) of the WiFi module's antenna is aligned with the orientation of the front of the display device, concentrating electromagnetic wave energy to cover the front of the display device, thereby improving the accuracy of subsequent human perception. For example, as shown... Figure 3 As shown, Figure 3 This is a front view of the display device provided in the embodiment of this application. The display device includes three WiFi module groups. In each WiFi module group, one WiFi module acts as a WiFi signal transmitting module (RF-TX), and the other WiFi module acts as a WiFi signal receiving module (RF-RX). Figure 3 In this configuration, the transmitting module (RF-TX) and receiving module (RF-RX) of each WiFi module group are symmetrically arranged on the display device, and the radiation direction of each transmitting module (RF-TX) and receiving module (RF-RX) faces the front of the interaction plane 10, as shown below. Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the radiation direction of a WiFi module according to an embodiment of this application. Furthermore, the number of WiFi module groups can be appropriately increased or decreased based on cost and performance requirements; this embodiment does not specifically limit the number of WiFi module groups.

[0077] As described above, this embodiment of the application, by symmetrically arranging two WiFi modules of the WiFi module group on the display device, and with the radiation direction of the WiFi modules being the same as the orientation of the front of the display device, allows the electromagnetic wave energy of the WiFi modules to be concentrated and cover the front area of ​​the display device. When a human body is moving directly in front of the display device, the influence of body movements on the WiFi signal reflection path is maximized and captured, significantly enhancing the amplitude and phase change characteristics related to human movement in the channel state information. Simultaneously, it allows multipath signals generated by human activity (such as gestures and movement) to form a stronger superposition or cancellation effect with direct signals, improving the detectability of dynamic changes in channel state information.

[0078] Step 203: Process the sound information to determine the second location information of the target object in the environment and the first quantity of the target object, and obtain the second perception information corresponding to the sound acquisition module.

[0079] In this embodiment, the sound acquisition module can be a microphone array. When processing the sound information, the display device can distinguish between human voices and non-human voices, extract the voiceprint features of human voices, and use a voiceprint clustering algorithm to distinguish different sound source individuals, thus counting the first number of human figures. Simultaneously, for the sound information acquired by the microphone array, a second location information of the sound source can be calculated using a time-difference-of-arrival (TDOA) algorithm. This TDOA algorithm determines the location of the sound source based on the time difference between the arrival times of the same sound at different microphones in the array and the distance between each pair of microphones. It is understood that if the sound information determines that no target object exists in the environment, it is unnecessary to determine the second location information of the target object or the first number of target objects.

[0080] Step 204: Process the Bluetooth beacon information to determine the second number of target objects in the environment and obtain the third perception information corresponding to the Bluetooth module.

[0081] The display device can periodically scan the Bluetooth beacon information broadcast by surrounding terminal devices via the Bluetooth module. When processing the Bluetooth beacon information, it can analyze the Received Signal Strength Indication (RSSI) and determine whether the received signal strength exceeds a preset signal strength. If it does, it determines that a new terminal device exists in the surrounding environment, indicating that a person has brought the new terminal device into the environment. The second number of human beings is updated based on the count of new terminal devices. In one embodiment, step 204 processes the Bluetooth beacon information to determine whether a target object exists in the environment and the second number of target objects, obtaining third sensing information corresponding to the Bluetooth module, including:

[0082] Step 2041: Determine the signal strength of the terminal device scanned by the Bluetooth module based on the Bluetooth beacon information.

[0083] Step 2042: Determine a target terminal device whose signal strength is greater than the preset signal strength and whose duration is greater than the preset duration.

[0084] Step 2043: Based on the number of target terminal devices, determine the second number of target objects in the environment, and obtain the third sensing information corresponding to the Bluetooth module.

[0085] In one embodiment, the Bluetooth beacon information contains the MAC address information corresponding to the terminal device. When processing the Bluetooth beacon information, the display device can save the MAC address information. When the Bluetooth beacon information determines that a target terminal device exists with a signal strength greater than a preset signal strength and a duration greater than a preset duration, it is determined that a new terminal device exists in the surrounding environment, and that a person has brought the new terminal device into the surrounding environment. Therefore, the second number of human beings can be updated based on the determined number of target terminal devices. Furthermore, it is understood that the preset signal strength and preset duration can be set according to actual needs, and this embodiment does not impose specific limitations. It is also understood that if the Bluetooth beacon information determines that no target object exists in the environment, it is not necessary to determine the second number of target objects.

[0086] As described above, the embodiments of this application can effectively eliminate short-term interference signals (such as passing equipment) and environmental noise by setting preset signal strength and preset duration, ensuring that only stable target terminal devices are counted, thereby improving the accuracy of target object perception.

[0087] Step 205: Analyze and process the light intensity information, determine the light sensitivity change rate, and obtain the fourth sensing information corresponding to the ambient light acquisition module.

[0088] For light intensity information, the display device can calculate the light sensitivity change rate based on the light intensity information and determine whether a human body is present in the vicinity based on the light sensitivity change rate. The light sensitivity change rate refers to the dynamic change of light sensitivity parameters per unit time, typically used to describe the rate at which light intensity changes with time or environmental conditions. Specifically, under sunlight, the light sensitivity change rate is relatively low. If there is an obstruction within the acquisition range of the ambient light acquisition module, the light sensitivity change rate will increase, indicating that an object is moving, and there is a high probability that a person is approaching the display device, thus confirming the presence of a human body. In one embodiment, to achieve good detection results, the ambient light acquisition module includes at least one ambient light sensor. Each ambient light sensor is positioned at a different location on the display device, and the orientation of the light incident surface of each ambient light sensor is the same as the orientation of the front of the display device. For example, as shown... Figure 5 As shown, Figure 5 This is a schematic diagram illustrating the location of an ambient light sensor on a display device according to an embodiment of this application. The ambient light acquisition module includes three ambient light sensors (LS). One ambient light sensor (LS) is disposed in each of the side bezels and the bottom bezel of the display device 10 to ensure the accuracy of light intensity sensing. In another embodiment, as shown... Figure 6 As shown, Figure 6 This is a schematic diagram illustrating the location of another ambient light sensor on a display device, as provided in an embodiment of this application. Figure 6In this embodiment, the ambient light acquisition module includes four ambient light sensors (LS). One ambient light sensor (LS) is located on each of the two side bezels of the display device 10, and two ambient light sensors (LS) are located on the bottom bezel. It is understood that the number of ambient light sensors can be increased or decreased appropriately based on cost and performance requirements; this embodiment does not impose a specific limit on the number of ambient light sensors.

[0089] As described above, the embodiments of this application determine the perception information corresponding to different perception modules, so as to realize the perception of the target object based on the perception information of different perception modules, which significantly improves the perception accuracy, anti-interference ability and scene generalization performance of the target object in complex environments.

[0090] Step 206: Determine the confidence level corresponding to each sensing module.

[0091] In this embodiment, the display device first needs to determine the confidence level corresponding to each sensing module. In one embodiment, for the confidence level of the WiFi module group, a noise baseline can be established in advance based on historical channel state information, and the offset between the current channel state information and the noise baseline can be compared in real time. The smaller the offset, the higher the confidence level. For the confidence level of the sound acquisition module, spectral subtraction can be used to filter background noise, calculate the proportion of effective sound segments to the total duration (e.g., >70%), and determine the confidence level based on the signal-to-noise ratio (SNR). The higher the SNR, the higher the confidence level. For the Bluetooth module, the probability of the Bluetooth beacon's presence in three consecutive scans can be statistically analyzed. The confidence level is set based on the RSSI intensity attenuation rate and the probability of presence of the Bluetooth beacon information. The smaller the RSSI intensity attenuation rate and the larger the probability of presence, the higher the confidence level. For the ambient light acquisition module, the standard deviation of light intensity within one second can be statistically analyzed. The larger the standard deviation, the more valid the change, and the higher the confidence level. After determining the confidence level for different sensing information, the confidence level is normalized and set within the range [0,1].

[0092] Step 207: Determine the weight of the sensing information corresponding to each sensing module based on the confidence level.

[0093] After determining the confidence level, the display device needs to determine the weight of the sensing information corresponding to each sensing module based on the confidence level corresponding to each sensing module. In one embodiment, the sensing information corresponding to each sensing module has a preset basic weight ratio (e.g., 30% for the WiFi module group, 25% for the sound acquisition module, 25% for the Bluetooth module, and 20% for the ambient light acquisition module), reflecting the prior importance of different sensing modules in different scenarios. Then, the weight of the sensing information corresponding to each sensing module needs to be further adjusted based on the confidence level of each sensing module. For example, if the confidence level of a certain sensing module is lower than the confidence level threshold, its weight is reduced, and the weights of other sensing modules are increased.

[0094] Step 208: Determine the perception result of the target object based on the perception information and weight corresponding to each perception module.

[0095] Finally, the perception result of the target object can be determined based on the perception information and weight corresponding to each perception module. In one embodiment, the perception result (such as "target object exists" / "target object does not exist") can be voted on based on the weight of the perception information corresponding to each perception module. For example, if the perception information corresponding to the WiFi module group and the perception information corresponding to the Bluetooth module both detect the existence of a target object and the sum of their weights is >50%, then the perception result is determined to be the existence of a target object.

[0096] As described above, the embodiments of this application can significantly improve the accuracy, robustness, and scene adaptability of perceiving target objects through confidence assessment, dynamic weight allocation, and multimodal fusion.

[0097] In another embodiment, the user can also pre-train a deep learning model. After obtaining the perceptual information corresponding to different perception modules, the user inputs the perceptual information corresponding to different perception modules into the pre-trained deep learning model. The deep learning module then outputs the final perceptual result of the target object, such as... Figure 7 As shown, Figure 7 This is a schematic diagram illustrating the output of perception results through a deep learning model, as provided in an embodiment of this application.

[0098] In one embodiment, it further includes:

[0099] Step 209: If the target object exists and the screen is currently off, perform the screen-on operation.

[0100] In this embodiment, the perception result of the target object includes whether a human body is present. If the display device determines that a human body is present based on the perception result, and the current display screen is in a screen-off state, then the display device needs to perform a screen-on operation. Here, the screen-off state means the display screen is turned off, and the screen-on operation is used to activate the display device's display screen to display an image. It is understood that if the current display screen is already in a screen-on state, then there is no need to perform the screen-on operation.

[0101] Step 2010: If the target object does not exist and the screen is currently on, perform a screen-off operation.

[0102] If the display device determines that there is no human presence and the screen is currently on, then the display device needs to perform a screen-off operation. Here, "screen on" means the display is powered on, and "screen-off" means turning off the display. It's understandable that if the screen is not currently on, then a screen-off operation is unnecessary.

[0103] As described above, the embodiments of this application simplify the user's operation steps, improve the intelligence level of the display device, and reduce the energy consumption of the display device by performing a screen-on operation when a target object is present and the screen is currently off, and performing a screen-off operation when no target object is present and the screen is currently on.

[0104] In one embodiment, the perception result of the target object further includes whether the target object is within a preset distance, and the method further includes:

[0105] Step 211: When the screen is on and the target object is within a preset distance, display the corresponding menu control on the display interface.

[0106] Step 212: When the screen is on and the target object is not within the preset distance, hide the menu control in the display interface.

[0107] In one embodiment, the perception result of the target object also includes whether the target object is within a preset distance, where the preset distance is a pre-set distance between the target object and the front of the display device. The preset distance can be set according to actual needs, for example, the preset distance can be set to 30cm. In one embodiment, whether the target object is within the preset distance can be determined by the light sensitivity change rate. For example, when the display device detects that the light sensitivity change rate is greater than a certain preset threshold, it can be identified that a human body is approaching the display screen. At this time, the human body is within the preset distance, and the display device displays the corresponding menu control on the display screen's display interface. The menu control is a component used for user interaction, such as a sidebar, to facilitate user operation. When the display device detects that the light sensitivity change rate is less than the preset threshold, it can be identified that the human body is moving away from the display screen and is not within the preset distance. The display device can hide the displayed menu control on the display screen's display interface.

[0108] In another embodiment, channel state information, sound information, Bluetooth beacon information, and light intensity information can be integrated to perceive the human body's position, thereby determining the distance between the human body and the front of the display device. For example, after making a preliminary judgment on the human body's position based on information corresponding to multiple modalities, the judgment results corresponding to each modality can be fused, such as using voting methods or Bayesian fusion, to obtain a fused human body position result. A deep learning model, such as a Long Short-Term Memory (LSTM) network or a Convolutional Neural Network (CNN), can then be used to process the fused human body position result. The fused human body position result is then input into a trained deep learning model to obtain the final human body position prediction result, and based on the final prediction result, it is determined whether the human body is within a preset distance. During the training of the deep learning model, a labeled human body position dataset can be used to train the model. By adjusting the parameters of the deep learning model, the prediction error can be minimized, improving the accuracy and generalization ability of the deep learning model.

[0109] As described above, in the embodiments of this application, when the display device is in a screen-on state, if the target object is determined to be within a preset distance based on the perception result, the corresponding menu control is displayed; if the target object moves out of the preset distance, the menu control is hidden. These embodiments simplify user operation steps, improve the intelligence level of the display device, and enhance the user experience.

[0110] Figure 8 This is a schematic diagram of the structure of the display device provided in the embodiments of this application, such as... Figure 8As shown, the display device includes a processor 300, a memory 301, an input device 302, an output device 303, and various types of sensing modules 304. The types of sensing modules 304 include WiFi module groups and auxiliary sensing modules. The auxiliary sensing modules include at least one of a sound acquisition module, a Bluetooth module, and an ambient light acquisition module. The WiFi module groups include two WiFi modules.

[0111] The number of processors 300 in the display device can be one or more. Figure 8 Taking a processor 300 as an example; in a display device, the processor 300, memory 301, input device 302, output device 303, and sensing module 304 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0112] The memory 301, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the multimodal target perception method in the embodiments of this application. The processor 300 executes various functional applications and data processing of the display device by running the software programs, instructions, and modules stored in the memory 301, thereby realizing the aforementioned multimodal target perception method.

[0113] The memory 301 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the display device. Furthermore, the memory 301 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 301 may further include memory remotely located relative to the processor 300, which can be connected to the display device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0114] Input device 302 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the display device. Output device 303 may include a display screen or other display device. In addition, the display device may also include a communication device to enable communication with external devices.

[0115] The aforementioned display device can be used to execute any multimodal target perception method, possessing corresponding functions and beneficial effects.

[0116] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and equipment can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0117] Furthermore, embodiments of this application also provide a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform relevant operations in the multimodal target perception method provided in any embodiment of this application, and have corresponding functions and beneficial effects.

[0118] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0119] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0120] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0121] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0122] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0123] The above specific embodiments have further detailed the purpose, technical solution, and beneficial effects of this application. It should be understood that the above are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application for those skilled in the art.

Claims

1. A target perception method based on multimodal sensing, characterized in that, The method is applicable to a display device, which includes multiple types of sensing modules. The sensing modules include a WiFi module group and an auxiliary sensing module. The auxiliary sensing module includes at least one of a sound acquisition module, a Bluetooth module, and an ambient light acquisition module. The WiFi module group includes two WiFi modules. The method includes: The channel status information scanned by the WiFi module group and the auxiliary sensing information collected by the auxiliary sensing module are obtained. The channel state information and the auxiliary sensing information are processed respectively to obtain the sensing information of the target object corresponding to each sensing module; The perception result of the target object is determined based on multiple pieces of the perception information.

2. The multimodal target perception method according to claim 1, characterized in that, The auxiliary sensing module includes a sound acquisition module, a Bluetooth module, and an ambient light acquisition module. The channel state information and the auxiliary sensing information are processed respectively to obtain sensing information of the target object corresponding to each sensing module, including: The channel state information, the sound information collected by the sound acquisition module, the Bluetooth beacon information scanned by the Bluetooth module, and the light intensity information collected by the ambient light acquisition module are processed respectively to obtain the perception information of the target object corresponding to each of the perception modules.

3. The multimodal target perception method according to claim 2, characterized in that, The process involves processing the channel state information, the sound information acquired by the sound acquisition module, the Bluetooth beacon information scanned by the Bluetooth module, and the light intensity information acquired by the ambient light acquisition module to obtain the perception information of the target object corresponding to each perception module, including: The channel state information is processed to determine the first location information of the target object in the environment, and the first sensing information corresponding to the WiFi module group is obtained. The sound information is processed to determine the second location information of the target object in the environment and the first number of the target object, thereby obtaining the second perception information corresponding to the sound acquisition module; The Bluetooth beacon information is processed to determine the second number of target objects in the environment, and to obtain third sensing information corresponding to the Bluetooth module; The light intensity information is analyzed and processed to determine the light sensitivity change rate, thereby obtaining the fourth sensing information corresponding to the ambient light acquisition module.

4. The multimodal target perception method according to claim 3, characterized in that, The process of processing the Bluetooth beacon information to determine the second number of target objects in the environment and obtain third sensing information corresponding to the Bluetooth module includes: Based on the Bluetooth beacon information, the signal strength of the terminal devices scanned by the Bluetooth module is determined; Determine that the signal strength is greater than the preset signal strength and that there is a target terminal device with a duration greater than the preset duration; Based on the number of target terminal devices, a second number of target objects in the environment is determined, and third sensing information corresponding to the Bluetooth module is obtained.

5. The multimodal target perception method according to any one of claims 1 to 4, characterized in that, Determining the perception result of the target object based on multiple pieces of perception information includes: Determine the confidence level corresponding to each of the sensing modules; Based on the confidence level, the weight of the sensing information corresponding to each sensing module is determined; The perception result of the target object is determined based on the perception information and weight corresponding to each of the perception modules.

6. The multimodal target perception method according to any one of claims 1 to 4, characterized in that, The perception result of the target object also includes whether the target object is within a preset distance, and the method further includes: When the screen is on and the target object is within a preset distance, the corresponding menu control is displayed on the display interface; When the screen is on and the target object is not within a preset distance, the menu control is hidden in the display interface.

7. The multimodal target perception method according to any one of claims 1 to 4, characterized in that, The display device includes at least one WiFi module group, with two WiFi modules in each WiFi module group symmetrically arranged in the display device, and the radiation direction of each WiFi module is the same as the orientation of the front of the display device.

8. The multimodal target perception method according to any one of claims 1 to 4, characterized in that, The ambient light acquisition module includes at least one ambient light sensor, each of which is disposed at a different position on the display device, and the orientation of the light incident surface of each ambient light sensor is the same as the orientation of the front of the display device.

9. A display device, characterized in that, The display device includes a processor, a memory, and various types of sensing modules. The types of sensing modules include WiFi module groups and auxiliary sensing modules. The auxiliary sensing modules include at least one of a sound acquisition module, a Bluetooth module, and an ambient light acquisition module. The WiFi module group includes two WiFi modules. The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to execute a multimodal target perception method as described in any one of claims 1 to 8 according to instructions in the computer program.

10. A storage medium for storing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform a multimodal target perception method as described in any one of claims 1 to 8.

Images