LED display screen interaction method and LED display screen interaction system

Abstract

The invention relates to the technical field of man-machine interaction, and discloses an LED display screen interaction method and an LED display screen interaction system.The method comprises the following steps that user interaction data are collected through a multi-sensor module, and the user interaction data comprise visual data, touch data, infrared data and environment data; preprocessing the collected multi-modal user interaction data, including data noise reduction, feature extraction and unified coding; performing time sequence modeling on the multi-modal user interaction data based on the dynamic data fusion model to generate a fusion state; and identifying the user interaction intention based on the user interaction intention prediction model. Through the technical scheme of combining multi-modal data fusion and deep learning, all-around acquisition and comprehensive perception of user interaction information are realized by collecting visual, touch, infrared and environmental data of a user, and the defects of limited data dimension, poor environmental adaptability and insufficient interaction precision in single-modal interaction are overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to an LED display screen interaction method and an LED display screen interaction system. Background Art

[0002] With the rapid development of smart display technology and human-computer interaction needs, LED displays have gradually become an important medium for advertising, education and entertainment. However, most of the current LED displays can only achieve simple content display and single-mode interaction, which is difficult to meet users' requirements for multi-modal, real-time, and intelligent interactive experience. In complex application scenarios, the diversity of user operations and the dynamic changes in environmental conditions make the limitations of existing technologies more obvious;

[0003] Existing LED display screen interaction technologies are usually based on single-mode interaction technologies, such as gesture recognition through cameras, plane operation through touch screens, and distance sensing through infrared sensors. The basic principle of these technologies is to use a single sensor to collect user interaction data, match the data with predefined interaction rules, and then trigger a simple switch of screen content or adjustment of display mode;

[0004] However, the existing interactive technology of LED display screens only relies on data collection of a single mode and cannot fully perceive the user's interactive behavior, resulting in poor system adaptability. In addition, the static fusion method in the existing technology lacks the ability to model the dynamic changes of time series data and cannot capture the continuity characteristics of the user's operation process. In a noisy environment, the existing technology fails to effectively handle the uncertainty of multimodal data, which can easily lead to system misjudgment or response delays. The feedback control of the existing system mostly adopts fixed rules and lacks real-time optimization capabilities, resulting in poor response speed and accuracy in complex scenarios. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides an LED display screen interaction method and an LED display screen interaction system, which solve the problems that the prior art LED display screen interaction technology only relies on data collection of a single mode and cannot fully perceive the user's interaction behavior, resulting in poor system adaptability. In addition, the static fusion method in the prior art lacks the ability to model the dynamic changes of time series data and cannot capture the continuity characteristics of the user's operation process in a noisy environment, resulting in poor response speed and accuracy in complex scenarios.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: an LED display screen interactive method and an LED display screen interactive system, comprising the following steps:

[0007] Collecting user interaction data through a multi-sensor module, wherein the user interaction data includes visual data, touch data, infrared data and environmental data;

[0008] Preprocess the collected multimodal user interaction data, including data noise reduction, feature extraction and unified coding;

[0009] Based on the dynamic data fusion model, the multimodal user interaction data is modeled in time series to generate the fusion state;

[0010] Identify user interaction intention based on user interaction intention prediction model;

[0011] The display content and interactive feedback of the LED display are adjusted in real time according to the identified user interaction intentions.

[0012] Preferably, the collection of the user interaction data includes:

[0013] The user's gesture trajectory and facial expression are acquired through the camera, and the acquisition result is time series image data;

[0014] The touch sensor collects the coordinates and pressure information of the user's touch point, and the collection results are two-dimensional coordinate data and pressure values;

[0015] The distance and spatial position between the user and the LED display screen are acquired through the infrared sensor, and the acquisition results are the distance value and three-dimensional spatial coordinates;

[0016] Collect light brightness and sound characteristic data through environmental sensors.

[0017] Preferably, the pretreatment comprises:

[0018] Use convolution filtering to reduce noise on visual data;

[0019] Wavelet transform is used to reduce noise on touch, infrared and environmental data;

[0020] The preprocessed multimodal data is uniformly encoded through a variational autoencoder (VAE) to generate a high-dimensional latent representation.

[0021] Preferably, the dynamic data fusion model includes:

[0022] Use recurrent neural networks (RNNs) to capture temporal dependencies of multimodal data;

[0023] The attention mechanism is used to assign weights to multimodal data. The attention weight calculation formula is:

[0024]

[0025] in, is the weight of the i-th modal data, W i is the learning parameter, h t is the hidden state, ∑ j exp(W j h t ) represents the sum of the weighted results of all modal data after taking the exponential value, and exp represents the natural exponential function.

[0026] Preferably, the dynamic data fusion model is combined with a state transition model and a Bayesian filter for recursive updating, including:

[0027] Predict the fusion state based on the state transition model:

[0028] X t =AX t-1 +BU t +η t

[0029] Among them, X t represents the fusion state at the current time t, X t-1 represents the fusion state at the previous time t-1, A is the state transfer matrix, B is the control matrix, U t is the external input variable of the system at time t, η t is the process noise;

[0030] The fusion state is updated using a Bayesian filter, and the accuracy of the current fusion state is optimized through the posterior probability distribution.

[0031] Preferably, the dynamic data fusion model optimizes the time series modeling process of multimodal data through a recursive update mechanism, comprising the following steps:

[0032] Based on the time series characteristics of multimodal data, the posterior probability distribution of the fusion state is calculated using the current fusion state and new observation data;

[0033] In the recursive update process, the prediction-update mechanism of Bayesian filtering is combined to dynamically optimize the fusion state, where the observation model is used to associate the current observation data with the fusion state, and the state transition model is used to predict the next fusion state.

[0034] By dynamically adjusting the observation weights and state prediction weights, the influence of observation noise can be suppressed and the stability and robustness of multimodal data fusion can be improved.

[0035] Preferably, the user interaction intention prediction adopts a deep neural network model and uses a Softmax function for classification, including:

[0036] P(intent|X t )=Softmax(WX t +b)

[0037] Among them, X t represents the high-dimensional feature vector generated by the dynamic data fusion model at time t, and W represents the fusion state X in the model t The linear mapping relationship between each interaction intention category, b represents the independent offset of each interaction intention category, P(intent|X t ) represents the probability of a specific interaction intention.

[0038] Preferably, the user interaction includes:

[0039] Control the switching of LED display screen content according to gesture data;

[0040] Adjust the enlargement or reduction of the LED display screen content according to the touch data;

[0041] Dynamically adjust display brightness based on user distance data provided by infrared sensors;

[0042] Optimizes the visual quality and responsiveness of display content based on data provided by environmental sensors.

[0043] Preferably, the interaction feedback optimizes the interaction strategy through a deep reinforcement learning model, wherein:

[0044] Take the fused multimodal state as input;

[0045] The adjustment of the LED display content is used as output;

[0046] The reward function is used to minimize system latency and maximize the response accuracy of user interactions. The specific expression is:

[0047] R=λ1·Response accuracy-λ2·Delay

[0048] Among them, R represents the comprehensive performance score of the system in an interaction process, λ1 is the response accuracy weight coefficient, λ2 is the system delay weight coefficient, response accuracy is the degree to which the LED display interactive system correctly responds to the user's interactive input, and delay is the time interval from the system receiving the user input to completing the interactive feedback.

[0049] Preferably, including:

[0050] Multi-sensor module for collecting user interaction data, including visual data, touch data, infrared data and environmental data;

[0051] Data preprocessing module, used for noise reduction and feature extraction of multimodal data;

[0052] Dynamic data fusion module, used to perform high-dimensional time series modeling on multimodal data and generate fusion states;

[0053] User intention prediction module, used to identify the user's interaction intention;

[0054] Real-time feedback control module, used to adjust the content display and response behavior of the LED display according to the user's interaction intention

[0055] The invention provides an LED display screen interactive method and an LED display screen interactive system.

[0056] It has the following beneficial effects:

[0057] 1. The present invention uses a technical solution combining multimodal data fusion and deep learning to collect the user's visual, touch, infrared and environmental data to achieve all-round acquisition and comprehensive perception of user interaction information. Compared with the technical solution in the prior art that only relies on a single modality (such as visual or touch) data for interaction, it solves the defects of limited data dimension, poor environmental adaptability and insufficient interaction accuracy in single-modal interaction.

[0058] 2. The present invention adopts a dynamic data fusion model based on recurrent neural network (RNN) and attention mechanism, combined with the modeling method of multimodal time series, and dynamically adjusts the modal contribution through attention weights to achieve accurate modeling of user intentions. Compared with the technical solution in the prior art that simply relies on static feature fusion, it solves the shortcomings that the dependency relationship of time series data cannot be effectively captured and the details of user operations cannot be fully reflected.

[0059] 3. The present invention realizes uncertainty processing and real-time optimization of user interaction data in the dynamic data fusion model by combining Bayesian filtering and recursive update mechanism. Through recursive update, combined with the prediction-update mechanism of observation model and state transition model, the robustness and dynamic response capability of multimodal data fusion are effectively improved. Compared with the technical solutions in the prior art that do not handle the dynamic changes and noise problems of data, the problems of poor stability, feedback hysteresis and insufficient precision of interactive data are solved.

[0060] 4. The present invention optimizes the real-time feedback control mechanism through deep reinforcement learning, directly links the user's interaction intention with the adjustment of the display content of the LED display, and dynamically optimizes the feedback strategy in combination with the reward function, which significantly improves the real-time response and interaction matching of the system. Compared with the technical solution of adjusting the display content with fixed rules in the prior art, it solves the problems of high interactive response delay, inflexible operation and poor adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flow chart of the method of the present invention;

[0062] Figure 2 is a system architecture diagram of the present invention; DETAILED DESCRIPTION

[0063] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] The present invention provides an LED display screen interaction method, which realizes intelligent LED display screen interaction based on user behavior and environmental data through multimodal data collection, dynamic data fusion, user intention prediction and real-time feedback control, and improves the response speed, accuracy and user experience of the display screen interaction.

[0065] Please see attached Figure 1 The LED display screen interactive method of the present invention may include the following steps:

[0066] S1. Collect visual, touch, infrared and environmental data;

[0067] S2. Reduce noise, extract features and uniformly encode multimodal data;

[0068] S3. Multimodal data fusion modeling based on dynamic model;

[0069] S4. Use the model to identify user interaction intentions;

[0070] S5. Adjust the LED display content and interactive response according to the intention.

[0071] For step S1, multimodal data collection relies on the coordinated work of visual sensors, touch sensors, infrared sensors and environmental sensors. These sensors monitor the user's gestures, touch behaviors, spatial positions and environmental conditions in real time and output multi-dimensional data information. As an option, the type and resolution of the sensor can be adjusted according to actual needs to adapt to different application scenarios.

[0072] In some embodiments, the collection of multimodal data can be based on a synchronized timestamp mechanism to ensure that the data collected by different sensors are consistent in the time dimension. This is of great significance for the accurate modeling of dynamic interactions. Specifically:

[0073] In this embodiment, visual data is collected by a camera, mainly used to monitor the user's gesture trajectory, facial expression or other visual information. Generally, the camera resolution is 1920×1080 or higher to facilitate the extraction of detailed features.

[0074] In one possible implementation, the camera collects continuous image frame data in real time. Where H, W, and C represent the height, width, and number of channels of the image, respectively. In order to facilitate subsequent time series modeling, these image frames will be timestamped to ensure the temporal relevance of the data.

[0075] As an option, a binocular camera or a depth camera can be used to collect three-dimensional structural information of the user's actions. For example, in some embodiments, the depth camera can provide depth information of the gesture. Further enhance the spatial information dimension of visual data.

[0076] Specifically, the camera can be deployed on the top or below the LED display to ensure that the field of view covers the user's activity area. In some application scenarios, such as large user movements or multiple people interacting at the same time, more accurate data collection can be achieved through the collaborative work of multiple cameras.

[0077] In this embodiment, the touch data is collected by a capacitive touch sensor integrated in the LED display screen to monitor the user's touch operation behavior on the display screen surface. The touch data includes the two-dimensional coordinates (x t ,y t ) and pressure value F t , the sampling frequency is generally 100Hz or higher.

[0078] In one possible implementation, the touch sensor outputs coordinate data of the touch point when the user touches the screen. For example, when the user slides on the screen, a time series (x t ,y t ,t), where t is the timestamp used to record the timing information of the touch action.

[0079] As an option, a touch pressure sensing function can be added to detect the force information of the user's touch F t For example, when the user taps the screen with different forces, the pressure value F t The changes distinguish different interactive instructions.

[0080] In this embodiment, infrared data is collected by an integrated infrared sensor, and is mainly used to obtain distance information and spatial position information between the user and the LED display screen.

[0081] Specifically, the infrared sensor calculates the distance D between the user and the screen by emitting infrared light and detecting the time difference of the reflected signal. t In some embodiments, the infrared data also includes the user's spatial location information (X t ,Y t ,Z t) is used to support three-dimensional spatial interaction between users and the display screen.

[0082] As an option, multiple infrared sensors can be combined to achieve more accurate distance measurement. For example, in some application scenarios, the user may move a large range in front of the screen. By fusing the data of multiple infrared sensors, the accuracy and robustness of position detection can be improved.

[0083] In this embodiment, environmental data is collected by light sensors and microphones, and is mainly used to monitor the environmental conditions of the interactive scene to assist in adjusting the display effect of the LED display screen.

[0084] Generally, the light sensor is used to collect the ambient light brightness L t , whose measurement range is usually 0 to 100,000 lux. As an option, the brightness of the screen can be dynamically adjusted by continuously sampling light data. For example, when the ambient light suddenly dims, the visibility of the screen can be enhanced through brightness adjustment.

[0085] The microphone is used to collect the sound characteristics S in the environment t , such as detecting the intensity of background noise or specific voice commands. As an implementation method, the user's voice input can be analyzed in conjunction with a voice recognition algorithm and used as part of the interaction.

[0086] Generally, in order to ensure the time consistency of multimodal data, each sensor generates a timestamp t when collecting data, and unifies the time reference through a synchronization mechanism. For example, the frame time of visual data and the sampling time of touch data need to be strictly aligned to facilitate subsequent time series modeling.

[0087] In one possible implementation, the data collected by all sensors will be stored uniformly as a time series (D t ,(x t ,y t ),I t ,L t ,S t ), where D t 、(x t ,y t ), I t , L t and S t They correspond to infrared data, touch data, visual data, light data and sound data respectively. For each moment t, the multimodal data M t The unified representation is:

[0088] M t = {I t ,(x t ,y t ),Dt ,L t ,S t}

[0089] in, Represents visual image data, H is the image height, W is the image width, C is the number of image channels, is the two-dimensional coordinate of the touch point, Indicates the touch pressure value. Indicates the distance between the user and the LED display. Indicates the spatial coordinates of the user in front of the LED display, where X t , Y t , Z t are the three-dimensional space coordinate values, Indicates the ambient light brightness. represents the sound feature vector, and d is the feature dimension.

[0090] For step S2, in order to ensure the accuracy and robustness of subsequent multimodal data fusion, noise reduction, feature extraction and unified encoding are performed on the collected multimodal user interaction data. Through reasonable preprocessing, the noise and redundant information that may exist in the sensor acquisition process can be effectively removed, and the different modal data can be mapped to a unified latent space to form a high-dimensional latent representation, providing high-quality input data for dynamic modeling in subsequent steps.

[0091] Generally speaking, for visual data processing, it is necessary to first adopt a noise reduction method to address the image noise problem in order to enhance the expressiveness of image features. Specifically, the Gaussian filter method is used to smooth and reduce noise on the image, and the formula is as follows:

[0092]

[0093] in, Represents the image data after noise reduction, I t is the original image data, G σ is the Gaussian kernel function.

[0094] As an option, the parameter σ of the Gaussian kernel can be dynamically adjusted according to the image noise intensity. For example, a larger σ can be selected for a high-noise image, while a smaller σ can be selected for a clearer image.

[0095] In another possible implementation, wavelet transform is used to reduce noise for touch data and infrared data. Wavelet transform can effectively remove high-frequency noise by performing multi-scale decomposition on the signal while retaining the key features of the original signal. Specifically, the touch coordinates (x t ,y t ) and pressure data Ft The discrete wavelet transform is used to decompose the data into different frequency bands and reconstruct the data after removing the high frequency components. The wavelet basis function can be Daubechies wavelet or Haar wavelet, and its parameters need to be adjusted according to the characteristics of the data.

[0096] In some embodiments, in order to extract key features from visual data, such as key points of user's gestures or facial expression features, a convolutional neural network (CNN) is used for feature extraction. Specifically, it can be based on ResNet or MobileNet structure, and the input is denoised image data The output is a high-dimensional feature vector Its dimension can be set to 128, 256 or 512 according to specific needs.

[0097] As an option, in order to further extract dynamic information from the gesture trajectory, the optical flow method can be combined to extract motion features and fused with the static features extracted by CNN. The formula for optical flow calculation is as follows:

[0098]

[0099] in, represents the image gradient, v represents the velocity vector of the pixel, represents the time gradient.

[0100] In the implementation of the present invention, the processing of infrared data places more emphasis on the reliability of distance and spatial position information. Specifically, for the distance data D collected by the infrared sensor t and spatial coordinates (X t ,Y t ,Z t ), first normalize it to eliminate the measurement deviation under different equipment or environmental conditions. The normalization formula is as follows:

[0101]

[0102] in, Represents the normalized distance data, D min and D max are the minimum and maximum values ​​of the distance data, respectively. Through normalization, the dynamic range of the infrared data can be adapted to the input requirements of the subsequent model.

[0103] In order to achieve a unified expression of multimodal data, a variational autoencoder (VAE) is proposed to embed different modal data into a unified latent space. Specifically, the feature vectors of visual data, touch data, infrared data, and environmental data are input into the encoder of VAE respectively to obtain a unified latent representation. The core training goal of VAE is to maximize the evidence lower bound (ELBO) of the data, and its formula is as follows:

[0104]

[0105] in, represents the total loss function of VAE, represents the reconstruction error term, q φ (z|x) represents the approximate posterior distribution of the encoder, p θ (x|z) represents the conditional distribution of the decoder, D KL (q φ (z|x)||p(z)) represents the KL divergence term (Kullback-Leibler divergence), and p(z) represents the prior distribution of the latent variable.

[0106] In one possible implementation, in order to further optimize the preprocessing effect, an attention mechanism can be introduced for specific modal data. Specifically, for each modal data feature vector, its correlation score with the global feature is calculated, and a weight is assigned based on the correlation to dynamically adjust the contribution of the modal data. The calculation formula for the attention score is as follows:

[0107]

[0108] Among them, α i is the weight of the i-th mode, W i is the learning parameter, z t represents the multimodal potential representation at the current time t, ∑ j exp(W j z t ) represents the sum of attention scores of all modal data at the current time t, and exp represents the exponential function.

[0109] For step S3, dynamic data fusion is performed on multimodal data, aiming to model the multimodal data in the time series, capture its time-dependent characteristics, and optimize the fusion state in combination with the recursive update mechanism, so as to generate a high-precision fusion representation, providing a reliable data foundation for subsequent user intention prediction and real-time interaction feedback.

[0110] In general, dynamic data fusion needs to solve the problems of time correlation of multimodal data, weight allocation of different modal data, and noise interference. This step systematically solves the above problems through dynamic modeling based on recurrent neural network (RNN), weight allocation of attention mechanism, and recursive optimization mechanism combined with Bayesian filtering.

[0111] In this embodiment, the time series dependency relationship of multimodal data is modeled based on a recurrent neural network (RNN).

[0112] In one possible implementation, RNN is used to capture the dynamic trend of multimodal data in time series. The input is the preprocessed high-dimensional latent representation The output is the hidden state at the current moment The state update formula of RNN is as follows:

[0113] h t =f θ (h t-1 ,z t )

[0114] Among them, h t represents the hidden state at the current moment, h t-1 is the hidden state at the previous moment, f θ is the state update function of RNN, z t It is the multimodal data input at the current moment.

[0115] In the specific implementation, the hidden state h t It captures the dynamic characteristics of time series data, such as the continuity of user gestures, the motion trajectory of touch points, and the changing trend of environmental data.

[0116] Generally speaking, the importance of different modal data will change dynamically over time. For example, when a user performs a gesture operation, the importance of visual data may be higher than that of touch data, and when a user performs a touch operation with the screen, the weight of touch data will increase accordingly.

[0117] Alternatively, the output of the attention mechanism can be combined with the hidden state of the recurrent neural network to further generate a fused representation:

[0118]

[0119] in, For dynamic fusion representation, is the attention weight of the i-th modal data at time t, W i is the linear transformation matrix of the ith mode, h t is the hidden state of the recurrent neural network at the current time t.

[0120] In addition, during the recursive updating process, Bayesian filtering is combined to optimize the fusion state of multimodal data.

[0121] Specifically, Bayesian filtering recursively adjusts the current fusion state through a prediction-update mechanism, thereby suppressing noise interference and improving data robustness. In one possible implementation, the fusion state X t The updating process of is divided into a prediction step and an update step:

[0122] The prediction step is used to predict the fusion state at the current moment based on the state transition model:

[0123]

[0124] in, Represents the predicted state, is the state transfer matrix, is the control matrix, is the control input,

[0125] The update step corrects the predicted state by combining the current observation data. The update formula is:

[0126]

[0127] in, is the observation data at the current moment, is the observation matrix, is the Kalman gain, and the calculation formula is:

[0128]

[0129] in, represents the prediction error covariance, R is the observation noise covariance, The location of the observation matrix.

[0130] In some embodiments, the recursive update mechanism is also combined with dynamic weight adjustment to further improve the robustness and accuracy of the fusion state by dynamically optimizing the observation weight and prediction weight. For example, the observation weight can be dynamically adjusted according to the signal-to-noise ratio of the sensor to reduce the impact of the modality with poor signal quality on the fusion result.

[0131] For step S4, the user's behavior and operation intention are identified through the user interaction intention prediction model. The identified intention will be directly used as the input of real-time interaction feedback to provide a basis for the dynamic response of the LED display.

[0132] Generally, a deep neural network is used as the intention prediction model in this step to classify the feature representation of the fusion state and output the user's operation category. In the specific implementation, the model can handle a variety of interaction scenarios, including gesture control, touch operation, distance response, etc., and can accurately determine the user's interaction needs in different scenarios.

[0133] In this embodiment, the specific implementation of the user interaction intention prediction model is as follows:

[0134] The input is the fusion state vector X generated by the dynamic data fusion module tThis vector contains the time series characteristics of multimodal data and is the key representation for describing the user's current interaction behavior.

[0135] As an option, the intent prediction model is implemented using a deep neural network, including an input layer, multiple hidden layers, and an output layer. The hidden layer may include a fully connected layer, an activation function (such as ReLU), a regularization layer (such as Dropout), etc., to extract high-level features and prevent overfitting.

[0136] Specifically, the output layer of the intent prediction model uses the Softmax function to predict the probability distribution of multi-category user intentions. The formula is:

[0137]

[0138] Among them: P (intention i ∣X t ) indicates that in the input fusion state X t The probability of the i-th type of intention, W i represents the weight matrix, b i represents the bias term, exp represents the natural exponential function, N is the total number of intent categories, represents the sum of the unnormalized scores of all N categories.

[0139] In one possible implementation, in order to enhance the understanding of complex user behaviors, an attention mechanism can be introduced in the hidden layer. Through the attention weights, the model can dynamically pay attention to the fusion state X t The different feature dimensions of attention are used to improve the accuracy of intent prediction. The calculation formula of attention weight is:

[0140]

[0141] Among them, α j represents the importance weight of the jth feature dimension, exp represents the exponential function, and W′ j is the learning parameter of the attention mechanism, X t represents the multimodal fusion state generated by the dynamic data fusion module at time t, and M represents the fusion state X t The total feature dimension of .

[0142] Specifically, the output of the intention prediction model is a probability distribution, which indicates the possible operation category of the user. For example, in some embodiments, if the user input is: the gesture trajectory feature is more significant, the intention prediction model may output a high probability of "content switching"; if the touch pressure data is high, the model may output a high probability of "content enlargement"; if the environmental data indicates that the user is away from the screen, the model may output a high probability of "screen brightness reduction".

[0143] As an option, supervised learning methods can be used in model training, using labeled multimodal interaction datasets for training. The loss function for training is generally cross entropy loss, with the formula:

[0144]

[0145] Where L represents the cross entropy loss value, N represents the total number of all possible intent categories, and y i Indicates the actual intent category label, with a value of 0 or 1, P(intent i ∣X t ) represents the predicted probability.

[0146] In order to improve the model's adaptability to diverse scenarios, the following enhancement measures can be introduced during training: introducing random noise into the training data or simulating data under different environmental conditions; combining the user's emotional state or behavioral habit prediction to improve the model's generalization ability.

[0147] In some embodiments, to further improve the efficiency of the model in real-time interaction, the intention prediction module can be implemented through a lightweight neural network. For example, using the MobileNet or TinyML architecture to reduce computational complexity and resource usage, and adapt to the operating environment of edge computing devices.

[0148] In one possible implementation, the intent prediction model can also be combined with a reinforcement learning module to dynamically adjust the weight of the prediction results. For example, the prediction accuracy of a user's specific operation category can be optimized through the reward signal of reinforcement learning, thereby better meeting personalized needs.

[0149] For step S5, the display content and interactive feedback of the LED display are dynamically adjusted using the identified interactive intent to form the real-time response capability of the system. This step is one of the core links of the entire interactive method, and its goal is to improve the intelligent performance of the LED display and user experience through real-time feedback.

[0150] In general, the real-time feedback of LED display screens needs to dynamically adjust the update strategy of the displayed content according to the results of user intention prediction. By combining the fusion state of multimodal data and the results of intention classification, the user's expected interaction method for the current displayed content can be determined, such as switching, zooming in, zooming out, or brightness adjustment. At the same time, the feedback control process needs to combine the dynamic changes of multimodal input data and fully consider the impact of ambient light, user distance and other information on the display effect.

[0151] As an option, this step adopts the design idea of ​​deep reinforcement learning to optimize the feedback strategy, dynamically evaluates the real-time and accuracy of the system response through the reward function, and optimizes the matching degree between the display content and the interactive behavior of the display. Specifically, through the feedback control mechanism, the user's operation intention is directly linked to the change of the content displayed on the LED display, so as to achieve efficient content display and operation response in different scenarios.

[0152] In a possible implementation, firstly, based on the multimodal data fusion state X t and the user intention prediction result P(intent|X t ), defines the input parameters of the feedback control, including the classification results of the user operation at the current moment and the corresponding weight values. Combined with these inputs, the feedback control module generates the corresponding display adjustment strategy.

[0153] In some embodiments, the adjustment of the displayed content may be achieved by:

[0154] Gesture control content switching: When the user intention prediction result indicates a "slide gesture", the system adjusts the switching of the display content according to the slide direction. For example:

[0155] Content Index = Content Index + Δ 滑动方向

[0156] Among them, Δ 滑动方向 Calculate based on the sliding direction and speed of the gesture to ensure that the sensitivity of the switching action matches the user operation.

[0157] Touch control zoom operation: For touch interaction, the system calculates the zoom factor S based on the coordinates and pressure data of the touch point t :

[0158] S t =S t-1 +k·(F t -F 基准 )

[0159] Among them, F t is the current touch pressure value, F 基准 is the preset reference pressure value of the system, k is the proportional factor, S t Represents the scaling factor at the current time t.

[0160] Infrared distance brightness adjustment: combined with the distance D between the user and the display t , dynamically adjust screen brightness B t :

[0161] B t =B 最大 ·exp(-λD t )

[0162] Among them, B 最大 is the maximum brightness of the screen, λ is the attenuation coefficient, and exp represents the exponential function.

[0163] In order to improve the feedback performance of the system, a deep reinforcement learning model is introduced in the feedback control process in this embodiment. Specifically, the design of reinforcement learning includes the following contents:

[0164] The system will merge state X t and the current user intention prediction result P(intent|X t ) as state input. The fusion state includes the user's dynamic interaction information, such as gesture trajectory, touch point pressure value, and user distance; the action is defined as the adjustment operation of the LED display content, such as switching content, zooming content, or brightness adjustment; the reward function is used to evaluate the accuracy and real-time performance of the system feedback, and the specific expression is:

[0165] R=λ1·Response accuracy-λ2·System delay

[0166] Among them, λ1 and λ2 are weight parameters, the response accuracy is determined by the consistency between user intention and system feedback, and the system delay is a measure of the time required to complete the display adjustment.

[0167] Training and updating: The reinforcement learning model is trained through the policy gradient method to continuously optimize the feedback strategy of the LED display in different operating scenarios.

[0168] In a specific implementation, when the user continuously performs a sliding gesture, the system dynamically adjusts the sensitivity of content switching through reinforcement learning to adapt to the user's sliding speed and the complexity of the screen content. For example, when the user slides quickly, the system can predict the next switching content in advance and shorten the response delay.

[0169] In one embodiment, when the user drags a virtual object on the LED display screen through a touch operation, the feedback control module adjusts the position of the object in real time. The specific implementation method is as follows:

[0170] The current position of the touch point (x t ,y t ) and the initial position (x0, y0) to calculate the displacement vector:

[0171] Δx=x t -x0,Δy=y t -y0

[0172] According to the touch pressure F t , to judge the validity of the operation, when F t >F 阈值 When , the position adjustment is triggered:

[0173] Object position = object position + (Δx, Δy)

[0174] The system combines real-time distance data D t Dynamically adjust the object scale S t , the formula is:

[0175] S t =S 初始 ·exp(-λD t )

[0176] Among them, Δx, Δy represent the displacement of the current position of the touch point and the initial position in the x-axis and y-axis directions, x, y represent the two-dimensional coordinate values ​​of the touch point at the current time t, x0, y0 represent the two-dimensional coordinate values ​​of the touch point at the initial time of the interactive operation, the object position represents the screen display content, S t represents the current scale of the object on the LED display at time t, S 初始 Indicates the default scaling of the object in the initial display state, D t represents the real-time distance between the user and the LED display at time t, and λ represents the distance D t The degree of attenuation of the object's scaling.

[0177] Please see attached Figure 2 The present invention also provides an interactive system for an LED display screen, which combines technologies such as multimodal data fusion, deep learning, and real-time feedback control. Through the synergistic effect of a multi-sensor module, a data preprocessing module, a dynamic data fusion module, a user intention prediction module, and a real-time feedback control module, the system can collect multimodal data of user interaction, perform dynamic modeling and fusion, accurately predict the user's operating intention, and adjust the display content and interactive response of the LED display screen in real time according to the user's needs, thereby realizing an intelligent and efficient user interaction experience. It provides efficient and stable technical support for fields such as smart advertising display, immersive education, and virtual reality.

[0178] The multi-sensor module is used to collect multi-modal data generated by the interaction between the user and the LED display in real time, including visual, touch, infrared and environmental data. Through multi-modal information collection, it provides high-quality basic data for dynamic data fusion and user intention prediction.

[0179] In this embodiment, the multi-sensor module integrates multiple sensor devices to collect data of specific modes respectively: visual data is captured by a high-resolution camera to generate time series images; touch data obtains touch point coordinates and pressure values ​​through a touch sensor, and the touch frequency is generally set to 100 Hz; the infrared sensor collects the distance and spatial position between the user and the LED display screen, and the resolution accuracy can reach the millimeter level; environmental data is collected by light sensors and microphones respectively. Light brightness and sound characteristic signals, all data are timestamped to ensure time synchronization.

[0180] The multi-sensor module can comprehensively collect dynamic data of the interaction between the user and the environment, providing accurate data input for subsequent pre-processing and fusion analysis. Through multi-modal collection, the comprehensiveness of user operation information and the environmental adaptability of the system can be significantly improved.

[0181] The data preprocessing module is used to reduce noise, extract features and standardize the raw data collected by multiple sensors to generate high-quality input data. Through data preprocessing, noise and redundant information are effectively removed to provide structured feature data for the dynamic data fusion module.

[0182] In this embodiment, the data preprocessing module includes: using a convolution filter to reduce noise on visual data; using wavelet transform or fast Fourier transform to suppress high-frequency noise on touch, infrared and environmental data; normalizing the preprocessed data to unify the scale range of eigenvalues; finally, all preprocessed data are uniformly embedded into a high-dimensional latent space for use by the dynamic data fusion module.

[0183] The data preprocessing module can significantly improve the system's processing efficiency and fusion accuracy for multimodal data, providing reliable support for subsequent dynamic data analysis.

[0184] The dynamic data fusion module dynamically fuses multimodal data through time series modeling and weight distribution mechanism. This module can capture the temporal correlation between data, optimize the fusion state with the recursive update mechanism, and provide accurate fusion results for the user intention prediction module.

[0185] In this embodiment, the dynamic data fusion module performs time series modeling on multimodal data and improves the robustness of the fusion through a dynamic weight adjustment mechanism: a recurrent neural network (RNN) is used to perform time series modeling on multimodal data to extract the dynamic change characteristics of the data; the attention mechanism is combined to dynamically allocate weights according to the importance of each modal data; the Bayesian filtering and recursive update mechanism are used to optimize the fusion state, and the posterior probability distribution is calculated based on the current observation data and the historical fusion state to further improve the fusion accuracy.

[0186] The dynamic data fusion module can process the complex data stream of user interaction in real time and provide stable and reliable fusion state output for the user intention prediction module. Through the recursive update mechanism, it can effectively deal with the uncertainty and dynamic changes in multimodal data.

[0187] The user intention prediction module identifies the user's operation intention according to the fusion state provided by the dynamic data fusion module and generates the corresponding classification results. This module provides decision-making basis for the real-time feedback control module and is the key to realizing intelligent interaction.

[0188] In this embodiment, the user intention prediction module adopts a deep neural network model to identify the user operation intention through a classification mechanism: the model input is the multimodal fusion state x t , the output is the probability distribution of the operation intention; the classification model uses the Softmax activation function to convert the user operation category into a probability value to determine the user's specific intention; for example, when the system recognizes a sliding gesture, the prediction result can indicate the direction and target of the LED screen content switching.

[0189] The user intention prediction module can efficiently identify the user's operational needs and provide accurate behavioral decision support for real-time interaction, thereby achieving smooth and intelligent operation.

[0190] The real-time feedback control module dynamically adjusts the content and response behavior of the LED display screen according to the classification results of the user intention prediction module. Through the feedback control mechanism, efficient interaction between users and the system can be achieved.

[0191] In this embodiment, the real-time feedback control module dynamically adjusts the content displayed on the LED display screen according to the user intention category and the current interactive environment: for sliding operations, the content switching is adjusted in real time; for touch operations, the display ratio or position is dynamically adjusted; for changes in ambient light, the display effect is optimized by adjusting the brightness; the reinforcement learning model is used to optimize the feedback strategy, and the reward function is dynamically evaluated based on the response accuracy and delay to continuously improve the user experience.

[0192] The real-time feedback control module realizes the dynamic response capability of the LED display, providing users with a more intuitive and efficient interactive experience, while significantly reducing operation delays and enhancing the intelligence level of the system.

[0193] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An interactive method for LED display screen, characterized in that: The following steps are involved: Collecting user interaction data through a multi-sensor module, wherein the user interaction data includes visual data, touch data, infrared data and environmental data; Preprocess the collected multimodal user interaction data, including data noise reduction, feature extraction and unified coding; Based on the dynamic data fusion model, the multimodal user interaction data is modeled in time series to generate the fusion state; Identify user interaction intention based on user interaction intention prediction model; The display content and interactive feedback of the LED display are adjusted in real time according to the identified user interaction intentions.

2. The LED display screen interactive method according to claim 1, characterized in that: The collection of the user interaction data includes: The user's gesture trajectory and facial expression are acquired through the camera, and the acquisition result is time series image data; The touch sensor collects the coordinates and pressure information of the user's touch point, and the collection results are two-dimensional coordinate data and pressure values; The distance and spatial position between the user and the LED display screen are acquired through the infrared sensor, and the acquisition results are the distance value and three-dimensional spatial coordinates; Collect light brightness and sound characteristic data through environmental sensors.

3. The LED display screen interactive method according to claim 1, characterized in that: The pre-processing comprises: Use convolution filtering to reduce noise on visual data; Wavelet transform is used to reduce noise on touch, infrared and environmental data; The preprocessed multimodal data is uniformly encoded through a variational autoencoder to generate a high-dimensional latent representation.

4. The LED display screen interactive method according to claim 1, characterized in that: The dynamic data fusion model includes: Use recurrent neural networks to capture temporal dependencies in multimodal data; The attention mechanism is used to assign weights to multimodal data. The attention weight calculation formula is: in, is the weight of the i-th modal data, W i is the learning parameter, h t is the hidden state, ∑ j exp(W j h t ) represents the sum of the weighted results of all modal data after taking the exponential value, and exp represents the natural exponential function.

5. The LED display screen interactive method according to claim 4, characterized in that: The dynamic data fusion model combines the state transfer model and the Bayesian filter for recursive updating, including: Predict the fusion state based on the state transition model: X t =AX t-1 +BU t +η t Among them, X t represents the fusion state at the current time t, X t-1 represents the fusion state at the previous time t-1, A is the state transfer matrix, B is the control matrix, U t is the external input variable of the system at time t, η t is the process noise; The fusion state is updated using a Bayesian filter, and the accuracy of the current fusion state is optimized through the posterior probability distribution.

6. The LED display screen interactive method according to claim 5, characterized in that: The dynamic data fusion model optimizes the time series modeling process of multimodal data through a recursive update mechanism, including the following steps: Based on the time series characteristics of multimodal data, the posterior probability distribution of the fusion state is calculated using the current fusion state and new observation data; In the recursive update process, the prediction-update mechanism of Bayesian filtering is combined to dynamically optimize the fusion state, where the observation model is used to associate the current observation data with the fusion state, and the state transition model is used to predict the next fusion state. By dynamically adjusting the observation weights and state prediction weights, the influence of observation noise can be suppressed and the stability and robustness of multimodal data fusion can be improved.

7. The LED display screen interactive method according to claim 1, characterized in that: The user interaction intention prediction adopts a deep neural network model and uses a Softmax function for classification, including: P(Intent|X t )=Softmax(WX t +b) Among them, X t represents the high-dimensional feature vector generated by the dynamic data fusion model at time t, and W represents the fusion state X in the model t The linear mapping relationship between each interaction intention category, b represents the independent offset of each interaction intention category, P(intent|X t ) represents the probability of a specific interaction intention.

8. The LED display screen interactive method according to claim 1, characterized in that: The user interaction includes: Control the switching of LED display screen content according to gesture data; Adjust the enlargement or reduction of the LED display screen content according to the touch data; Dynamically adjust display brightness based on user distance data provided by infrared sensors; Optimizes the visual quality and responsiveness of display content based on data provided by environmental sensors.

9. The LED display screen interactive method according to claim 1, characterized in that: The interaction feedback optimizes the interaction strategy through a deep reinforcement learning model, where: Take the fused multimodal state as input; The adjustment of the LED display content is used as output; The reward function is used to minimize system latency and maximize the response accuracy of user interactions. The specific expression is: R=λ1·Response accuracy-λ2·Delay Among them, R represents the comprehensive performance score of the system in an interaction process, λ1 is the response accuracy weight coefficient, λ2 is the system delay weight coefficient, response accuracy is the degree to which the LED display interactive system correctly responds to user interaction input, and delay is the time interval from the system receiving user input to completing interaction feedback.

10. An LED display screen interactive system, applied to an LED display screen interactive method according to claims 1-9, characterized in that: include: Multi-sensor module for collecting user interaction data, including visual data, touch data, infrared data and environmental data; Data preprocessing module, used for noise reduction and feature extraction of multimodal data; Dynamic data fusion module, used to perform high-dimensional time series modeling on multimodal data and generate fusion states; User intention prediction module, used to identify the user's interaction intention; The real-time feedback control module is used to adjust the content display and response behavior of the LED display according to the user's interaction intention.

Images