A light and shadow interactive control method for sand table

Through the combination of user touch input and light and shadow interaction hybrid control model, the problem of lack of dynamicity and interactivity in traditional sand table display is solved, a more natural operation experience and high interaction depth are achieved, and the authenticity and stability of sand table display is improved.

CN118689365BActive Publication Date: 2024-12-03LIFANG DIGITAL TECH GRP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411162832.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-12-03
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

Traditional sand table displays lack dynamics and interactivity, and static models and simple lighting effects are difficult to simulate complex light and shadow, reducing the realism and immersion of the display, and difficult to attract the audience's long-term interest.

Method used

By combining user touch input and light and shadow interaction hybrid control model, touch data and depth images are collected and processed, and integrated into model input, the pre-trained light and shadow interaction hybrid control model generates dynamic light and shadow effects and projected onto the surface of the sand table to achieve natural and intuitive user operation experience and high interactive depth.

Benefits of technology

It significantly improves the accuracy and response speed of the light and shadow interaction of sand tables, improves the real-time and interaction depth of displays, and improves stability and reliability through uncertainty estimation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118689365B_ABST
    Figure CN118689365B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of dynamic interaction technology of sandboxes, and in particular to a light and shadow interaction control method for a sandbox, the method comprising the following steps: collecting touch input data of a user and obtaining a surface depth image of the sandbox; converting the touch input data of the user into standardized coordinates, and performing noise reduction and normalization processing on the surface depth image of the sandbox; integrating the processed touch input data and the depth image into a model input format, and inputting them into a pre-trained light and shadow interaction hybrid control model for calculation, and outputting the light and shadow interaction type and the corresponding interaction intensity of the sandbox; mapping the interaction intensity to light and shadow parameters within a set range, selecting a corresponding light and shadow effect template based on the light and shadow interaction type of the sandbox, dynamically adjusting the light and shadow parameters in combination with the surface depth image of the sandbox, projecting the generated light and shadow effects onto the surface of the sandbox, and completing the light and shadow interaction control of the sandbox.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of dynamic sandbox interaction, and in particular to a light and shadow interaction control method for a sandbox. Background Art

[0002] The sand table display in science and technology museums and exhibition halls is an important explanation tool. The light and shadow interactive technology can greatly enhance its appeal and explanation effect. At present, the traditional sand table display mainly relies on static models and simple lighting effects. Static models generally lack flexibility and cannot dynamically display the scene changes of the sand table, which limits the richness of information transmission. In addition, simple lighting effects are difficult to accurately simulate complex natural light and shadow, which reduces the realism and immersion of the display. In addition, the traditional fixed sand table display method also lacks interactivity and is difficult to attract and maintain the long-term interest of the audience. Based on this, in response to the above problems, we designed a light and shadow interactive control method for the sand table. Summary of the invention

[0003] The purpose of the present invention is to provide a light and shadow interaction control method for a sandbox, which can significantly improve the accuracy and response speed of the sandbox light and shadow interaction by combining the user's touch input with the light and shadow interaction hybrid control model. It can not only achieve a more natural and intuitive user operation experience, but also greatly improve the real-time and interactive depth of the sandbox display. At the same time, by introducing uncertainty estimation, the stability and reliability of the sandbox light and shadow interaction are also improved.

[0004] The embodiments of the present invention are implemented by the following technical solutions:

[0005] A light and shadow interactive control method for a sandbox, the method comprising the steps of:

[0006] Collect the user's touch input data and obtain the surface depth image of the sandbox;

[0007] Convert the user's touch input data into standardized coordinates, and perform noise reduction and normalization on the depth image of the sandbox surface;

[0008] Integrate the processed touch input data and depth image into the model input format, and input it into the pre-trained light and shadow interaction hybrid control model for calculation, and output the light and shadow interaction type and corresponding interaction intensity of the sandbox;

[0009] The interaction intensity is mapped to the light and shadow parameters in the set range, and the corresponding light and shadow effect template is selected based on the light and shadow interaction type of the sandbox. The light and shadow parameters are dynamically adjusted in combination with the surface depth image of the sandbox, and the generated light and shadow effects are projected onto the sandbox surface to complete the light and shadow interaction control of the sandbox.

[0010] Optionally, the light and shadow interactive hybrid control model consists of an input layer, a Transformer encoder, a multi-head self-attention layer, an MLP layer, an output layer and a task-specific layer.

[0011] Optionally, the calculation formula of the input layer is:

[0012]

[0013]

[0014]

[0015] in, is the symbol of the real number set, is the input image, specifically a multi-channel image formed by summarizing the touch input data and the depth image, are the image height, width and number of channels respectively, is the image after segmentation, is the number of image blocks, is the size of the image block, is the initial input sequence, is the embedding matrix, is the position code, For time coding, For classification marks, Control the hidden dimension of the model for interactive blending of light and shadow.

[0016] Optionally, the Transformer encoder has a calculation formula of:

[0017]

[0018]

[0019] in, is the index, , is the number of Transformer layers, For the The output of the layer Transformer, For the The output of the layer Transformer, For the The intermediate output after multi-head self-attention in the layer, MSA is multi-head self-attention, LN is layer normalization, and MLP is multi-layer perceptron.

[0020] Optionally, the calculation formula of the multi-head self-attention layer is:

[0021]

[0022]

[0023]

[0024]

[0025] in, are the weight matrices, are matrices, is the dimension of each attention head, is the number of attention heads, is the attention weight matrix, For the The output of an attention head is is the output projection matrix.

[0026] Optionally, the calculation formula of the MLP layer is:

[0027]

[0028] in, are the weight and bias of the first layer respectively, are the second layer weight and bias respectively, is the hidden dimension of the feedforward network and GELU is the activation function.

[0029] Optionally, the calculation formula of the output layer is:

[0030]

[0031] in, is the first token output by the last layer of Transformer. It is the final output of the light and shadow interactive mixing control model.

[0032] Optionally, the task-specific layer is calculated as follows:

[0033]

[0034]

[0035]

[0036]

[0037]

[0038]

[0039] in, are the weight and bias of interaction type prediction, respectively. are the weight and bias of interaction strength prediction, respectively. are the weight and bias of uncertainty estimation, respectively, are the intermediate hidden layer outputs for interaction type and interaction strength prediction, respectively. Predict the results for the interaction type, is the interaction strength prediction result, are the uncertainty estimates for interaction type and interaction strength predictions, respectively. is the sigmoid activation function, ReLU, softmax, and softplus are activation functions respectively.

[0040] Optionally, the specific training process of the light and shadow interactive hybrid control model is as follows:

[0041] Obtain the historical data of light and shadow interaction of the sandbox, pre-process the historical data of light and shadow interaction of the sandbox, and divide it into a training set and a test set;

[0042] The training set is input into the light and shadow interaction hybrid control model for training, the predicted sand table light and shadow interaction type and the corresponding interaction strength are obtained, and the loss value between the test set and the predicted sand table light and shadow interaction type and the corresponding interaction strength is calculated;

[0043] According to the gradient of the loss value relative to the light and shadow interactive hybrid control model, the parameters of the light and shadow interactive hybrid control model are iteratively updated through the Adam optimization algorithm until the maximum number of iterations is reached, and the trained light and shadow interactive hybrid control model is output.

[0044] Optionally, the loss value between the calculated test set and the predicted sandbox light and shadow interaction type and the corresponding interaction intensity is calculated using the following specific formula:

[0045]

[0046]

[0047]

[0048]

[0049]

[0050] in, is the total loss objective function, are the parameters of the light and shadow interactive mixing control model, is the weight coefficient of the interaction type loss, is the weight coefficient of interaction strength loss, is the weight coefficient of uncertainty loss, is the weight coefficient of the regularization term, is the loss function for interaction type prediction, is the loss function for interaction strength prediction, is the loss function of uncertainty estimation, is the regularized loss function, For the The true interaction type labels of training samples, The light and shadow interactive hybrid control model is used for the Interaction type prediction for training samples, The light-shadow interaction hybrid control model predicts the interaction intensity. is the true interaction strength label, is the uncertainty estimation of the interaction type prediction of the light-shadow interaction hybrid control model. is the uncertainty estimation of the interaction intensity prediction of the light-shadow interaction hybrid control model, It is the weight matrix of the light and shadow interaction mixing control model.

[0051] The technical solution of the embodiment of the present invention has at least the following advantages and beneficial effects:

[0052] The embodiments of the present invention can significantly improve the accuracy and response speed of sandbox light and shadow interaction by combining the user's touch input with the light and shadow interaction hybrid control model. It can not only achieve a more natural and intuitive user operation experience, but also greatly improve the real-time and interactive depth of the sandbox display. At the same time, by introducing uncertainty estimation, the stability and reliability of the sandbox light and shadow interaction are also improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic flow chart of a light and shadow interactive control method for a sandbox provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0055] Reference Figure 1 As shown, Figure 1 A schematic flow chart of a light and shadow interactive control method for a sandbox provided in an embodiment of the present invention.

[0056] In one embodiment, the present invention provides a light and shadow interactive control method for a sandbox, the method comprising the steps of:

[0057] Collect the user's touch input data and obtain the surface depth image of the sandbox;

[0058] Convert the user's touch input data into standardized coordinates, and perform noise reduction and normalization on the depth image of the sandbox surface;

[0059] Integrate the processed touch input data and depth image into the model input format, and input it into the pre-trained light and shadow interaction hybrid control model for calculation, and output the light and shadow interaction type and corresponding interaction intensity of the sandbox;

[0060] The interaction intensity is mapped to the light and shadow parameters in the set range, and the corresponding light and shadow effect template is selected based on the light and shadow interaction type of the sandbox. The light and shadow parameters are dynamically adjusted in combination with the surface depth image of the sandbox, and the generated light and shadow effects are projected onto the sandbox surface to complete the light and shadow interaction control of the sandbox.

[0061] In this embodiment, for the specific design of the sandbox, this embodiment introduces a light and shadow interaction control system for the sandbox. This system includes: a sandbox, a touch sensing layer, a depth camera, a projection, an adjustable LED light strip and a high-performance computing unit; wherein, the touch sensing layer is integrated below and around the surface of the sandbox to capture the user's direct interaction on the sandbox, the depth camera is installed on the ceiling or bracket directly above the sandbox to overlook the entire sandbox and capture the 3D contour changes on the sandbox surface, the projector is also installed on the ceiling or bracket directly above the sandbox to accurately project the generated light and shadow effects onto the sandbox surface, the adjustable LED light strip is a supplement to the embodiment, arranged around the sandbox or installed at the edge of the sandbox, further providing ambient light effects to enhance the visual experience of the sandbox, and the high-performance computing unit serves as a carrier of this system to perform data processing.

[0062] Specifically, in this embodiment, both the real-time collected data and the historical data for model training are preprocessed, and the preprocessing includes: noise reduction, normalization and other processing steps.

[0063] In this embodiment, the light and shadow interactive hybrid control model is specifically carried by the Vision Transformer model. The light and shadow interactive hybrid control model in this embodiment is composed of an input layer, a Transformer encoder, a multi-head self-attention layer, an MLP layer, an output layer and a task-specific layer.

[0064] The calculation formula of the input layer is:

[0065]

[0066]

[0067]

[0068] in, is the symbol of the real number set, is the input image, specifically a multi-channel image formed by summarizing the touch input data and the depth image, are the image height, width and number of channels respectively, is the image after segmentation, is the number of image blocks, is the size of the image block, is the initial input sequence, is the learnable embedding matrix, is the position code, For time coding, is a learnable classification label, Control the hidden dimension of the model for interactive blending of light and shadow.

[0069] The Transformer encoder has the following calculation formula:

[0070]

[0071]

[0072] in, is the index, , is the number of Transformer layers, For the The output of the layer Transformer, For the The output of the layer Transformer, For the The intermediate output after multi-head self-attention in the layer, MSA is multi-head self-attention, LN is layer normalization, and MLP is multi-layer perceptron.

[0073] The calculation formula of the multi-head self-attention layer is:

[0074]

[0075]

[0076]

[0077]

[0078] in, are the weight matrices, are matrices, is the dimension of each attention head, is the number of attention heads, is the attention weight matrix, For the The output of an attention head is is the output projection matrix.

[0079] The calculation formula of the MLP layer is:

[0080]

[0081] in, are the weight and bias of the first layer respectively, are the second layer weight and bias respectively, is the hidden dimension of the feedforward network and GELU is the activation function.

[0082] The calculation formula of the output layer is:

[0083]

[0084] in, is the first token output by the last layer of Transformer. It is the final output of the light and shadow interactive mixing control model.

[0085] The calculation formula of the task specific layer is:

[0086] Light and shadow interaction type prediction:

[0087]

[0088]

[0089] Light and shadow interaction intensity prediction:

[0090]

[0091]

[0092] Uncertainty Estimates:

[0093]

[0094]

[0095] in, are the weight and bias of interaction type prediction, respectively. are the weight and bias of interaction strength prediction, respectively. are the weight and bias of uncertainty estimation, respectively, are the intermediate hidden layer outputs for interaction type and interaction strength prediction, respectively. Predict the results for the interaction type, is the interaction strength prediction result, are the uncertainty estimates for interaction type and interaction strength predictions, respectively. is the sigmoid activation function, ReLU, softmax, and softplus are activation functions respectively.

[0096] Based on the light and shadow interactive hybrid control model constructed in this embodiment, this embodiment also designs a specific training process of the light and shadow interactive hybrid control model:

[0097] Obtain the historical data of light and shadow interaction of the sandbox, pre-process the historical data of light and shadow interaction of the sandbox, and divide it into a training set and a test set;

[0098] The training set is input into the light and shadow interaction hybrid control model for training, the predicted sand table light and shadow interaction type and the corresponding interaction strength are obtained, and the loss value between the test set and the predicted sand table light and shadow interaction type and the corresponding interaction strength is calculated;

[0099] According to the gradient of the loss value relative to the light and shadow interactive hybrid control model, the parameters of the light and shadow interactive hybrid control model are iteratively updated through the Adam optimization algorithm until the maximum number of iterations is reached, and the trained light and shadow interactive hybrid control model is output.

[0100] During implementation, historical data includes depth image data, corresponding touch input data, and recorded light and shadow interaction types and intensities. The depth image and touch data are fused into a multi-channel image as the input image. After data preprocessing of the historical data, the parameters of the light and shadow interaction hybrid control model are initialized, as well as the weights and biases of the task-specific layers. After that, the training of the light and shadow interaction hybrid control model is completed through forward propagation, loss calculation, back propagation and parameter update. The loss calculation of this embodiment includes: calculating the cross entropy loss of the interaction type and the mean square error loss of the interaction intensity, calculating the uncertainty perception loss based on the model prediction and the true label, and adding the regularization term to calculate the total loss.

[0101] In the specific application of this embodiment, the goal of this embodiment is to minimize the loss, so its specific calculation formula is:

[0102]

[0103]

[0104]

[0105]

[0106]

[0107] in, is the total loss objective function, are the parameters of the light and shadow interactive mixing control model, is the weight coefficient of the interaction type loss, is the weight coefficient of interaction strength loss, is the weight coefficient of uncertainty loss, is the weight coefficient of the regularization term, is the loss function for interaction type prediction, is the loss function for interaction strength prediction, is the loss function of uncertainty estimation, is the regularized loss function, For the The true interaction type labels of training samples, The light and shadow interactive hybrid control model is used for the Interaction type prediction for training samples, The light-shadow interaction hybrid control model predicts the interaction intensity. is the true interaction strength label, is the uncertainty estimation of the interaction type prediction of the light-shadow interaction hybrid control model. is the uncertainty estimation of the interaction intensity prediction of the light-shadow interaction hybrid control model, It is the weight matrix of the light and shadow interaction mixing control model.

[0108] In implementation, this embodiment sets the weight coefficient according to the actual application effect. is 1, is 1, is 0.5, The weight coefficient value provided in this embodiment is 0.0001, and the application of the weight coefficient value provided in this embodiment can effectively balance multiple task objectives in the light and shadow interaction analysis, which not only ensures that the accuracy of interaction type identification and interaction intensity prediction are given equal attention, but also moderately introduces uncertainty estimation, thereby improving the reliability and robustness of the light and shadow interaction hybrid control model.

[0109] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A light and shadow interactive control method for a sandbox, characterized in that: The steps of the method include: Collect the user's touch input data and obtain the surface depth image of the sandbox; Convert the user's touch input data into standardized coordinates, and perform noise reduction and normalization on the depth image of the sandbox surface; Integrate the processed touch input data and depth image into the model input format, and input it into the pre-trained light and shadow interaction hybrid control model for calculation, and output the light and shadow interaction type and corresponding interaction intensity of the sandbox; Map the interaction intensity to the light and shadow parameters within the set range, select the corresponding light and shadow effect template based on the light and shadow interaction type of the sandbox, dynamically adjust the light and shadow parameters in combination with the surface depth image of the sandbox, and project the generated light and shadow effects onto the surface of the sandbox to complete the light and shadow interaction control of the sandbox; The light-shadow interactive hybrid control model consists of an input layer, a Transformer encoder, a multi-head self-attention layer, an MLP layer, an output layer, and a task-specific layer; The calculation formula of the input layer is: in, is the symbol of the real number set, is the input image, specifically a multi-channel image formed by summarizing the touch input data and the depth image, are the image height, width and number of channels respectively, is the image after segmentation, is the number of image blocks, is the size of the image block, is the initial input sequence, is the embedding matrix, is the position code, For time coding, For classification marks, Hidden dimensions of the model for interactive mixing of light and shadow; The Transformer encoder has the following calculation formula: in, is the index, , is the number of Transformer layers, For the The output of the layer Transformer, For the The output of the layer Transformer, For the The intermediate output after multi-head self-attention in the layer, MSA is multi-head self-attention, LN is layer normalization, and MLP is multi-layer perceptron; The calculation formula of the multi-head self-attention layer is: in, are the weight matrices, are matrices, is the dimension of each attention head, is the number of attention heads, is the attention weight matrix, For the The output of an attention head is is the output projection matrix; The calculation formula of the MLP layer is: in, are the weight and bias of the first layer respectively, are the second layer weight and bias respectively, is the hidden dimension of the feedforward network, and GELU is the activation function; The calculation formula of the output layer is: in, is the first token output by the last layer of Transformer. It is the final output of the light and shadow interactive hybrid control model; The calculation formula of the task specific layer is: in, are the weight and bias of interaction type prediction, respectively. are the weight and bias of interaction strength prediction, respectively. are the weight and bias of uncertainty estimation, respectively, are the intermediate hidden layer outputs for interaction type and interaction strength prediction, respectively. Predict the results for the interaction type, is the interaction strength prediction result, are the uncertainty estimates for interaction type and interaction strength predictions, respectively. is the sigmoid activation function, ReLU, softmax, and softplus are activation functions respectively.

2. The light and shadow interactive control method for a sandbox according to claim 1, characterized in that: The specific training process of the light-shadow interactive hybrid control model is as follows: Obtain the historical data of light and shadow interaction of the sandbox, pre-process the historical data of light and shadow interaction of the sandbox, and divide it into a training set and a test set; The training set is input into the light and shadow interaction hybrid control model for training, the predicted sand table light and shadow interaction type and the corresponding interaction strength are obtained, and the loss value between the test set and the predicted sand table light and shadow interaction type and the corresponding interaction strength is calculated; According to the gradient of the loss value relative to the light and shadow interactive hybrid control model, the parameters of the light and shadow interactive hybrid control model are iteratively updated through the Adam optimization algorithm until the maximum number of iterations is reached, and the trained light and shadow interactive hybrid control model is output.

3. The light and shadow interactive control method for a sandbox according to claim 2, characterized in that: The loss value between the calculated test set and the predicted sandbox light and shadow interaction type and the corresponding interaction intensity is calculated using the following specific formula: in, is the total loss objective function, are the parameters of the light and shadow interactive mixing control model, is the weight coefficient of the interaction type loss, is the weight coefficient of interaction strength loss, is the weight coefficient of uncertainty loss, is the weight coefficient of the regularization term, is the loss function for interaction type prediction, is the loss function for interaction strength prediction, is the loss function of uncertainty estimation, is the regularized loss function, For the The true interaction type labels of training samples, The light and shadow interactive hybrid control model is used for the Interaction type prediction for training samples, The light-shadow interaction hybrid control model predicts the interaction intensity. is the true interaction strength label, is the uncertainty estimation of the interaction type prediction of the light-shadow interaction hybrid control model. is the uncertainty estimation of the interaction intensity prediction of the light-shadow interaction hybrid control model, It is the weight matrix of the light and shadow interaction mixing control model.