Self-adaptive flat plate angle adjustment method and system based on hidden space data fusion, storage medium and equipment

Abstract

The invention relates to an adaptive panel angle adjustment method and system based on hidden space data fusion, a storage medium and equipment. The method comprises the following steps: S1, acquiring human face and light field environment information by using a camera and a photosensitive element respectively; s2, establishing a head posture estimation model and a light field model by using an artificial neural network, and extracting related features; s3, establishing a multi-modal fusion framework based on Transform, performing interconnection fusion on features output by the head posture estimation model and the light field model, and outputting multi-modal fusion features; and S4, inputting the multi-modal fusion features into a full-connection layer, outputting a final result, and obtaining an optimal adjustment angle of the panel support. According to the method, a camera and a photosensitive element are adopted to collect face information and light field information respectively, a deep neural network is utilized to establish a head posture estimation model and a light field model, the two models are interconnected and fused, and finally, the optimal visual angle is output through the fused model, so that self-adaptive adjustment of the panel is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data fusion and head posture estimation, and in particular to an adaptive tablet angle adjustment method, system, storage medium and device based on latent space data fusion. Background Art

[0002] The main function of the adaptive tablet angle adjustment method is to adjust the tablet display angle for users to obtain the best viewing angle, thereby providing a better visual experience and comfort. To achieve automatic adjustment of the tablet, it is necessary to consider the ambient light factor and analyze the head posture in conjunction with the face image to ensure that the tablet is at the best viewing angle. There are many existing solutions for single scenarios, but in complex scenarios, the existing technology has poor effects and cannot coordinate the processing of the two types of data.

[0003] In the existing technical solutions, for the solution of adjusting the bracket by collecting ambient light, such as the light-sensing angle adjustment intelligent bracket with publication number CN114893690A, the function of adjusting the bracket by ambient light is realized; this invention uses a light sensor to detect the light intensity and color temperature of the environment to drive the movement of its four motors, so as to adjust the tablet to the best viewing angle and visual effect. For the solution of the user's head posture, such as the display intelligent bracket and its control method with publication number CN109595438A, it involves image processing of the user's head posture and dynamic adjustment of the bracket, so as to achieve the parallelism between the tablet and the face.

[0004] The above technical solutions only consider a single factor, but in actual experience, the environment is relatively complex. If only the lighting or the face angle is considered, the model has great limitations and cannot make the tablet at the best viewing angle. Only by considering both at the same time can better effects be achieved in complex scenes. Summary of the invention

[0005] In order to solve the technical problems existing in the prior art, the present invention provides an adaptive tablet angle adjustment method based on latent space data fusion. A camera and a photosensitive element are used to collect facial information and light field information respectively, and a deep neural network is used to establish a head posture estimation model and a light field model. The two models are interconnected and fused. Finally, the optimal viewing angle is output through the fused model to achieve adaptive adjustment of the tablet.

[0006] The method of the present invention is implemented by the following technical scheme: an adaptive flat panel angle adjustment method based on latent space data fusion, comprising the following steps:

[0007] S1, using a camera and a photosensitive element to collect facial information and light field environment information respectively;

[0008] S2. Use artificial neural networks to establish a head posture estimation model and a light field model, process facial information and light field environment information respectively, and extract relevant features;

[0009] S3. A multimodal fusion framework is established based on Transformer to interconnect and fuse the features output by the head posture estimation model and the light field model, adjust and correct the face angle information according to the light field environment information, and output multimodal fusion features.

[0010] S4. By inputting the multimodal fusion features into the fully connected layer, the final result is output to obtain the optimal adjustment angle of the flat panel bracket.

[0011] The system of the present invention is implemented by the following technical solutions: an adaptive flat panel angle adjustment system based on latent space data fusion, comprising: an information acquisition module: collecting face information and light field environment information respectively by using a camera and a photosensitive element;

[0012] Model building module: By using artificial neural networks to build head posture estimation models and light field models, the facial information and light field environment information are processed respectively, and relevant features are extracted;

[0013] Feature fusion module: A multimodal fusion framework is established based on Transformer, the features output by the head posture estimation model and the light field model are interconnected and fused, the face angle information is adjusted and corrected according to the light field environment information, and the multimodal fusion features are output;

[0014] Tablet adjustment angle acquisition module: By inputting multimodal fusion features into the fully connected layer, the final result is output to obtain the optimal adjustment angle of the tablet stand.

[0015] The present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the adaptive flat panel angle adjustment method of the present invention are implemented.

[0016] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the adaptive flat panel angle adjustment method of the present invention is implemented.

[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0018] 1. The present invention adopts a camera and a photosensitive element to collect facial information and light field information respectively, uses a deep neural network to establish a head posture estimation model and a light field model, and interconnects and fuses the two models. Finally, the fused model outputs the optimal viewing angle to achieve adaptive adjustment of the tablet.

[0019] 2. The present invention adopts the VGG16+Dilated-CNN model to extract a wider range of image context information through dilated convolution. This model structure enhances the understanding of global information and helps to improve model performance, especially when processing large-scale images, and can better capture key features.

[0020] 3. The present invention inputs the light field environment signal into the LSTM network, and captures the temporal correlation and key features by gradually analyzing the time series data. The gating mechanism of the LSTM network enables it to selectively forget and store information, thereby better understanding the patterns and structures in the sequence, which helps to improve the effectiveness of the light field information analysis and make the model more robust.

[0021] 4. The present invention realizes multimodal data fusion by adopting Transformer. With its powerful global information capture capability, it can effectively integrate all facial features and ambient light information with time series. Therefore, the generalization ability of the model is stronger and better results can be obtained in complex scenes.

[0022] 5. The present invention can make full use of different types of data provided by cameras and photosensitive elements by simultaneously collecting facial information and light field environment information, thereby obtaining more comprehensive and accurate information. The fusion of such multimodal information can improve the generalization ability of the model and enhance the accuracy and robustness of the face recognition system. By fusing the light field environment information, it can also effectively deal with the recognition problem of lighting changes when the lighting changes greatly, and improve the accuracy of the optimal viewing angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the model structure of the method of the present invention;

[0024] Figure 2 It is a data collection flow chart of the present invention;

[0025] Figure 3 It is a schematic diagram of multi-modal data collection of user's subjective feelings of the present invention;

[0026] Figure 4 This is a schematic diagram of the VGG16+Dilated CNN structure;

[0027] Figure 5 This is a schematic diagram of the LSTM structure. DETAILED DESCRIPTION

[0028] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0029] Example

[0030] like Figure 1 , Figure 2 As shown, the adaptive tablet angle adjustment based on latent space data fusion in this embodiment mainly involves the following technologies: 1) A data fusion method based on Transformer: a camera and a photosensitive element are used to collect face information and light field information respectively, a head posture estimation model and a light field model are established by a deep neural network, and the two models are interconnected and fused. Finally, the optimal viewing angle is output through the fused model to achieve adaptive adjustment of the tablet; 2) A multimodal data acquisition method: data is collected based on the user's subjective feelings, and the data collected in the experiment is used as the data set. The data fusion method is used in combination with artificial neural network technology to effectively solve the impact of lighting changes and face rotation on the visual comfort of the tablet.

[0031] like Figure 1 As shown, the adaptive flat panel angle adjustment method based on latent space data fusion in this embodiment includes the following steps:

[0032] S1, using a camera and a photosensitive element to collect facial information and light field environment information respectively;

[0033] S2. Use artificial neural networks to establish a head posture estimation model and a light field model, process facial information and light field environment information respectively, and extract relevant features;

[0034] S3. A multimodal fusion framework is established based on Transformer to interconnect and fuse the features output by the head posture estimation model and the light field model, and the face angle information is adjusted and corrected according to the light field environment information, thereby outputting multimodal fusion features.

[0035] S4. The multimodal fusion features are input into the fully connected layer to output the final result and obtain the optimal adjustment angle of the flat panel bracket.

[0036] Specifically, in this embodiment, in step S1, a commonly used USB camera, such as Logitech C270, is used to capture facial information; in order to ensure the quality of the collected light field environment information, a photosensitive element, such as a TSL2561 photosensitive element, is used to collect light field environment information.

[0037] Specifically, Figure 2 , Figure 3 As shown, regarding the specific situation of information collection, the present invention can be used to solve the situation where the current public data set is not applicable by designing a multimodal data collection process based on the subjective feelings of users. Because the design is based on the subjective feelings of users, the generalization ability of the model will gradually increase with the increase of volunteers. The method mainly includes environment preparation, data pair collection, data subset construction and complete data set construction, which are described in detail as follows:

[0038] Environmental preparation: Choose an environment suitable for the experiment. There should be uneven light in the environment to simulate the real situation. Prepare a tablet device and a tablet stand. The tablet stand should be equipped with a photosensitive module and a camera. The photosensitive module is used to record the intensity and color information of the ambient light, and the camera is used to record the posture of the face.

[0039] Data pair 1 collection: The experimenter sits in front of the tablet, with uneven light coming from behind; the photosensitive module of the bracket records the current ambient light information, including the intensity and color of the light. The TSL2561 photosensitive element is used; the Logitech C270 camera of the bracket records the current face posture, including head angle, facial expression, etc. The face key coordinates and posture estimation algorithm are used to extract the face posture information, and OpenCV, Dlib and other libraries are used for face detection and key point positioning.

[0040] Data pair 2 collection: The experimenter manually adjusts the angle and position of the tablet to obtain the best visual situation. Use a stand or adjust the support angle of the tablet to adjust the tablet. The photosensitive module of the stand records the adjusted ambient light information again. The camera of the stand records the posture of the face after adjustment again, including the head angle, facial expression, etc. At the same time, the posture of the tablet is recorded, including the tilt angle and translation position, etc. Sensors such as gyroscopes, accelerometers, and magnetometers are used to obtain the posture information of the tablet.

[0041] Data subset construction: Combine data pair 1 and data pair 2 into a data subset, which contains ambient light information, face posture information, and tablet posture information. Store the data as images, videos, or data files for subsequent processing and analysis.

[0042] Construction of a complete data set: By changing the experimenter and adjusting the experimenter's head posture, the angle of the light source behind him, etc., the above steps are repeated to continuously collect data subsets and gradually build a complete data set.

[0043] Specifically, when collecting data, it is necessary to ensure that the changes in the experimenter and the ambient light have a certain degree of randomness to ensure the diversity and representativeness of the data set. The diversity of the data can be increased by randomly selecting the experimenter and adjusting the position and angle of the light source. In order to obtain more accurate data, multiple photosensitive modules and cameras can be used during the data collection process to improve the accuracy and reliability of data collection. Multiple photosensitive modules and cameras can be placed in different positions to obtain more comprehensive data information.

[0044] like Figure 4As shown, in this embodiment, the head posture estimation model in step S2 is established by passing the face information captured by the USB camera into the VGG16+Dilated-CNN artificial neural network, wherein the face image passed in is an RGB color image, including 3 color channels, and the size of the face image is 224X224 pixels. The network structure of the VGG16+Dilated-CNN model consists of Conv1-5, Dilated-Conv1-4, Pool1-2, and Fc1-2 layers. The specific process includes:

[0045] S211, pass the VGG16 convolution layer Conv1 to receive the 3 channel data of the input image, the convolution layer Conv2 receives the feature map of 64 channels, uses a 3x3 convolution kernel, and outputs the feature map of 64 channels; enter the pooling layer Pool1, and reduce the size of the feature map to half of the original; the convolution layer Conv3 converts the input 64 channel feature map to a 128 channel feature map; the convolution layer Conv4 continues to process the 128 channel feature map; perform the pooling layer Pool2 operation, and reduce the size of the feature map to half of the original size again; the convolution layer Conv5 uses a 1x1 convolution kernel to adjust the feature map channel, and converts the input 128 channel feature map to a 64 channel feature map; each convolution layer is normalized and ReLU activation function is performed after each convolution layer;

[0046] S212, the input to the dilated convolution layer Dilated-Conv1 uses a 3x3 convolution kernel and a dilation rate of 2; the input is a feature map of 64 channels, and the output is still a feature map of 64 channels; the dilated convolution layer Dilated-Conv2 continues to process the feature map of 64 channels, using a 3x3 convolution kernel and a dilation rate of 3; the dilated convolution layers Dilated-Conv3 and Dilated-Conv4 use dilation rates of 5 and 11 respectively for feature extraction in a wider range; each dilated convolution layer is normalized and ReLU activation function is performed, and the final output feature map is 14×14×128;

[0047] S213, the feature map is input into the Fc1 fully connected layer and flattened into a one-dimensional vector, and then passed through the Fc2 fully connected layer, and finally reduced to a 1024 feature vector as output.

[0048] Specifically, in this embodiment, the light field model in step S2 is established by inputting the light field environment information collected by the photosensitive element into the LSTM network, and the specific process includes:

[0049] S221, LSTM network processes input signals in a time series manner, operating step by step; each time step includes an input gate (i t ), forget gate (ft ), cell state and output gate (o t ),like Figure 5 As shown in the figure; the input gate uses the sigmoid activation function to control the input signal of the current time step to be integrated into the cell state, and add information to the cell state. The forget gate uses the sigmoid activation function to control the retention and discarding of old information in the cell state; according to the output of the input gate and the forget gate, the LSTM network updates the cell state; new information is added to the cell state, and old information is forgotten; the output gate uses the sigmoid activation function to determine that the information in the cell state is passed to the hidden state of the current time step; at the same time, the tanh activation function is applied to the cell state to adjust the output amplitude; through the output of the output gate and the tanh transformation of the cell state, the LSTM network calculates the hidden state of the current time step; among them, the hidden state contains key information about the input sequence;

[0050] S222, the LSTM network will repeatedly perform the time step processing in step S221 on the entire input sequence to gradually analyze the data of each time step;

[0051] S223. In order to unify the dimension of the feature vector output by the head posture estimation model to 1024, a fully connected layer is added at the end of the LSTM network. The parameters of this layer are trained to map the output of the LSTM network to 1024 dimensions to generate the final feature vector as output.

[0052] Specifically, the specific process of step S3 includes: inputting the feature vectors output by the head posture estimation model and the light field model into the Transformer fusion module, using the self-attention mechanism to capture the global features, thereby automatically learning and capturing the complex relationship and correlation between the two feature vectors, and interconnecting and fusing them into a single 1024-dimensional feature vector.

[0053] Specifically, the specific process of step S4 includes: inputting the multimodal fusion features into the fully connected layer, outputting a two-dimensional vector, which is the pitch angle and the yaw angle, and transmitting it back to the controller to adjust the motor rotation to achieve adaptive adjustment of the tablet angle.

[0054] Based on the same inventive concept, the present invention also proposes an adaptive flat panel angle adjustment system based on latent space data fusion, comprising:

[0055] Information collection module: collects facial information and light field environment information by using cameras and photosensitive elements respectively;

[0056] Model building module: By using artificial neural networks to build head posture estimation models and light field models, the facial information and light field environment information are processed respectively, and relevant features are extracted;

[0057] Feature fusion module: A multimodal fusion framework is established based on Transformer, the features output by the head posture estimation model and the light field model are interconnected and fused, the face angle information is adjusted and corrected according to the light field environment information, and the multimodal fusion features are output;

[0058] Tablet adjustment angle acquisition module: By inputting multimodal fusion features into the fully connected layer, the final result is output to obtain the optimal adjustment angle of the tablet stand.

[0059] In addition, the present invention also proposes a storage medium and a computer device. The storage medium stores a computer program, and when the computer program is executed by a processor, steps S1-S4 of the adaptive flat panel angle adjustment method of the present invention are implemented. The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the adaptive flat panel angle adjustment method of the present invention is implemented, that is, the process including the above steps S1-S4.

[0060] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. An adaptive flat panel angle adjustment method based on latent space data fusion, characterized in that: The following steps are involved: S1, using a camera and a photosensitive element to collect facial information and light field environment information respectively; S2. Use artificial neural networks to establish a head posture estimation model and a light field model, process facial information and light field environment information respectively, and extract relevant features; S3. A multimodal fusion framework is established based on Transformer to interconnect and fuse the features output by the head posture estimation model and the light field model, adjust and correct the face angle information according to the light field environment information, and output multimodal fusion features. S4. By inputting the multimodal fusion features into the fully connected layer, the final result is output to obtain the optimal adjustment angle of the flat panel bracket.

2. The method for adaptive flat panel angle adjustment based on latent space data fusion according to claim 1, characterized in that: In step S2, the head posture estimation model is established by passing the face information captured by the USB camera into the VGG16+Dilated-CNN artificial neural network, where the face image passed in is an RGB color image, including 3 color channels, and the size of the face image is 224X224 pixels. The network structure of the VGG16+Dilated-CNN model consists of Conv1-5, Dilated-Conv1-4, Pool1-2, and Fc1-2 layers. The specific process includes: S211, pass the VGG16 convolution layer Conv1 to receive the 3 channel data of the input image, the convolution layer Conv2 receives the feature map of 64 channels, uses a 3x3 convolution kernel, and outputs the feature map of 64 channels; enter the pooling layer Pool1, and reduce the size of the feature map to half of the original; the convolution layer Conv3 converts the input 64 channel feature map to a 128 channel feature map; the convolution layer Conv4 continues to process the 128 channel feature map; perform the pooling layer Pool2 operation, and reduce the size of the feature map to half of the original size again; the convolution layer Conv5 uses a 1x1 convolution kernel to adjust the feature map channel, and converts the input 128 channel feature map to a 64 channel feature map; each convolution layer is normalized and ReLU activation function is performed after each convolution layer; S212, the input to the dilated convolution layer Dilated-Conv1 uses a 3x3 convolution kernel and a dilation rate of 2; the input is a feature map of 64 channels, and the output is still a feature map of 64 channels; the dilated convolution layer Dilated-Conv2 continues to process the feature map of 64 channels, using a 3x3 convolution kernel and a dilation rate of 3; the dilated convolution layers Dilated-Conv3 and Dilated-Conv4 use dilation rates of 5 and 11 respectively for feature extraction; each dilated convolution layer is normalized and ReLU activation function is performed, and the final output feature map is 14×14×128; S213, the feature map is input into the Fc1 fully connected layer and flattened into a one-dimensional vector, and then passed through the Fc2 fully connected layer, and finally reduced to a 1024 feature vector as output.

3. The method for adaptive flat panel angle adjustment based on latent space data fusion according to claim 1, characterized in that: In step S2, the light field model is established by inputting the light field environment information collected by the photosensitive element into the LSTM network. The specific process includes: S221, LSTM network processes input signals in a time series manner, operating step by step; each time step includes an input gate (i t ), forget gate (f t ), cell state and output gate (o t ); the input gate uses the sigmoid activation function to control the input signal of the current time step to be integrated into the cell state, and add information to the cell state. The forget gate uses the sigmoid activation function to control the retention and discarding of old information in the cell state; according to the output of the input gate and the forget gate, the LSTM network updates the cell state; new information is added to the cell state, and old information is forgotten; the output gate uses the sigmoid activation function to determine that the information in the cell state is passed to the hidden state of the current time step; at the same time, the tanh activation function is applied to the cell state to adjust the output amplitude; through the output of the output gate and the tanh transformation of the cell state, the LSTM network calculates the hidden state of the current time step; where the hidden state contains key information about the input sequence; S222, the LSTM network will repeatedly perform the time step processing in step S221 on the entire input sequence to gradually analyze the data of each time step; S223. Add a fully connected layer at the end of the LSTM network. The parameters of this layer are trained to map the output of the LSTM network to 1024 dimensions to generate the final feature vector as output.

4. The method for adaptive flat panel angle adjustment based on latent space data fusion according to claim 1, characterized in that: The specific process of step S3 includes: inputting the feature vectors output by the head posture estimation model and the light field model into the Transformer fusion module, using the self-attention mechanism to capture global features, automatically learning and capturing the complex relationship and correlation between the two feature vectors, and interconnecting and fusing them into a single 1024-dimensional feature vector.

5. The method for adaptive flat panel angle adjustment based on latent space data fusion according to claim 1, characterized in that: The specific process of step S4 includes: inputting the multimodal fusion features into the fully connected layer, outputting a two-dimensional vector, which is the pitch angle and the yaw angle, and transmitting it back to the controller to adjust the motor rotation for adaptive adjustment of the tablet angle.

6. The adaptive flat panel angle adjustment system based on latent space data fusion is characterized by: include: Information collection module: collects facial information and light field environment information by using cameras and photosensitive elements respectively; Model building module: By using artificial neural networks to build head posture estimation models and light field models, the facial information and light field environment information are processed respectively, and relevant features are extracted; Feature fusion module: A multimodal fusion framework is established based on Transformer, the features output by the head posture estimation model and the light field model are interconnected and fused, the face angle information is adjusted and corrected according to the light field environment information, and the multimodal fusion features are output; Tablet adjustment angle acquisition module: By inputting multimodal fusion features into the fully connected layer, the final result is output to obtain the optimal adjustment angle of the tablet stand.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the adaptive flat panel angle adjustment method according to any one of claims 1 to 5 are implemented.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the adaptive flat panel angle adjustment method according to any one of claims 1 to 5 is implemented.

Images