Control method and device of virtual reality display equipment and storage medium

Abstract

The invention relates to a control method and device of virtual reality display equipment and a storage medium. The control method of the virtual reality display equipment comprises the steps that display information of the virtual reality display equipment is determined, the display information is determined based on the psychological state of a user wearing the virtual reality display equipment, and the psychological state is determined based on multi-modal data of the user; and controlling the virtual reality display equipment to display according to the display information. According to the method and the device, the psychological state can be acquired, the display information is provided based on the psychological state, the psychological health condition can be improved, and the use experience of a user is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular, to a control method, device, and storage medium for a virtual reality display device. Background Art

[0002] With the continuous progress and development of technology, the functions and performance of virtual reality display devices are also constantly improving, bringing a better immersive experience to people. In related technologies, virtual reality display devices can perform theme switching of light and dark styles based on ambient light changes and time changes. However, in related technologies, there is a problem that the virtual display device cannot be controlled for display based on the user data collected by the virtual reality display device. Summary of the Invention

[0003] To overcome the problems existing in related technologies, the present disclosure provides a control method, device, and storage medium for a virtual reality display device.

[0004] According to a first aspect of an embodiment of the present disclosure, a control method for a virtual reality display device is provided, including:

[0005] Determine the display information of the virtual reality display device, where the display information is determined based on the mental state of the user wearing the virtual reality display device, and the mental state is determined based on the multi-modal data of the user; control the virtual reality display device to perform display according to the display information.

[0006] In one implementation, the display information includes at least one of the following:

[0007] Display information; user interface display attribute information, where the display attribute information includes at least one of the following: interface style, color, background picture, interaction method with the user, and operation process.

[0008] In one implementation, the multi-modal data is determined in the following manner:

[0009] Determine the multi-modal data based on the data collected by the virtual reality display device and / or other intelligent wearable devices worn by the user; the multi-modal data includes at least one of the following: eye movement data; facial state expression; gesture state; heart rate information; wrist state.

[0010] In one implementation, the mental state is determined based on the user's multi-modal data in the following manner:

[0011] Based on a first model and the user's current multi-modal data, respectively determine the mental state corresponding to each modality, where the first model is pre-determined based on modality data and mental state; based on the mental state corresponding to each modality, determine the mental state of the user.

[0012] In one implementation, the display information is determined based on the mental state of the user wearing the virtual reality display device in the following manner:

[0013] Based on the mental state of the user and a second model, obtain display information that matches the mental state of the user; the second model is pre-determined based on mental states and display information.

[0014] In one implementation, the obtaining of the display information that matches the mental state of the user based on the mental state of the user and the second model includes:

[0015] Extract features of the mental state of the user through the second model to obtain feature information corresponding to the mental state of the user, and based on the feature information, screen out display information that matches the feature information; or determine the natural language input content of the second model based on the mental state of the user and a prompt template, analyze and process the natural language input content through the second model to generate an output text, and determine the display information that matches the mental state of the user based on the output text.

[0016] In one implementation, the method further includes:

[0017] Adjust the parameters of the first model and / or the second model based on user feedback information and / or user behavior data;

[0018] Retrain the first model and / or the second model based on annotated multi-modal data and / or display information.

[0019] According to the second aspect of the embodiments of the present disclosure, there is provided a control device for a virtual reality display device, including:

[0020] A determination unit, configured to determine the display information of the virtual reality display device, where the display information is determined based on the mental state of the user wearing the virtual reality display device, and the mental state is determined based on the multi-modal data of the user;

[0021] A processing unit, configured to control the virtual reality display device to perform display according to the display information.

[0022] In one implementation, the display information includes at least one of the following:

[0023] Display information; user interface display attribute information, where the display attribute information includes at least one of the following: interface style, color, background picture, interaction method with the user, and operation process.

[0024] In one implementation, the multi-modal data is determined in the following manner:

[0025] Determine the multimodal data based on data collected by the virtual reality display device and / or other intelligent wearable devices worn by the user; the multimodal data includes at least one of the following: eye movement data; facial state expressions; gesture states; heart rate information; wrist states.

[0026] In one implementation, the determining unit determines the mental state based on the multimodal data of the user in the following manner: respectively determine the mental state corresponding to each modality based on the first model and the current multimodal data of the user, where the first model is pre-determined based on modality data and mental states; determine the mental state of the user based on the mental states corresponding to each modality.

[0027] In one implementation, the processing unit determines the display information based on the mental state of the user wearing the virtual reality display device in the following manner: obtain the display information matching the mental state of the user based on the mental state of the user and the second model; the second model is pre-determined based on mental states and display information.

[0028] In one implementation, the processing unit obtains the display information matching the mental state of the user based on the mental state of the user and the second model in the following manner:

[0029] Extract features from the mental state of the user through the second model to obtain the feature information corresponding to the mental state of the user, and based on the feature information, screen out the display information matching the feature information; or determine the natural language input content of the second model based on the mental state of the user and the prompt template, analyze and process the natural language input content through the second model to generate an output text, and determine the display information matching the mental state of the user based on the output text.

[0030] In one implementation, the processing unit is further configured to:

[0031] Adjust the parameters of the first model and / or the second model based on user feedback information and / or user behavior data; re-train the first model and / or the second model based on annotated multimodal data and / or display information.

[0032] According to the third aspect of the embodiments of the present disclosure, there is provided a control device for a virtual reality display device, including:

[0033] A processor; a memory for storing instructions executable by the processor; wherein, the processor is configured to: execute the control method of the virtual reality display device described in the first aspect or any one of the implementations of the first aspect.

[0034] According to a fourth aspect of the embodiments of the present disclosure, there is provided a storage medium storing instructions that, when executed by a processor of a terminal, enable the terminal to perform the method described in any item of the first aspect or any implementation manner of the first aspect.

[0035] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: determining the mental state of a user based on the user's multimodal data, and controlling a virtual reality display device according to the mental state of the user to determine and display display information. It can enhance the application field of the virtual reality display device, improve the user experience, and contribute to improving the user's mental health status.

[0036] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0038] Figure 1 is a flowchart of a control method for a virtual reality display device shown according to an exemplary embodiment.

[0039] Figure 2 is a flowchart of a multimodal data determination method shown according to an exemplary embodiment.

[0040] Figure 3 is a flowchart of a mental state determination method shown according to an exemplary embodiment.

[0041] Figure 4 is a flowchart of a display information determination method shown according to an exemplary embodiment.

[0042] Figure 5 is a flowchart of a display information acquisition method shown according to an exemplary embodiment.

[0043] Figure 6 is a flowchart of a display information acquisition method shown according to an exemplary embodiment.

[0044] Figure 7 is a schematic diagram of a control process of a virtual reality display device shown according to an exemplary embodiment.

[0045] Figure 8 is a block diagram of a product sales volume prediction device shown according to an exemplary embodiment.

[0046] Figure 9It is a block diagram of a device for controlling a virtual reality display device shown according to an exemplary embodiment. Detailed implementation manners

[0047] Here, the exemplary embodiments will be described in detail, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure.

[0048] In the drawings, the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some embodiments of the present disclosure, rather than all embodiments. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present disclosure, and should not be construed as a limitation of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present disclosure without creative efforts shall fall within the scope of protection of the present disclosure. The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0049] The control method for a virtual reality display device provided by an embodiment of the present disclosure is applied to the field of artificial intelligence. With the rapid development of virtual reality technology, how to improve the user experience and provide more personalized services has become a research hotspot. Virtual Reality (VR) technology is a computer technology that can create and experience virtual worlds. It enables users to enter a highly realistic three-dimensional virtual environment as if they were on the scene by simulating human vision, hearing, and touch.

[0050] In related technologies, virtual reality devices usually focus on users' physical interactions and visual experiences, without an adaptive adjustment function, or only include simple functions such as switching the brightness style according to environmental light changes and time. And in related technologies, there is no function to obtain the user's mental state based on the data obtained by the virtual reality device, nor is there a function to determine the display information based on the user's mental state and perform display according to the display information. The user's mental state is ignored, there is no function to detect the user's mental state, nor is there a function to control the display information of the virtual reality display device according to the mental state. For example, adaptively adjusting the display interface and function recommendations based on the user's mental state.

[0051] In related technologies, only some wearable devices have the function of obtaining the user's unimodal data, such as heart rate data, and detecting the mental state based on the unimodal data.

[0052] In view of this, embodiments of the present disclosure provide a control method for a virtual reality display device, which obtains multimodal data through a virtual reality device and a smart bracelet, obtains the user's mental state based on the multimodal data, and then determines the display information of the virtual reality display device according to the user's mental state, including content recommendation information and user interface display attribute information. By obtaining the user's multimodal data, accurately identifying the user's mental state when using the virtual reality device, and controlling the virtual reality display device to display according to the display information determined based on the mental state, the user's virtual reality experience is improved, the mental health condition is improved, and the application field of the virtual reality device is broadened.

[0053] It should be understood that the control method for the virtual reality display device provided by the embodiments of the present disclosure can not only be applied to virtual reality display devices, but also be equally applicable to extended reality (XR) display devices such as augmented reality (AR) display devices and mixed reality (MR) display devices.

[0054] Figure 1 is a flowchart of a control method for a virtual reality display device shown according to an exemplary embodiment, as Figure 1 shown, and includes the following steps.

[0055] In step S11, determine the display information of the virtual reality display device.

[0056] In the embodiments of the present disclosure, the display information is determined based on the mental state of the user wearing the virtual reality display device, and the mental state is determined based on the user's multimodal data. Among them, multimodal data is used to represent data obtained by the user from different fields or perspectives, and these data include different forms of existence or information sources, such as text, pictures, audio, video, mixed data, etc. Using multimodal data to determine the user's mental state can improve the accuracy of determining the user's mental state and has better anti-interference ability compared with determining the mental state based on a single data.

[0057] In the embodiments of the present disclosure, the mental state of the user can be determined by other electronic devices based on the user's multimodal data, and the display information of the virtual reality display device can be determined based on the mental state of the user. For example, a mobile phone is used to connect to the virtual reality display device, receive the multimodal data of the user obtained by each sensor in the virtual reality display device, and use the mobile phone processor to process the multimodal data of the user and determine the mental state of the user.

[0058] In step S12, control the virtual reality display device to display according to the display information.

[0059] In the embodiments of the present disclosure, the determined display information is displayed through a virtual reality display device. It should be understood that the virtual reality device for display may also be other extended reality display devices such as an augmented reality display device or a mixed reality display device. Moreover, the virtual reality display device can control itself to display according to the display information, or other electronic devices connected to the virtual reality display device can control the virtual reality display device to display according to the display information. For example, if a mobile phone determines the display information of the virtual reality display device, the mobile phone will send the display information to be displayed to the virtual reality display device, or directly control the virtual reality display device to display according to the display information.

[0060] In the embodiments of the present disclosure, the determined display information of the virtual reality display device at least includes display information or user interface display attributes.

[0061] In the embodiments of the present disclosure, the display information mainly includes entertainment recommendation information such as music, food, games, and movies, and also includes weather information, sports information, etc. For example, when the user's mental state is inclined to anxiety, the display information will include some content that can help the user relax, such as light exercise, food, movies, or games; when the user's mental state is inclined to anxiety, the display information will include some content that can boost the user's mood, such as sunny scenery, cheerful music, etc.

[0062] In the embodiments of the present disclosure, for the user interface display attribute information, the display attribute information includes at least one of the following: interface style, color, background image, interaction method with the user, and operation process. For example, when the user's mood is inclined to anxiety or tension, the interface can be adjusted to a cool color tone, and a natural scenery can be selected as the background image to help the user relax; when the user's mood is inclined to melancholy or sadness, the interface can be adjusted to a warm color tone, and a sunny scenery can be selected as the background image to boost the user's mood.

[0063] It should be understood that in the embodiments of the present disclosure, the process of obtaining the user's multimodal data, determining the user's mental state, and determining and displaying the display information of the virtual reality display device based on the user's mental state is carried out in real time, that is, the above content can be executed at fixed time intervals to realize the dynamic adjustment of the display information of the virtual reality display device.

[0064] In the embodiments of the present disclosure, the multimodal data can not only include the data obtained by the sensors in the virtual reality display device, but also be based on the data obtained by the sensors of other intelligent wearable devices, and the multimodal data is determined based on the data obtained by the sensors of the virtual reality device and other intelligent wearable devices.

[0065] Figure 2It is a flowchart of a multimodal data determination method shown according to an exemplary embodiment, as Figure 2 shown, and includes the following steps.

[0066] In step S21, multimodal data is determined based on data collected by a virtual reality display device and / or other intelligent wearable devices worn by a user.

[0067] In the embodiments of the present disclosure, the multimodal data includes at least one of the following: eye movement data; facial state expressions; gesture states; heart rate information; wrist states. Among them, the eye movement data can be obtained based on an eye movement camera in the virtual reality display device; the facial state expressions can be obtained based on a facial capture sensor or a facial camera in the virtual reality display device; the gesture state information can be obtained based on a glove-type sensor compatible with the virtual reality display device or a camera in the virtual reality display device; the heart rate information can be obtained based on an infrared sensor in the virtual reality display device or a sensor of other intelligent wearable devices; the wrist state can be obtained based on a glove-type sensor compatible with the virtual reality display device or a sensor of other intelligent wearable devices. It should be understood that the above-mentioned acquisition methods of multimodal data are only for illustrative purposes, and the embodiments of the present disclosure do not limit the methods for acquiring multimodal data.

[0068] In step S22, the display information of the virtual reality display device is determined.

[0069] In step S23, the virtual reality display device is controlled to perform a display according to the display information.

[0070] Since Figure 2 the steps in step S22 and step S23 are the same as the steps in Figure 1 and are not described herein again. Reference can be made to the relevant descriptions of the above embodiments.

[0071] In the embodiments of the present disclosure, after acquiring the multimodal data, feature extraction is performed on the data of each modality in the multimodal data, and the current mental state of the user is determined based on the extracted feature information.

[0072] Figure 3 It is a flowchart of a mental state determination method shown according to an exemplary embodiment, as Figure 3 shown, and includes the following steps.

[0073] In step S31, based on the first model and the user's current multimodal data, the mental state corresponding to each modality is respectively determined.

[0074] In the embodiments of the present disclosure, the first model is pre-determined based on modal data and mental state, and the first model can be a Convolutional Neural Network (CNN). By splitting the user's current multi-modal data into multiple single-modal data, using the first model to extract features from the multiple single-modal data respectively, and determining the mental state corresponding to the multiple single-modal data based on the extracted feature information.

[0075] In step S32, based on the mental state corresponding to each modality, determine the mental state of the user.

[0076] In the embodiments of the present disclosure, the mental state corresponding to each modality can be weighted, and the mental states corresponding to each single-modal data are integrated to finally determine the mental state of the user. This reduces the impact on the accuracy of the finally determined mental state of the user due to errors in the data of a certain modality.

[0077] In the embodiments of the present disclosure, after determining the mental state of the user, the display information of the virtual reality display device can be determined based on the mental state of the user and the second model.

[0078] Figure 4 It is a flowchart of a method for determining display information shown according to an exemplary embodiment, as Figure 4 shown, including the following steps.

[0079] In step S41, determine multi-modal data based on the data collected by the virtual reality display device and / or other intelligent wearable devices worn by the user.

[0080] In step S42, based on the mental state of the user and the second model, obtain the display information that matches the mental state of the user.

[0081] In the embodiments of the present disclosure, the second model is pre-determined based on mental state and display information, and the second model can be a Convolutional Neural Network model or a Large Language Model (LLM). The mental state of the user is used as the input information or part of the input information and input into the second model, the output result of the second model is obtained, and a similarity search is performed on the output result of the second model to determine the display information that matches the mental state of the user.

[0082] In step S43, control the virtual reality display device to display according to the display information.

[0083] Since Figure 4 the steps in step S41 and step S43 are the same as the steps in Figure 2 herein, they will not be repeated, and reference can be made to the relevant descriptions of the above embodiments.

[0084] In the embodiments of the present disclosure, the second model may be a convolutional neural network model or a large language model. Different models have different processing methods and output contents for the user's mental state.

[0085] Figure 5 It is a flowchart of a display information acquisition method shown according to an exemplary embodiment, as Figure 5 shown, and includes the following steps.

[0086] In step S51, multimodal data is determined based on the data collected by the virtual reality display device and / or other intelligent wearable devices worn by the user.

[0087] In step S52a, the second model extracts features from the user's mental state to obtain feature information corresponding to the user's mental state, and based on the feature information, the display information matching the feature information is filtered out.

[0088] In the embodiments of the present disclosure, when the second model is a convolutional neural network model, features are extracted from the user's mental state to obtain a feature value or a feature vector corresponding to the user's mental state, that is, the feature information corresponding to the user's mental state. Based on the obtained feature information of the user's mental state, a similarity comparison is made with the feature information already existing in the database to obtain at least one feature information with a similarity greater than a preset threshold, and the display information corresponding to at least one feature information with a similarity greater than the preset threshold is output as the display information corresponding to the user's mental state.

[0089] In step S52b, based on the user's mental state and the prompt template, the natural language input content of the second model is determined. The second model analyzes and processes the natural language input content to generate an output text, and based on the output text, the display information matching the user's mental state is determined.

[0090] In the embodiments of the present disclosure, when the second model is a large language model, a prompt template is set. The prompt template can enable the second model to better understand the user's mental state and give more appropriate display information. For example, the prompt template can be "My current mood is (), please recommend some movies, music, games and food according to my mood", where the parentheses are the user's mental state. The user's mental state is input into the preset prompt template, and the prompt template containing the user's mental state is input into the second model. The second model combines the semantics and context of the input content to generate reasonable recommendation content and outputs it as the output text. By splitting and processing the output text and performing a similarity search in the database, at least the display information matching the user's mental state is obtained.

[0091] In step S53, the virtual reality display device is controlled to display according to the display information.

[0092] Since Figure 5 the steps in step S51 and step S53 in Figure 4 are the same as the steps in step S41 and step S43 in

[0093] In the embodiments of the present disclosure, based on the user's evaluation and actions on the display information, the first model and / or the second model are optimized to provide display information that better conforms to the user's mental state.

[0094] Figure 6 is a flowchart of a method for obtaining display information shown according to an exemplary embodiment. As Figure 6 shown, it includes the following steps.

[0095] In step S61, determine the display information of the virtual reality display device.

[0096] In step S62, control the virtual reality display device to display according to the display information.

[0097] Since Figure 6 the steps in step S61 and step S62 in Figure 1 are the same as the steps in step S11 and step S12 in

[0098] In the embodiments of the present disclosure, the first model and / or the second model are optimized by collecting user feedback information and / or analyzing user behavior data.

[0099] In step S63, based on the user feedback information and / or user behavior data, adjust the parameters of the first model and / or the second model.

[0100] In the embodiments of the present disclosure, collecting user feedback information includes: displaying a questionnaire or a user evaluation request in the virtual reality display device; in response to completing the questionnaire or user evaluation, using the questionnaire or user evaluation as user feedback information. For example, by sending a questionnaire to the user or inviting the user to make an evaluation, requesting to obtain the user's feedback information, and using the completed questionnaire or evaluation of the user as the result of the last obtained user mental state or the feedback data for controlling the virtual reality display device to display according to the display information.

[0101] In the embodiments of the present disclosure, analyzing user behavior data includes: annotating multimodal data and / or display information in response to the operation situation of the user on the display information. For example, continuously collecting the usage situation of the user on the display information, such as the click-through rate, residence time, score, etc. of the user on the display information, as well as the smoothness of the user's interface operation, operation error rate, etc., and scoring the multimodal data and / or display information based on the above content, and using the score as annotation information to annotate the multimodal data and / or display information.

[0102] In step S64, based on the annotated multimodal data and / or display information, retrain the first model and / or the second model.

[0103] In the embodiments of the present disclosure, optimizing the first model and the second model includes: adjusting the parameters of the first model and / or the second model based on user feedback information and / or user behavior data; or training the first model and / or the second model based on the annotated multimodal data and / or display information. For example, in response to the user's feedback information being a negative review, use an automated tool to adjust the parameters of the first model and / or the second model; or input the annotated multimodal data into the first model, and train the first model based on this data to optimize the processing ability of the model.

[0104] Figure 7 It is a schematic diagram of the control process of a virtual reality display device shown according to an exemplary embodiment.

[0105] In Figure 7Among them, the multimodal information includes eye movement information, facial data, gesture information, wrist status, and heart rate information. Among them, the eye movement information, facial status expressions, and gesture status are captured in real time by setting an eye movement tracking device and an omnidirectional camera on the AR / VR / MR glasses. The heart rate information and wrist status are obtained through a bracelet device worn by the user. The multimodal information provides rich information about the user's current behavior and physiological condition, which helps to more accurately infer the user's mental state. The first model processes the collected multimodal information through a deep learning algorithm, and the first model can be a convolutional neural network (CNN). Such models have been widely used in the processing of complex data such as images, texts, and audios and have good performance. Through the training of a large number of users' mental states and corresponding multimodal information data, the first model can learn the complex relationship between different mental states and multimodal information, so as to accurately infer the user's mental state. According to the inferred mental state of the user, the virtual reality display device or electronic device will recommend a series of content recommendation information matching the user's mental state as display information based on the second model, including music, food, games, and movies. Among them, the recommendation algorithm can include collaborative filtering, content-based recommendation, deep learning, etc. And, according to the user's mental state, the system will adaptively adjust the display attribute information of the user interface in the virtual reality device, including the user interface and the background picture, to improve the user's comfort and satisfaction. During the user's use process, the system will continuously collect the user's feedback on the recommended content and the usage situation of the user interface, such as the click-through rate, stay time, score, etc. of the user on the recommended content, and the smoothness of the user's interface operation, operation error rate, etc. The feedback information is used as new training data to optimize the deep learning model and the recommendation algorithm, so that the mental state detection and recommendation functions of the system are continuously optimized to better meet the user's needs.

[0106] In the embodiments of the present disclosure, the mental state of the user is determined through the user's multimodal information, the display information of the virtual reality display device is determined based on the mental state of the user, and the virtual reality display device is controlled to display according to the display information. It can detect the mental state accurately and provide matching display information, provide an experience that better meets the user's needs, improve the user's feeling of using the virtual reality device, and thus improve the user's satisfaction. And by obtaining the user's mental state, such as anxiety, stress, depression, etc., corresponding psychological intervention measures can be provided in time to help the user relieve stress and improve the mental health condition. And make the virtual reality device not only can be used for games and entertainment, but also can be applied in the fields of mental health and psychological counseling, expanding the application scenarios of the virtual reality device. Provide more personalized and considerate services for users, increase the user's usage frequency, improve the user retention rate, and thus bring more benefits to the device manufacturer.

[0107] Based on the same concept, an embodiment of the present disclosure further provides a control device for a virtual reality display device.

[0108] It can be understood that, in order to implement the above functions, the control device for the virtual reality display device provided by the embodiment of the present disclosure includes the corresponding hardware structure and / or software module for executing each function. Combining the units and algorithm steps of the various examples disclosed in the embodiments of the present disclosure, the embodiments of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving the hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described function, but such implementation should not be considered to exceed the scope of the technical solution of the embodiments of the present disclosure.

[0109] Figure 8 It is a block diagram of a product sales volume prediction device 100 shown according to an exemplary embodiment. Referring to Figure 8 , the device includes a determination unit 101 and a processing unit 102.

[0110] The determination unit 101 is configured to determine the display information of the virtual reality display device, where the display information is determined based on the mental state of the user wearing the virtual reality display device, and the mental state is determined based on the multi-modal data of the user.

[0111] The processing unit 102 is configured to control the virtual reality display device to perform display according to the display information.

[0112] In one embodiment, the display information includes at least one of the following:

[0113] Content recommendation information; user interface display attribute information, where the display attribute information includes at least one of the following: interface style, color, background picture, interaction method with the user, and operation process.

[0114] In one embodiment, the multi-modal data is determined in the following manner:

[0115] The multi-modal data is determined based on the data collected by the virtual reality display device and / or other intelligent wearable devices worn by the user; the multi-modal data includes at least one of the following: eye movement data; facial state expression; gesture state; heart rate information; wrist state.

[0116] In one embodiment, the determination unit 101 determines the mental state based on the multi-modal data of the user in the following manner:

[0117] Based on the first model and the current multi-modal data of the user, the mental state corresponding to each modality is determined respectively, where the first model is pre-determined based on the modality data and the mental state; based on the mental state corresponding to each modality, the mental state of the user is determined.

[0118] In one embodiment, the processing unit 102 determines the display information based on the mental state of the user wearing the virtual reality display device in the following manner:

[0119] Based on the mental state of the user and the second model, obtain the display information that matches the mental state of the user; the second model is a convolutional neural network model or a large language model.

[0120] In one embodiment, the processing unit 102 obtains the display information that matches the mental state of the user based on the mental state of the user and the second model in the following manner:

[0121] Extract features from the mental state of the user through the second model to obtain the feature information corresponding to the mental state of the user, and based on the feature information, filter out the display information that matches the feature information; or determine the natural language input content of the second model based on the mental state of the user and the prompt template, analyze and process the natural language input content through the second model to generate an output text, and determine the display information that matches the mental state of the user based on the output text.

[0122] In one embodiment, the processing unit 102 is further configured to:

[0123] Collect user feedback information and / or analyze user behavior data to optimize the first model and / or the second model; collecting user feedback information includes: displaying a questionnaire or a user evaluation request in the virtual reality display device; in response to completing the questionnaire or the user evaluation, using the questionnaire or the user evaluation as user feedback information; analyzing user behavior data includes: in response to the operation situation of the user on the content recommendation information, annotating the multimodal data and / or the display information;

[0124] Optimizing the first model and the second model includes: adjusting the parameters of the first model and / or the second model based on the user feedback information and / or analyzing the user behavior data; or retraining the first model and / or the second model based on the annotated multimodal data and / or the display information.

[0125] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment related to the method, and will not be elaborated here.

[0126] Figure 9 It is a block diagram of a device 200 for controlling a virtual reality display device shown according to an exemplary embodiment. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0127] Refer toFigure 9 The apparatus 200 may include one or more of the following components: a processing component 202, a memory 204, a power component 206, a multimedia component 208, an audio component 210, an input / output (I / O) interface 212, a sensor component 214, and a communication component 216.

[0128] The processing component 202 generally controls the overall operation of the apparatus 200, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to complete all or part of the steps of the above-described methods. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0129] The memory 204 is configured to store various types of data to support the operation of the apparatus 200. Examples of such data include instructions for any application or method operating on the apparatus 200, contact data, phone book data, messages, pictures, videos, and the like. The memory 204 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0130] The power component 206 provides power to the various components of the apparatus 200. The power component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the apparatus 200.

[0131] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0132] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC) that is configured to receive external audio signals when the device 200 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 204 or transmitted via the communication component 216. In some embodiments, the audio component 210 further includes a speaker for outputting audio signals.

[0133] The I / O interface 212 provides an interface between the processing component 202 and a peripheral interface module, which can be a keyboard, a click wheel, buttons, etc. These buttons can include but are not limited to: a home button, a volume button, a power button, and a lock button.

[0134] The sensor component 214 includes one or more sensors for providing an assessment of the various aspects of the state of the device 200. For example, the sensor component 214 can detect the on / off state of the device 200, the relative positioning of components, such as the display and the keypad of the device 200. The sensor component 214 can also detect a change in the position of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200, and the temperature change of the device 200. The sensor component 214 can include a proximity sensor that is configured to detect the presence of nearby objects without any physical contact. The sensor component 214 can also include a light sensor, such as a CMOS or a CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 214 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0135] The communication component 216 is configured to facilitate communication, in a wired or wireless manner, between the device 200 and other devices. The device 200 may access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0136] In an exemplary embodiment, the device 200 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0137] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 204 including instructions, is also provided, and the above instructions may be executed by the processor 220 of the device 200 to complete the above method. For example, the non-transitory computer-readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0138] It can be understood that the term "a plurality of" in this disclosure means two or more, and other quantifiers are similar thereto. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms of "a", "the", and "said" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0139] It can be further understood that the terms "first", "second", etc. are used to describe various information, but this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other and do not indicate a specific order or importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of this disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0140] It can be further understood that, unless otherwise specified, "connection" includes both direct connection without other components between the two and indirect connection with other elements between the two.

[0141] It can be further understood that although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be construed as requiring the operations to be performed in the specific order shown or in a serial order, or requiring all the operations shown to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0142] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include well-known knowledge or conventional technical means in the technical field not disclosed in the present disclosure.

[0143] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A control method for a virtual reality display device, characterized in that: include: Determining display information of a virtual reality display device, wherein the display information is determined based on a psychological state of a user wearing the virtual reality display device, wherein the psychological state is determined based on multimodal data of the user; Control the virtual reality display device to display according to the display information.

2. The control method of the virtual reality display device according to claim 1, characterized in that: The displayed information includes at least one of the following: Content recommendation information; The user interface displays attribute information, and the display attribute information includes at least one of the following: interface style, color, background image, interaction method with the user, and operation process.

3. The control method of the virtual reality display device according to claim 1 or 2, characterized in that: The multimodal data is determined in the following manner: Determining the multimodal data based on data collected by the virtual reality display device and / or other smart wearable devices worn by the user; The multimodal data includes at least one of the following: Eye movement data; Facial state expressions; Gesture status; Heart rate information; Wrist status.

4. The control method of the virtual reality display device according to claim 1 or 2, characterized in that: The mental state is determined based on the multimodal data of the user in the following manner: Determining the psychological state corresponding to each modality based on a first model and the current multimodal data of the user, wherein the first model is predetermined based on the modal data and the psychological state; The user's psychological state is determined based on the psychological states corresponding to the respective modalities.

5. The control method of the virtual reality display device according to claim 1 or 2, characterized in that: The display information is determined based on the psychological state of the user wearing the virtual reality display device in the following manner: Based on the user's psychological state and the second model, acquiring display information matching the user's psychological state; The second model is predetermined based on the psychological state and the display information.

6. The control method of the virtual reality display device according to claim 5, characterized in that: The acquiring, based on the user's psychological state and the second model, display information matching the user's psychological state includes: Extracting features of the user's psychological state through the second model to obtain feature information corresponding to the user's psychological state, and filtering display information matching the feature information based on the feature information; or Based on the user's psychological state and the prompt template, the natural language input content of the second model is determined, the natural language input content is analyzed and processed by the second model to generate output text, and display information matching the user's psychological state is determined based on the output text.

7. The control method of the virtual reality display device according to claim 1 or 2, characterized in that: The method further comprises: Adjusting parameters of the first model and / or the second model based on user feedback information and / or user behavior data; The first model and / or the second model are retrained based on the annotated multimodal data and / or display information.

8. A control device for a virtual reality display device, characterized in that: include: a determining unit, configured to determine display information of a virtual reality display device, wherein the display information is determined based on a psychological state of a user wearing the virtual reality display device, wherein the psychological state is determined based on multimodal data of the user; The processing unit is used to control the virtual reality display device to display according to the display information.

9. The control device of the virtual reality display device according to claim 8, characterized in that: The displayed information includes at least one of the following: Content recommendation information; The user interface displays attribute information, and the display attribute information includes at least one of the following: interface style, color, background image, interaction method with the user, and operation process.

10. The control device of the virtual reality display device according to claim 8 or 9, characterized in that: The multimodal data is determined in the following manner: Determining the multimodal data based on data collected by the virtual reality display device and / or other smart wearable devices worn by the user; The multimodal data includes at least one of the following: Eye movement data; Facial state expressions; Gesture status; Heart rate information; Wrist status.

11. The control device of the virtual reality display device according to claim 8 or 9, characterized in that: The determining unit determines the psychological state based on the multimodal data of the user in the following manner: Determining the psychological state corresponding to each modality based on a first model and the current multimodal data of the user, wherein the first model is predetermined based on the modal data and the psychological state; The user's psychological state is determined based on the psychological states corresponding to the respective modalities.

12. The control device of the virtual reality display device according to claim 8 or 9, characterized in that: The processing unit determines display information based on the psychological state of the user wearing the virtual reality display device in the following manner: Based on the user's psychological state and the second model, acquiring display information matching the user's psychological state; The second model is predetermined based on the psychological state and the display information.

13. The control device of the virtual reality display device according to claim 12, characterized in that: The processing unit acquires display information matching the user's psychological state based on the user's psychological state and the second model in the following manner: Extracting features of the user's psychological state through the second model to obtain feature information corresponding to the user's psychological state, and filtering display information matching the feature information based on the feature information; or Based on the user's psychological state and the prompt template, the natural language input content of the second model is determined, the natural language input content is analyzed and processed by the second model to generate output text, and display information matching the user's psychological state is determined based on the output text.

14. The control device of the virtual reality display device according to claim 8 or 9, characterized in that: The processing unit is also used for: Adjusting parameters of the first model and / or the second model based on user feedback information and / or user behavior data; The first model and / or the second model are retrained based on the annotated multimodal data and / or display information.

15. A control device for a virtual reality display device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to: execute the method described in any one of claims 1 to 7.

16. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor of the terminal, the terminal is enabled to execute the method according to any one of claims 1 to 7.

Images