Multi-modal information display method and device for augmented reality equipment
By obtaining and analyzing the multimodal data of the target object and displaying multi-dimensional information, it solves the problem that traditional communication analysis is difficult to deeply understand the deep meaning, and realizes the accuracy and immediacy of user decision-making, and is suitable for information display in many occasions.
Patent Information
- Application Number
- CN202411358770.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2025-01-07
AI Technical Summary
Traditional communication and analysis methods are difficult to deeply explore the deep meaning behind dialogue. The existing extended reality technology has limitations in information display, and it is difficult to improve the accuracy and immediacy of user decision-making.
By obtaining multimodal data of the target object, analyzing and determining multi-dimensional information, and displaying multimodal information to users based on inference results, including visual, auditory, tactile and olfactory information, improving the accuracy and timeliness of user decision-making.
It realizes the display of multimodal information to users, improves the accuracy and immediacy of user decision-making, enhances the protection of user safety, optimizes social and communication skills, is suitable for a variety of occasions, and improves personalization and cultural adaptation. sex.
Smart Images
Figure CN119271085A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present application relate to the field of extended reality, and more specifically, to a method, apparatus, electronic device, and storage medium for displaying multimodal information of an extended reality device, wherein the storage medium includes a computer-readable storage medium. Background Art
[0002] With the popularity of smart devices such as smartphones and smart watches, people have become accustomed to obtaining various information through these devices. Extended reality devices can help people identify, understand, and respond to the people or scenes they see, thereby enhancing their abilities. With the development of computer graphics technology, extended reality devices such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) are gradually being applied to people's lives.
[0003] In modern society, people often need to communicate with all kinds of people in various scenarios, including but not limited to entertainment and social interaction, daily work, business negotiations, dating, etc. In these communications, it is particularly important to understand the other party's true intentions, emotional state, and whether their words are true. However, traditional communication analysis methods often only stay at the acquisition of surface information and it is difficult to dig deep into the deep meaning behind the conversation.
[0004] In order to solve this problem and provide a more convenient way to display information, extended reality technology came into being. Extended reality technology integrates virtual information with the real world, allowing users to obtain the information they want in the corresponding actual scene.
[0005] However, current extended reality technology still has some limitations in information display. Summary of the invention
[0006] The embodiments of the present application provide a method, apparatus, electronic device, and storage medium for displaying multimodal information of an extended reality device, which can display multimodal information to a user and improve the accuracy and immediacy of the user's decision-making.
[0007] The present application provides a multimodal information display method for an extended reality device, which is applied to the extended reality device. The method includes:
[0008] Obtain multimodal data of the target object;
[0009] Analyzing the multimodal data of the target object to determine an inference result corresponding to the multimodal data of the target object;
[0010] The multimodal information corresponding to the inference result is displayed to the user according to the inference result.
[0011] In a possible implementation, determining the inference result corresponding to the multimodal data of the target object includes:
[0012] Determining multi-dimensional information of the target object according to the multi-modal data of the target object;
[0013] The inference result is determined according to the multi-dimensional information of the target object.
[0014] In a possible implementation, the multimodal information includes one or more of the following: visual information, auditory information, tactile information, or olfactory information.
[0015] Optionally, before acquiring the multimodal data of the target object, the multimodal information display method of the extended reality device provided in the embodiment of the present application further includes:
[0016] generating a first label of the target object;
[0017] Collecting and analyzing the multimodal data of the target object multiple times according to the first tag to generate a plurality of multi-dimensional information corresponding to the first tag;
[0018] Based on the inference results of the multiple multi-dimensional information, comprehensive multimodal information corresponding to the multiple inference results is displayed to the user.
[0019] In a possible implementation, the visual information includes information visualized as a quality 3D surround representation.
[0020] In a possible implementation, the auditory information includes a prompt sound effect corresponding to the target object.
[0021] In a possible implementation, the tactile information includes vibration of a motor.
[0022] In a possible implementation, the olfactory information includes an odor corresponding to the target object.
[0023] Accordingly, an embodiment of the present application further provides an extended reality device, the device comprising:
[0024] An acquisition module, used to acquire multimodal data of a target object;
[0025] An analysis module, configured to analyze the multimodal data of the target object and determine an inference result corresponding to the multimodal data of the target object;
[0026] A display module is used to display the multimodal information corresponding to the inference result to the user according to the inference result.
[0027] In a possible implementation, the analysis module is used to determine the inference result corresponding to the multimodal data of the target object, including:
[0028] The analysis module is used to determine the multi-dimensional information of the target object according to the multi-modal data of the target object;
[0029] The analysis module is used to determine the inference result according to the multi-dimensional information of the target object.
[0030] In a possible implementation, the multimodal information includes one or more of the following: visual information, auditory information, tactile information, or olfactory information.
[0031] Optionally, before the acquisition module acquires the multimodal data of the target object, the extended reality device provided in the embodiment of the present application further includes:
[0032] A generating module, used for generating a first label of the target object;
[0033] The analysis module is further used to collect and analyze the multimodal data of the target object multiple times according to the first tag to generate multiple multi-dimensional information corresponding to the first tag;
[0034] The display module is also used to display the comprehensive multimodal information corresponding to the multiple inference results to the user based on the inference results of the multiple multi-dimensional information.
[0035] In a possible implementation, the visual information includes information visualized as a quality 3D surround representation.
[0036] In a possible implementation, the auditory information includes a prompt sound effect corresponding to the target object.
[0037] In a possible implementation, the tactile information includes vibration of a motor.
[0038] In a possible implementation, the olfactory information includes an odor corresponding to the target object.
[0039] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and which implements the steps in the multimodal information display method of the above-mentioned extended reality device when the computer program is executed by the processor.
[0040] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the multimodal information display method of the above-mentioned extended reality device are implemented.
[0041] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.
[0042] In an embodiment of the present application, the extended reality device obtains multimodal data of the target object, analyzes the multimodal data of the target object, obtains multidimensional information of the target object corresponding to the multimodal data of the target object, and then, the extended reality device displays the multimodal information corresponding to the inference result to the user based on the inference result of the multidimensional information of the target object. The multimodal information display method of the extended reality device provided in the embodiment of the present application can display multimodal information to the user, and then, can improve the accuracy and timeliness of the user's decision-making. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0044] Figure 1 It is a flowchart of a multimodal information display method of an extended reality device provided in an embodiment of the present application;
[0045] Figure 2 1 is a schematic diagram of the optional functions of the extended reality device speech and expression analysis mode provided in an embodiment of the present application;
[0046] Figure 3 is a schematic diagram of an example of visual information provided by an embodiment of the present application;
[0047] Figure 4 is a schematic diagram of another example of visual information provided by an embodiment of the present application;
[0048] Figure 5 is a schematic diagram of the structure of an extended reality device provided in an embodiment of the present application;
[0049] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0050] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0051] The embodiments of the present application provide a multimodal information display method, device, electronic device and computer-readable storage medium for an extended reality device. Specifically, the extended reality device involved in the embodiments of the present application includes but is not limited to a head-mounted display and wearable glasses, which may be an integrated extended reality device built into a computing and processing unit, or a split extended reality device external to the computing and processing unit. Among them, the extended reality device includes but is not limited to an airborne optical display system, i.e., a head-up display system, applied to vehicles such as aircraft, automobiles, and ships, for example, an AR-HUD (Augmented reality hud-up display) mounted on an intelligent networked car, a handheld mobile device such as a mobile phone, a desktop computer, a laptop computer, a tablet, etc., and a wearable near-eye display system such as a head-mounted display, smart glasses, etc. When the extended reality device is a wearable head-mounted display or smart glasses, it may be an integrated extended reality device built into a computing and processing unit, or it may be a split extended reality device external to the computing and processing unit.
[0052] See also Figure 1 , Figure 1 : is a flow chart of a multimodal information display method of an extended reality device provided in an embodiment of the present application, and the flow chart includes the following steps:
[0053] S101, acquiring multimodal data of a target object.
[0054] In the embodiment of the present application, the extended reality device can determine the target object by observing the characters in the scene or the scene, etc., of the user wearing the extended reality device, and then obtain the multimodal data of the target object. For example, the target object can be a human subject and / or a model, etc. The embodiment of the present application is described with the target object being a human subject, but the embodiment of the present application does not limit the target object.
[0055] For example, a user wearing an extended reality device can gaze at the target object for a long time, or the user wearing the extended reality device can use a specific gesture to turn on a specific function of the extended reality device, thereby obtaining multimodal data of the target object. Figure 2 to enable the corresponding function.
[0056] Specifically, when a certain mode is turned on, the extended reality device will obtain multimodal data of the target object (for example, multimodal data of the corresponding mode function). For example, the extended reality device obtains the multimodal data of the target object by collecting the multimodal data of the target object through a sensor, or the extended reality device can obtain the multimodal data of the target object in other ways, which is not limited to the embodiments of the present application.
[0057] Optionally, the extended reality device can also collect multimodal data of the surrounding environment of the target object through sensors, so that the collected multimodal data is more comprehensive, so as to obtain more accurate analysis results.
[0058] For example, multimodal data may include voice data, image data, sound data, etc., which is not limited in the embodiments of the present application.
[0059] In the embodiment of the present application, the extended reality device may acquire the multimodal data of the target object in real time, thereby improving the timeliness of the collected multimodal data of the target object.
[0060] S102, analyzing the multimodal data of the target object to determine an inference result corresponding to the multimodal data of the target object.
[0061] In the embodiment of the present application, the extended reality device analyzing the multimodal data of the target object may be that the extended reality device processes and analyzes the multimodal data of the target object according to a preset multidimensional analysis model.
[0062] Optionally, determining an inference result corresponding to the multimodal data of the target object includes:
[0063] Determining multi-dimensional information of the target object according to the multi-modal data of the target object;
[0064] The inference result is determined based on the multi-dimensional information of the target object.
[0065] In one possible implementation, the extended reality device preprocesses the multimodal data of the target object according to the multidimensional analysis model to obtain the multidimensional information of the target object. Alternatively, the extended reality device may also determine the multidimensional information of the target object according to the multimodal data of the target object in other ways, which is not limited in the embodiments of the present application.
[0066] For example, when the multimodal data of the target object includes speech, the multidimensional information of the target object may include language, intonation, etc.
[0067] For another example, the multimodal data of the target object includes images, and the multidimensional information of the target object may include body movements, expressions, etc.
[0068] For another example, the multimodal data of the target object includes sound, and the multidimensional information of the target object may include tone, loudness, etc.
[0069] Optionally, in an embodiment of the present application, the extended reality device may analyze the multi-dimensional information of the target object through techniques such as emotion recognition algorithms and behavior analysis algorithms to obtain inference results of the multi-dimensional information of the target object, or obtain analysis results of the multi-dimensional information of the target object.
[0070] For example, the inference result may be the emotional state, intention, or behavior pattern of the target object, which is not limited in the embodiments of the present application.
[0071] S103, displaying multimodal information corresponding to the inference result to the user according to the inference result.
[0072] In the embodiments of the present application, the multimodal information may be expressed as a description of the target object's emotions, or an evaluation of the target object's behavior, or suggestions for coping strategies, etc., or it may be a vibration reminder, sound reminder, taste reminder, etc.
[0073] Exemplarily, the multimodal information may include one or more of the following: visual information, auditory information, tactile information, or olfactory information. Alternatively, the multimodal information may include information of other modes, which is not limited in the embodiments of the present application.
[0074] For example, the visual information may include information visualized as a quality 3D surround representation. Figure 3 and Figure 4 , the character trait of the target object is like water, or the character trait of the target object is emotional, etc., which can be expressed by corresponding visual styles, qualities, etc. For example, the extended reality device displays the visual information of the target object to the user in different dimensions through a text box around the target object.
[0075] Exemplarily, the auditory information may include a prompt sound effect corresponding to the target object. For example, when the target object is a person close to the user, a soothing and pleasant prompt sound is played, and when the target object is an enemy of the user, a sharp and harsh prompt sound is played, etc. This embodiment of the application is not limited to this.
[0076] Exemplarily, the tactile information may include the vibration of the motor. For example, different target objects correspond to different vibration frequencies. When the target object is a dangerous person, the vibration frequency of the motor is stronger, and when the target object is a non-dangerous person, the vibration frequency of the motor is slower.
[0077] Exemplarily, the olfactory information may include the smell corresponding to the target object. For example, different target objects correspond to different smells, and when the target object is a person that the user likes, the extended reality device releases a sweet smell, etc.
[0078] Table 1 is a multimodal information table provided in an embodiment of the present application.
[0079] Table 1
[0080]
[0081]
[0082]
[0083] Optionally, the user can customize the mode of interest. For example, the mode of interest is displayed in front of the target object, the mode of medium interest is displayed on the side of the target object, and the mode of little interest is displayed behind the target object. This embodiment of the application does not limit this.
[0084] It should be noted that the extended reality device can display one or more modes of multimodal information to the user at one time, and can call the multimodal information in combination with the actual scene, which is not limited in the embodiments of the present application.
[0085] Optionally, the multimodal information display method of the extended reality device provided in the embodiment of the present application further includes:
[0086] generating a first label of the target object;
[0087] Collecting and analyzing multimodal data of the target object multiple times according to the first label to generate multiple multi-dimensional information corresponding to the first label;
[0088] Based on the inference results of multiple multi-dimensional information, comprehensive multimodal information corresponding to multiple inference results is displayed to the user.
[0089] Exemplarily, by communicating with the target object multiple times, the background of the extended reality device will automatically integrate the multimodal data of the corresponding target object, and then the information of the corresponding analysis subject target object can be stored and formed into a first tag. When the user communicates with the target object corresponding to the first tag again, the extended reality device will display the previous integrated multimodal information in the front. For example, the multimodal information can be 3D surrounded (with depth, front and back display) around the user.
[0090] For example, when a user meets Olivia, the system will integrate all of her information based on previous communication with Olivia, display the information that the user cares about most at the front, and display the information that the user cares less about at the back, and use the corresponding texture to represent the personality of the corresponding target object. For example, if Olivia has a gentle personality like water, the image of a water bubble will be displayed. For another example, if Rebecca has a sharp personality like a hedgehog, the texture of an explosion will be displayed.
[0091] In an embodiment of the present application, the extended reality device obtains multimodal data of the target object, analyzes the multimodal data of the target object, determines the inference result corresponding to the multimodal data of the target object, and then, the extended reality device displays the multimodal information corresponding to the inference result to the user according to the inference result. The multimodal information display method of the extended reality device provided in an embodiment of the present application can display multimodal information to the user, improve the accuracy and immediacy of the user's decision-making, for example, when the user communicates with the target object, the inference result of the emotional state, honesty and other psychological characteristics of the target object can be received instantly, thereby greatly improving the accuracy and timeliness of the user's decision-making.
[0092] Furthermore, the multimodal information display method of the extended reality device provided in the embodiment of the present application can enhance the protection of user safety. For example, when the extended reality device infers that the target object may be a potential threat, it will warn the user through red warning, sound effects, vibration, etc., so as to prevent possible dangers in advance and enhance the user's personal safety.
[0093] Furthermore, the multimodal information display method of the extended reality device provided in the embodiment of the present application can optimize social and communication skills. For example, the extended reality device can not only analyze the psychological state of the target object, but also provide communication strategies based on the inference results, helping users to respond more appropriately in social occasions and improve social intelligence and communication effects.
[0094] Furthermore, the multimodal information display method of the extended reality device provided in the embodiment of the present application can be applied to a variety of occasions. For example, whether it is daily work, business negotiations, or social interaction occasions, the embodiment of the present application can provide key information analysis and response strategies to help users cope with various complex environments.
[0095] Furthermore, the multimodal information display method of the extended reality device provided in the embodiment of the present application can improve personalization and cultural adaptability. For example, the extended reality device can adjust the information analysis and display method according to different cultural backgrounds and personal preferences, ensure the accuracy and applicability of the information, and be suitable for different users around the world.
[0096] Furthermore, the multimodal information display method of the extended reality device provided in the embodiment of the present application can improve the intuitive interactive experience. For example, by using elements such as color, sound, and visual labels, the inference results can be intuitively displayed in the extended reality environment, so that users can quickly understand complex information even without professional background.
[0097] In order to better implement the multimodal information display method of the extended reality device of the present application, the present application also provides an extended reality device. For specific implementation details, please refer to the description in the method embodiment.
[0098] See also Figure 5 , Figure 5 : is a schematic diagram of the structure of an extended reality device provided in an embodiment of the present application. The device may be specifically as follows:
[0099] An acquisition module 501 is used to acquire multimodal data of a target object;
[0100] An analysis module 502 is used to analyze the multimodal data of the target object and determine an inference result corresponding to the multimodal data of the target object;
[0101] The display module 503 is used to display the multimodal information corresponding to the inference result to the user according to the inference result.
[0102] In a possible implementation, the analysis module 502 is used to determine the inference result corresponding to the multimodal data of the target object, including:
[0103] The analysis module 502 is used to determine the multi-dimensional information of the target object according to the multi-modal data of the target object;
[0104] The analysis module 502 is used to determine the inference result according to the multi-dimensional information of the target object.
[0105] In a possible implementation, the multimodal information includes one or more of the following: visual information, auditory information, tactile information, or olfactory information.
[0106] Optionally, the extended reality device provided in the embodiment of the present application further includes:
[0107] A generating module, used for generating a first label of the target object;
[0108] The analysis module 502 is further configured to collect and analyze the multimodal data of the target object multiple times according to the first tag to generate a plurality of multi-dimensional information corresponding to the first tag;
[0109] The display module 503 is further used to display the comprehensive multimodal information corresponding to the multiple inference results to the user according to the inference results of the multiple multi-dimensional information.
[0110] In a possible implementation, the visual information includes information visualized as a quality 3D surround representation.
[0111] In a possible implementation, the auditory information includes a prompt sound effect corresponding to the target object.
[0112] In a possible implementation, the tactile information includes vibration of a motor.
[0113] In a possible implementation, the olfactory information includes an odor corresponding to the target object.
[0114] In the embodiment of the present application, the acquisition module 501 acquires the multimodal data of the target object, the analysis module 502 analyzes the multimodal data of the target object, determines the inference result corresponding to the multimodal data of the target object, and then, the display module 503 displays the multimodal information corresponding to the inference result to the user according to the inference result. The extended reality device provided in the embodiment of the present application can display multimodal information to the user, thereby improving the accuracy and timeliness of the user's decision-making.
[0115] In addition, the present application also provides an electronic device, such as Figure 6 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:
[0116] The electronic device may include components such as a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a power supply 603, and an input unit 604. Those skilled in the art will appreciate that Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0117] The processor 601 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601.
[0118] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.
[0119] The electronic device also includes a power supply 603 for supplying power to various components. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so as to manage charging, discharging, power consumption management and other functions through the power management system. The power supply 603 can also include one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, power status indicators and other arbitrary components.
[0120] The electronic device may further include an input unit 604, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0121] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 601 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 602 according to the following instructions, and the processor 601 will run the application program stored in the memory 602, thereby implementing the steps in any multimodal information display method of an extended reality device provided in the embodiments of the present application.
[0122] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0123] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0124] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in the multimodal information display method of any extended reality device provided in the present application.
[0125] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0126] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0127] Since the instructions stored in the computer-readable storage medium can execute the steps in the multimodal information display method of any extended reality device provided in the present application, the beneficial effects that can be achieved by the multimodal information display method of any extended reality device provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0128] The above is a detailed introduction to the multimodal information display method, device, electronic device and computer-readable storage medium of an extended reality device provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and scope of application. In summary, the content of this specification should not be understood as a limitation on the present invention.
[0129] It should be noted that in the specific implementation of this application, data related to voice information, eye annotation information, historical behavior information, gesture operations, etc. are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
Claims
1. A multimodal information display method for an extended reality device, characterized in that: Applied to an extended reality device, the method comprises: Obtain multimodal data of the target object; Analyzing the multimodal data of the target object to determine an inference result corresponding to the multimodal data of the target object; The multimodal information corresponding to the inference result is displayed to the user according to the inference result.
2. The method according to claim 1, characterized in that The determining of the inference result corresponding to the multimodal data of the target object includes: Determining multi-dimensional information of the target object according to the multi-modal data of the target object; The inference result is determined according to the multi-dimensional information of the target object.
3. The method according to claim 2, characterized in that The multimodal information includes one or more of the following: visual information, auditory information, tactile information, or olfactory information.
4. The method according to claim 2, characterized in that: Before acquiring the multimodal data of the target object, the method further includes: generating a first label of the target object; Collecting and analyzing the multimodal data of the target object multiple times according to the first tag to generate a plurality of multi-dimensional information corresponding to the first tag; Based on the inference results of the multiple multi-dimensional information, comprehensive multimodal information corresponding to the multiple inference results is displayed to the user.
5. The method according to claim 3, characterized in that: The visual information includes information visualized as a quality 3D surround representation.
6. The method according to claim 3, characterized in that The auditory information includes a prompt sound effect corresponding to the target object.
7. The method according to claim 3, characterized in that The tactile information includes vibration of a motor.
8. The method according to claim 3, characterized in that The olfactory information includes the smell corresponding to the target object.
9. An extended reality device, characterized in that: The device comprises: An acquisition module, used to acquire multimodal data of a target object; An analysis module, configured to analyze the multimodal data of the target object and determine an inference result corresponding to the multimodal data of the target object; A display module is used to display the multimodal information corresponding to the inference result to the user according to the inference result.
10. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multimodal information display method of the extended reality device as described in any one of claims 1 to 8 is implemented.
11. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the multimodal information display method of the extended reality device according to any one of claims 1 to 8 is implemented.