Interaction method and device of augmented reality equipment

Abstract

The invention discloses an interaction method and device of augmented reality equipment, and belongs to the field of augmented reality. The method comprises the following steps: determining a reference scene in which the augmented reality equipment is located based on reference information, wherein the reference information comprises but is not limited to at least one of position information, environment information, equipment connection signals or time information; the augmented reality device is controlled to be automatically connected with a target terminal device by using the reference scene, the target terminal device is determined based on the reference scene and a candidate terminal device, and the candidate terminal device is a terminal device capable of interacting with the augmented reality device; determining an interaction scene mode based on the reference scene and the target terminal device, wherein the interaction scene mode comprises any one of a driving scene mode, a passenger scene mode, a leisure scene mode, a motion scene mode or a riding scene mode; and determining interaction information displayed by the augmented reality device based on the interaction scene mode. The interaction efficiency and the information display efficiency of the augmented reality equipment and the terminal equipment are improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of extended reality, and in particular, to an interaction method and device of an extended reality device. BACKGROUND

[0002] With the rapid development of extended reality (XR) technology, the XR technology is increasingly widely applied in the fields of consumer electronics, intelligent transportation, etc. The XR technology includes augmented reality (AR), virtual reality (VR), and mixed reality (MR). For example, AR glasses, as a wearable augmented reality device, are increasingly widely applied in different scenarios due to their characteristics of being able to display virtual information and real environment in fusion. SUMMARY

[0003] Embodiments of the present application provide an interaction method and device of an extended reality device, which can improve the interaction efficiency and information display efficiency of the extended reality device and a terminal device. The technical solutions are as follows:

[0004] In one aspect, the present application provides an interaction method of an extended reality device, which comprises:

[0005] determining a reference scene in which the extended reality device is located based on reference information, the reference information comprising at least one of position information, environment information, device connection signal, or time information;

[0006] controlling the extended reality device to automatically connect with a target terminal device by using the reference scene, the target terminal device being determined based on the reference scene and a candidate terminal device, the candidate terminal device being a terminal device capable of interacting with the extended reality device;

[0007] determining an interaction scene mode based on the reference scene and the target terminal device, the interaction scene mode comprising any one of a driving scene mode, a passenger scene mode, a leisure scene mode, a sports scene mode, or a cycling scene mode;

[0008] determining interaction information displayed by the extended reality device based on the interaction scene mode.

[0009] In another aspect, the present application provides an interaction device of an extended reality device, which comprises:

[0010] a determination module configured to determine a reference scene in which the extended reality device is located based on reference information, the reference information comprising at least one of position information, environment information, device connection signal, or time information;

[0011] a control module, configured to control the extended reality device to automatically connect with a target terminal device based on the reference scene, the target terminal device being determined based on the reference scene and a candidate terminal device, the candidate terminal device being a terminal device capable of interacting with the extended reality device;

[0012] The determination module is further configured to determine an interaction scene mode based on the reference scene and the target terminal device, the interaction scene mode comprising any one of a driving scene mode, a passenger scene mode, a leisure scene mode, a sports scene mode, or a cycling scene mode.

[0013] The determination module is further configured to determine interaction information displayed by the extended reality device based on the interaction scene mode.

[0014] In a possible implementation, the determination module is further configured to determine a candidate terminal device corresponding to the reference scene in a case where the extended reality device in the reference scene meets a connection condition; in a case where the number of the candidate terminal devices is one, determine the candidate terminal device as the target terminal device; or in a case where the number of the candidate terminal devices is a plurality, determine connection priorities of the candidate terminal devices with the extended reality device, and determine the target terminal device among the candidate terminal devices based on the connection priorities.

[0015] In a possible implementation, the determination module is configured to obtain scene feature information corresponding to the reference scene, the scene feature information being used to distinguish the interaction scene mode in the reference scene; match the scene feature information with a scene mode mapping strategy corresponding to the target terminal device to obtain a matching result, the scene mode mapping strategy representing a corresponding relationship between the scene feature information and the interaction scene mode; and determine the interaction scene mode based on the matching result.

[0016] In a possible implementation, the extended reality device is associated with at least one virtual object, and the control module is further configured to receive a wake-up operation for the virtual object, invoke a virtual object corresponding to the interaction scene mode based on the extended reality device; and in a case where an interaction operation based on the virtual object is received, control the virtual object to perform a target action based on the interaction operation, the target action being adapted to the interaction operation and the reference scene.

[0017] In a possible implementation, the determining module is configured to, when the interaction scenario mode is the driving scenario mode, determine, based on the driving parameter of the vehicle and the environment information, the interaction information displayed by the extended reality device, the interaction information being information related to a driving process of the vehicle, the interaction information including at least one of vehicle parameter information, driving assistance information, or the environment information, the driving assistance information including at least one of lane departure warning information, collision warning information, lane guidance information, or navigation information, the vehicle parameter information being directly obtained based on the driving parameter, and the driving assistance information being indirectly obtained based on the driving parameter.

[0018] In a possible implementation, the determining module is configured to, when the interaction scenario mode is the passenger scenario mode, determine, based on at least one of the environment information or selection operation information of a user, the interaction information displayed by the extended reality device, the interaction information including at least one of life service information or entertainment information.

[0019] In a possible implementation, the determining module is configured to, when the interaction scenario mode is the leisure scenario mode, determine, based on at least one of real-time data of the target terminal device, the environment information, or selection operation information of a user, the interaction information displayed by the extended reality device, the interaction information including at least one of basic information, social communication information, life service information, media entertainment information, or navigation information, the basic information including at least one of time information, location information, or weather information.

[0020] In a possible implementation, the determining module is configured to, when the interaction scenario mode is the sports scenario mode, determine, based on at least one of real-time data of the target terminal device and selection operation information of a user, the interaction information of the extended reality device, the interaction information including at least one of sports type identification or real-time sports data.

[0021] In a possible implementation, the determining module is configured to, when the interaction scenario mode is the riding scenario mode, determine, based on at least one of real-time data of the target terminal device, state information of a riding vehicle, or the environment information, the interaction information displayed by the extended reality device, the interaction information including at least one of navigation information, life service information, or safety warning information.

[0022] In another aspect, the embodiments of the present application provide a computer device, comprising a processor and a memory, the memory storing at least one program code, the at least one program code being loaded and executed by the processor to enable the computer device to implement the interaction method of the extended reality device as described above.

[0023] In another aspect, a computer readable storage medium is also provided, the computer readable storage medium storing at least one program code, the at least one program code being loaded and executed by a processor to enable a computer to implement the interaction method of the extended reality device as described above.

[0024] In another aspect, a computer program or computer program product is also provided, the computer program or computer program product storing at least one computer instruction, the at least one computer instruction being loaded and executed by a processor to enable a computer to implement the interaction method of the extended reality device as described above.

[0025] The technical solutions provided by the embodiments of the present application at least bring the following beneficial effects:

[0026] The present application identifies the reference scene in which the extended reality device is located by referring to information, automatically connects the target terminal device for interaction with the extended reality device, and does not need the user to manually switch the target terminal device for interaction with the extended reality device, thereby improving the interaction efficiency of the extended reality device and the terminal device. Furthermore, the adaptive interaction scene mode is determined based on the reference scene and the target terminal device, and the user does not need to manually switch the interaction scene mode, thereby realizing seamless fusion and cooperation of multi-device information and improving the convenience of cross-scene use of the extended reality device. The interaction information of the extended reality device is determined based on the interaction scene mode, the interaction information is matched with the interaction scene mode, invalid information interference is avoided, and the information display efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0028] Figure 1 is a schematic diagram of an implementation environment provided by the embodiments of the present application;

[0029] Figure 2 is a flowchart of an interaction method of an extended reality device provided by the embodiments of the present application;

[0030] Figure 3is a schematic diagram of an interaction device of an extended reality device provided by an embodiment of the present application.

[0031] Figure 4 is a structural schematic diagram of a server provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] For the purpose of making the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0033] It should be noted that the terms "first", "second", and the like in the present application are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0034] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application. As shown in Figure 1 the implementation environment can include an extended reality device 101, a terminal device 102 and a server 103, which realize data interaction through a wireless communication network, including but not limited to Bluetooth, wireless fidelity (Wi-Fi), cellular network and near field communication technology.

[0035] The extended reality device 101 is a wearable device with virtual-real fusion display capability, including but not limited to an augmented reality device, a virtual reality device and a mixed reality device. Taking the augmented reality device as an example, the augmented reality device includes AR glasses and AR headsets. The extended reality device 101 is a lightweight wearable device, including a display module 101-1, an interaction module 101-2, an audio module 101-3 and a communication module 101-4. Among them, the extended reality device 101 itself does not undertake complex data processing or undertakes a small amount of data processing.

[0036] The display module 101-1 includes lenses with customizable far and near vision power and an imaging device for receiving data from the terminal device 102 and the server 103 and projecting the data to the user's field of view to achieve virtual-real fusion display. The interaction module 101-2 integrates a camera (such as a Time of Flight (TOF) sensor), a microphone, and physical buttons to support basic operations such as three-dimensional environment perception (such as gesture positioning), gesture interaction (such as air-clicking and sliding), and voice commands (such as waking up virtual objects). The camera can replace a dashcam to record real-time environmental information while driving, cycling, walking, or exercising. The audio module 101-3 includes a miniature earphone for real-time information broadcasting (such as navigation prompts and message reminders). The communication module 101-4 includes communication components (such as Bluetooth and Wi-Fi modules) that establish communication connections with the terminal device 102 and transmit data with the support of the built-in rechargeable battery and the Printed Circuit Board Assembly (PCBA).

[0037] The terminal device 102 is a terminal device capable of interacting with the extended reality device 101, including but not limited to mobile terminals (such as smartphones, smartwatches, fitness bands, tablets, etc.), vehicle-mounted terminals (such as car infotainment systems), and other smart devices (such as computers and smart home control terminals). For example, the terminal device 102 is a vehicle-mounted terminal, a smartphone, and a computer, and each of the vehicle-mounted terminal, the smartphone, and the computer has a corresponding version of an application (APP) installed.

[0038] The server 103 is a server, or a server cluster composed of multiple servers, or any one of a cloud computing platform and a virtualization center, and the embodiments of the present application do not limit this. The server 103 has data receiving, processing, and sending functions. Of course, the server 103 can also have other functions, which are not limited by the embodiments of the present application.

[0039] Based on the above Figure 1 The embodiments of the present application provide an interaction method of an extended reality device based on the implementation environment shown in the figure. As Figure 2 The method can be cooperatively executed by the extended reality device 101, the terminal device 102, and the server 103, and can include steps 201 to 204.

[0040] In step 201, the reference scene in which the extended reality device is located is determined based on reference information, which includes at least one of position information, environmental information, device connection signals, or time information.

[0041] In the example embodiments of the present application, the reference information refers to data information required by the extended reality device to determine the reference scene. Take the augmented reality device as an example for description. The reference information includes but is not limited to the position information of the augmented reality device, the environmental information in which the augmented reality device is located, the device connection signal, and the time information.

[0042] Among them, the augmented reality device is built-in with a global positioning system (GPS) module or a Beidou positioning module, which can directly collect the longitude, latitude and altitude of the augmented reality device to obtain the position information.

[0043] The augmented reality device is built-in with a camera (such as a ToF depth sensor), a microphone and a light sensor. The camera is used to shoot the surrounding environment image (such as vehicle interior, sports equipment, etc.), the microphone is used to collect the environmental sound (such as engine noise, traffic siren, etc.), and the light sensor is used to detect the environmental light intensity (such as indoor light intensity, outdoor light intensity, etc.). The data of the camera, microphone and light sensor are integrated to determine the environmental information.

[0044] The augmented reality device is built-in with a communication module, which is used to scan the surrounding wireless signal. The wireless signal includes Bluetooth signal (such as vehicle Bluetooth signal, mobile phone Bluetooth signal, computer Bluetooth signal, sports watch Bluetooth signal, etc.), Wi-Fi signal (such as vehicle Wi-Fi signal, home router Wi-Fi signal, office LAN Wi-Fi signal, etc.), and the device connection signal is obtained by identifying the signal characteristics of the connectable terminal device.

[0045] The augmented reality device is built-in with a clock module, which can directly obtain the time information by using the clock module; or the augmented reality device receives the system time synchronized by the terminal device to obtain the time information.

[0046] In step 202, the extended reality device is controlled to automatically connect with the target terminal device based on the reference scene. The target terminal device is determined based on the reference scene and the candidate terminal device, and the candidate terminal device is a terminal device capable of interacting with the extended reality device.

[0047] In the example embodiments of the present application, the process of determining the target terminal device includes: determining the candidate terminal device corresponding to the reference scene when the extended reality device in the reference scene meets the connection condition; determining the candidate terminal device as the target terminal device when the number of candidate terminal devices is one; or determining the connection priority of each candidate terminal device with the extended reality device when the number of candidate terminal devices is multiple, and determining the target terminal device among the candidate terminal devices based on the connection priority.

[0048] Exemplarily, it is first judged whether the extended reality device meets the connection condition based on a reference scene in which the extended reality device is currently located, wherein the connection condition comprises that the extended reality device opens a connection permission and has a terminal device capable of being connected with the extended reality device. In the case that the extended reality device meets the connection condition, the terminal device capable of being connected with the extended reality device is determined as a candidate terminal device.

[0049] When the candidate terminal device is only one, it is not necessary to additionally judge, and the candidate terminal device is directly determined as a target terminal device. When the number of the candidate terminal devices is multiple, a connection priority is given to each candidate device according to a preset rule, and the device with the highest priority is selected as the target terminal device. After the target terminal device is determined, the extended reality device is controlled to automatically connect with the target terminal device.

[0050] The process of giving the connection priority to each candidate device according to the preset rule comprises: acquiring a connection influencing factor of the terminal device and a weight corresponding to the connection influencing factor, wherein the connection influencing factor comprises at least one of a functional adaptation degree of the terminal device to a current scene (reference scene), a historical connection frequency or a data transmission efficiency; determining a score of the candidate terminal device based on the connection influencing factor and the weight corresponding to the connection influencing factor, and determining the connection priority of the candidate terminal device by using the score of the candidate terminal device.

[0051] The functional adaptation degree of the terminal device to the current scene (reference scene) refers to the matching degree of the core function provided by the terminal device to the core demand of the current reference scene. For example, the reference scene is a vehicle scene, and the car machine can provide functions such as navigation and vehicle state monitoring (fuel quantity, tire pressure), and the functional adaptation degree of the car machine terminal device is high, and the functional adaptation degrees of the rest terminal devices are low. For another example, the current is a sports scene, and the core demand is sports data monitoring and guidance, and the smart watch can transmit data such as heart rate and sports duration in real time, and the functional adaptation degree of the smart watch is high, and the functional adaptation degrees of the rest terminal devices are low. The higher the functional adaptation degree of the terminal device to the current scene (reference scene) is, the greater the corresponding weight is.

[0052] The historical connection frequency refers to the number, frequency or proportion of times that the candidate terminal device successfully establishes a connection with the extended reality device (such as AR glasses) in the past similar scenes. The more the historical connection times are, the higher the frequency is, which indicates that the user is more inclined to use the terminal device in the current scene, and the priority is higher. For example, in the case that the reference scene is a vehicle scene, AR glasses are used to connect the car machine for 8 times out of 10 times in the past, and a mobile phone is connected for 2 times, and thus the historical connection frequency (80%) of the car machine is higher than that (20%) of the mobile phone. The greater the historical connection frequency is, the greater the corresponding weight is.

[0053] The data transmission efficiency refers to the performance of transmitting data (such as images, audio, sensor data, etc.) between the candidate terminal device and the extended reality device. The smaller the delay of data transmission, the greater the amount of data transmitted per unit time, and the higher the data transmission efficiency. The higher the data transmission efficiency, the greater the corresponding weight.

[0054] The weighted average of the connection influencing factor and the weight corresponding to the connection influencing factor is taken as the score of the candidate terminal device, and the connection priority of the candidate terminal device is determined according to the score from high to low. It should be noted that the way of determining the connection priority of the candidate terminal device is exemplary, and other ways can also be used to determine the connection priority of the candidate terminal device, which is not limited in the present application.

[0055] Taking the vehicle-mounted scene as the reference scene and the AR glasses as the extended reality device as an example for illustration. In the vehicle-mounted scene, the AR glasses judge that they meet the connection condition (Bluetooth is turned on), and scan the surrounding interactive terminal devices including the car terminal device and the mobile phone terminal device. If the mobile phone terminal device does not turn on the Bluetooth at this time, only the car terminal device is in a connectable state, then the car terminal device is determined as the target terminal device. If the mobile phone terminal device and the car terminal device both turn on the Bluetooth and are connectable, then the car terminal device and the mobile phone terminal device are candidate terminal devices. For example, in the vehicle-mounted scene, the connection priority of the car terminal device is higher than that of the mobile phone terminal device, and the car terminal device is determined as the target terminal device.

[0056] The embodiments of the present application determine the candidate terminal device by using the reference scene in which the extended reality device is located. When the connection condition is met, the interactive device can be quickly located, the time for the user to manually search and select the interactive terminal device is reduced, and the operation convenience is improved. Moreover, when there are multiple candidate terminal devices, the target terminal device is determined by using the connection priority of the candidate terminal device, so that the determined target terminal device is more in line with the actual needs and use habits of the user in the current scene, and the interaction efficiency and user experience of the extended reality device and the target terminal device are improved.

[0057] Optionally, in the case where there are multiple candidate terminal devices, the prompt information is displayed in at least one of the candidate terminal device or the extended reality device, the prompt information is used to prompt the user to select the target terminal device from the multiple candidate terminal devices; and the determined candidate terminal device is determined as the target terminal device in response to the determination operation of the prompt information.

[0058] The following explanation uses AR glasses as an example of an augmented reality device. For instance, after the AR glasses scan for multiple connectable candidate terminal devices (e.g., smartwatches, tablets, smartphones), in one embodiment, the AR glasses' lens display shows a prompt: "Three connectable devices detected: smartwatch, tablet, and smartphone. Please select the device to connect to." The user selects one of the three candidate devices using the AR glasses' voice control or gesture recognition function, and the selected device is then designated as the target terminal device.

[0059] In another embodiment, the smartwatch, tablet, and smartphone each display a prompt message, such as "AR glasses request connection, confirm connection?". After any prompt message is confirmed, the terminal that confirmed the prompt message is identified as the target terminal device.

[0060] In this embodiment of the application, when there are multiple candidate terminal devices, prompting information is displayed on at least one of the candidate terminal devices or extended reality devices, allowing users to choose independently from the prompting information based on their own needs. The candidate terminal device determined based on the prompting information is used as the target terminal device, which improves the accuracy and personalization of determining the target terminal device, enhances the flexibility of interaction between the extended reality device and different terminal devices, and thus improves the user experience.

[0061] In step 203, an interaction scenario mode is determined based on the reference scenario and the target terminal device. The interaction scenario mode includes any one of the following: driving scenario mode, passenger scenario mode, leisure scenario mode, sports scenario mode, or cycling scenario mode.

[0062] In the exemplary embodiment of this application, the process of determining the interaction scene mode based on the reference scene and the target terminal device includes steps 2031 to 2033 (not shown in the figure). The interaction scene mode is specifically described in step 204 and will not be repeated here.

[0063] In step 2031, scene feature information corresponding to the reference scene is obtained. The scene feature information is used to distinguish the interaction scene mode under the reference scene.

[0064] Exemplarily, the scene feature information includes but is not limited to position information, user behavior information, and target terminal device state information. For example, the position information is used to determine whether the extended reality device is located at a main driver position inside a vehicle, a co-driver position inside a vehicle, a gym position, and the like. The user behavior information is used to judge a real-time action or behavior state of the user, for example, whether the hand holds a steering wheel, whether a sports bracelet is worn and the arm is in a swing state, whether an entertainment device is being operated, and the like. The target terminal device state information is used to reflect a running mode or function output of the target terminal device, for example, whether the car machine is in a navigation mode, whether the smart treadmill is in a running state, whether the smart television is playing video content, and the like.

[0065] In step 2032, the scene feature information is matched with the scene mode mapping strategy corresponding to the target terminal device to obtain a matching result, and the scene mode mapping strategy represents a corresponding relationship between the scene feature information and the interaction scene mode.

[0066] Exemplarily, the scene mode mapping strategy is a set of rules constructed in advance, which clearly defines the corresponding relationship between different scene feature information combinations and each interaction scene mode. The process of matching the scene feature information with the scene mode mapping strategy corresponding to the target terminal device to obtain a matching result includes steps A1 to A4 (not shown in the figure).

[0067] In step A1, the scene mode mapping strategy of the target terminal device is divided into multiple independent feature items.

[0068] Taking the target terminal device as a car machine terminal device as an example for description. The scene mode mapping strategy in the car machine terminal device is "main driver position + holding steering wheel + car machine navigation mode" corresponding to "driving scene mode", which can be disassembled into 3 feature items: position feature (main driver position), user behavior feature (holding steering wheel), and device state feature (car machine navigation mode).

[0069] In step A2, the scene feature information is compared with each feature item to determine a matching score of the scene feature information and each feature item.

[0070] For example, the matching score ranges from 0 to 10 points, 10 points for complete matching, 5 points for partial matching, and 0 points for no matching. If the position in the scene feature information is "co-driver position", it does not match the "main driver position" feature item in the scene mode mapping strategy, and the feature item gets 0 points; if the user behavior is "holding steering wheel", it completely matches the corresponding feature item in the scene mode mapping strategy, and gets 10 points; if the car machine is in "navigation mode", it completely matches the corresponding feature item in the scene mode mapping strategy, and gets 10 points.

[0071] In step A3, based on the matching scores of the feature items and the weights corresponding to the feature items, a comprehensive matching score of the scene feature information is obtained.

[0072] The weights corresponding to the feature items can be assigned based on the importance of the feature items in the scene mode mapping strategy. The comprehensive matching score, i.e., the matching result, is obtained by multiplying the scores of each feature item by the corresponding weight and summing them up.

[0073] In step 2033, the interactive scene mode is determined based on the matching result.

[0074] Exemplarily, the comprehensive matching score in the matching result is compared with a matching score threshold. If the comprehensive matching score is greater than or equal to the matching score threshold, it is confirmed that the scene feature information matches the interactive scene mode. If the comprehensive matching score is less than the matching score threshold, it is confirmed that the scene feature information does not match the interactive scene mode, thereby determining the interactive scene mode corresponding to the scene feature information.

[0075] The embodiments of the present application are based on the scene feature information of the reference scene, and utilize the scene mode mapping strategy corresponding to the target terminal device for matching. The most suitable interactive scene mode is determined according to different scenes and target terminal devices. The complexity and diversity of actual scenes, as well as the individual differences of target terminal devices, are fully considered, thereby improving the intelligence and interaction efficiency of the process of displaying interactive information by the extended reality device.

[0076] In step 204, the interactive information displayed by the extended reality device is determined based on the interactive scene mode.

[0077] In the exemplary embodiments of the present application, determining the interactive information displayed by the extended reality device based on the interactive scene mode includes but is not limited to cases one to six.

[0078] Case one, in the case where the interactive scene mode is a driving scene mode, the interactive information displayed by the extended reality device is determined based on the driving parameters and environmental information of the vehicle. The interactive information is information related to the driving process of the vehicle. The interactive information includes at least one of vehicle parameter information, driving assistance information, or environmental information. The driving assistance information includes at least one of lane departure warning information, collision warning information, lane guidance information, or navigation information. The vehicle parameter information is directly obtained based on the driving parameters, and the driving assistance information is indirectly obtained based on the driving parameters.

[0079] Exemplarily, the driving parameters of the vehicle represent data characterizing the running state of the vehicle. The driving parameters of the vehicle include but are not limited to basic running parameters (such as vehicle speed, gear position, engine speed, remaining fuel quantity, remaining electric quantity, etc.), safety state parameters (such as tire pressure, brake state, low beam / high beam light opening prompt, and turn signal light non-return reminder), and other key parameters (such as water temperature and cruising range).

[0080] The environmental information refers to external environment data related to vehicle driving, and is used to help the driver to predict the road condition change and adjust the driving strategy. The environmental information includes but is not limited to weather, environmental location (such as the intersection of XX Road and XX Street), road marking (such as lane boundary), surrounding vehicle distance, light intensity, traffic signal state, speed limit information, and obstacle information around the vehicle.

[0081] The interaction information is the information displayed by the extended reality device in the driving scene mode. The interaction information is determined from the driving parameters and the environmental information. The interaction information is displayed in a manner that does not interfere with the driving line of sight, such as highlighting key information with a lightweight interface and integrating with the real scene.

[0082] Taking AR glasses as an example, after the user wears the AR glasses and starts the vehicle, the AR glasses are matched with the vehicle machine through Bluetooth or Wi-Fi. After the matching is successful, the connection state is displayed on the vehicle machine center screen and the AR glasses. The display interface of the vehicle machine center screen or the AR glasses displays a candidate information list, which includes but is not limited to vehicle parameter information, driving assistance information, and environmental information. The driving assistance information includes but is not limited to lane departure warning information, collision warning information, lane guidance information, and navigation information. The vehicle parameter information is directly obtained based on the driving parameters (such as speed, gear, tire pressure, etc.), and the driving assistance information is indirectly obtained based on the driving parameters. The user selects at least one item in the candidate information list as the interaction information. In the case where the user does not select, the default selection information in the driving scene mode can be used as the interaction information.

[0083] In the driving scene mode, the embodiment of the present application determines the interaction information displayed by the extended reality device by obtaining the vehicle driving parameters and the environmental information, and provides a self-defined display content option, which can provide personalized information closely related to driving for the driver, helps the driver to understand the vehicle state and the surrounding environment in a timely and comprehensive manner, makes correct driving decisions in advance, effectively prevents dangerous situations, and significantly improves driving safety. Moreover, the self-defined function of the interaction information meets the diversified needs of different drivers, avoids the interference of irrelevant information, enables the driver to focus more on the driving process, and improves the convenience and safety of driving.

[0084] In case two, in the case where the interaction scene mode is the passenger scene mode, the interaction information displayed by the extended reality device is determined based on at least one of the environmental information or the selection operation information of the user, and the interaction information includes at least one of life service information or entertainment information.

[0085] Exemplarily, the environmental information refers to external environment data related to the trip of the occupant, and is used to assist the occupant in planning a trip or obtaining a scenario-based service. The environmental information includes, but is not limited to, merchant information (such as a restaurant, a shopping mall, and a gas station) around the current location, real-time traffic, facilities (such as a parking lot and a restroom) around the destination, and weather information. The selection operation information of the user refers to a function selection initiated by the occupant through a voice instruction, a gesture, or a touch, such as a movie playing instruction, a nearby coffee shop query instruction, and a vehicle-mounted game opening instruction.

[0086] The interaction information is determined from the environmental information or the selection operation information of the user, and is information displayed by the extended reality device in the occupant scenario mode. The interaction information is presented in a manner consistent with the usage habits of the occupant, such as an immersive interface, support for multi-task switching, and the ability to adjust the display size and position according to the user's needs.

[0087] Taking AR glasses as an example, the AR glasses automatically recognize the position information (a non-driver position) and switch to the occupant scenario mode when the user wears the AR glasses and enters a passenger seat. The AR glasses obtain the surrounding environmental information through networking of the vehicle machine. At this time, the display interface of the AR glasses pops up a candidate information list, which includes life service information (such as food recommendations along the way and discount activities in a destination shopping mall) and entertainment information (such as a vehicle-mounted video library, an online music list, and an interactive game). The user selects at least one item in the candidate information list as the interaction information. In the case where the user does not make a selection, the default selection information of the occupant scenario mode can be used as the interaction information.

[0088] Optionally, to ensure driving safety and prevent the user from using the occupant scenario mode during driving, driving-related information cannot be determined as the interaction information in the occupant scenario mode.

[0089] In the occupant scenario mode, the embodiments of the present application determine the interaction information through environmental information or user selection operation, focus on life service and entertainment needs, allow the occupant to conveniently obtain scenario-based services and enjoy rich entertainment content through personalized selection, avoid the interference of information unrelated to driving on the occupant, and meet the diversified needs of different occupants, thereby significantly improving the comfort and experience of the ride.

[0090] In case three, in the case where the interaction scenario mode is a leisure scenario mode, the interaction information displayed by the extended reality device is determined based on at least one of real-time data of the target terminal device, environmental information, or selection operation information of the user. The interaction information includes at least one of basic information, social communication information, life service information, media entertainment information, or navigation information. The basic information includes at least one of time information, location information, or weather information.

[0091] Exemplarily, the real-time data of the target terminal device refers to a device running state or content data related to a leisure scene, and the real-time data of the target terminal device includes but is not limited to a playing progress of a smart television, a working state of a smart home device, a music list update of a smart speaker, express delivery information, and the like.

[0092] The environmental information refers to external environment data related to a leisure activity, such as a current place, indoor and outdoor light intensity, and surrounding leisure facilities. The selection operation information of the user refers to a function instruction initiated by the user through voice, gesture, or touch, such as viewing unread messages, playing light music, and searching for a nearby dessert shop.

[0093] The interaction information is information displayed by the extended reality device in the leisure scene mode, which is determined from the real-time data of the target terminal device, the environmental information, or the selection operation information of the user. The interaction information supports split-screen display or one-key switching, for example, social communication information is displayed in the form of a floating window, and media entertainment information is presented in an immersive full-screen mode, and the information layout can be adjusted according to user needs.

[0094] Taking an AR glasses as an example, the user wears the AR glasses in a living room at home (the environmental information is identified as a leisure scene), and the AR glasses automatically connect to a smart television, a mobile phone, and other target terminal devices and switch to a leisure scene mode. At this time, a display interface of the AR glasses pops up a candidate information list, which includes basic information (such as time information, location information, and weather information), social communication information, life service information (such as an estimated 20-minute delivery of takeout), media entertainment information (such as a recommended episode XX of a smart television), and navigation information (such as a walking route to a coffee shop). The user selects at least one item in the candidate information list as the interaction information. In the case where the user does not select, the default selection information in the leisure scene mode can be used as the interaction information.

[0095] In the leisure scene mode, the embodiment of the application determines the interaction information through the real-time data of the target terminal device, the environmental information, and the selection operation of the user, covers multiple needs such as life, social interaction, and entertainment, allows the user to conveniently master various information (such as time and message reminders), and enjoy leisure services (such as playing music, videos, and news videos) through personalized selection, avoids cumbersome operations for information acquisition, improves fragmented time, supports flexible switching of multiple types of information, and improves the use convenience and user experience in the leisure scene.

[0096] In case four, in the case where the interaction scene mode is a sports scene mode, the interaction information of the extended reality device is determined based on at least one of the real-time data of the target terminal device and the selection operation information of the user, and the interaction information includes at least one of a sports type identifier or real-time sports data.

[0097] Exemplarily, the real-time data of the target terminal device refers to device monitoring data related to the exercise process, including but not limited to heart rate of a smart bracelet (watch), running parameter of a smart treadmill, calorie consumption of a sports watch, etc. The selection operation information of the user refers to a function instruction initiated by the user through voice, gesture or device key, and the function instruction is used to interact with the real-time data of the target terminal device.

[0098] The interaction information is information displayed by the extended reality device in the exercise scene mode, and is determined from the real-time data of the target terminal device or the selection operation information of the user. The interaction information includes the exercise type identifier of the current exercise and the real-time exercise data.

[0099] Taking the AR glasses as an example, the user wears the AR glasses into a gym (identified as an exercise scene through environment information), and the AR glasses automatically connect the smart treadmill and the sports bracelet and switch to the exercise scene mode. At this time, the display interface of the AR glasses pops up a candidate information list, including the exercise type identifier (such as treadmill jogging, spinning, badminton, table tennis, basketball, football, swimming, skiing, etc.) and the real-time exercise data (such as heart rate 110 beats per minute, pace 6 minutes per kilometer). The user selects at least one item in the candidate information list as the interaction information. In the case where the user does not select, the default selection information of the exercise scene mode can be used as the interaction information.

[0100] In the exercise scene mode, the embodiments of the present application determine the interaction information through the real-time data of the target terminal device or the selection operation of the user, focus on the exercise type and real-time data monitoring, not only let the user clearly master the exercise state (such as heart rate, pace), but also can accurately obtain the required information through active selection, avoid the interference of redundant content to the exercise process, and the intuitive presentation mode helps the user to adjust the exercise rhythm in real time, significantly improves the use efficiency and user experience in the exercise scene.

[0101] In the case five, in the case where the interaction scene mode is the cycling scene mode, the interaction information displayed by the extended reality device is determined based on at least one of the real-time data of the target terminal device, the state information of the cycling vehicle or the environment information, and the interaction information includes at least one of the navigation information, the life service information or the safety warning information.

[0102] Exemplarily, the real-time data of the target terminal device refers to device monitoring or recording data related to the cycling process, including real-time speed of a cycling meter, cycling duration of a sports watch, cycling trajectory record of a mobile phone, etc. The state information of the cycling vehicle refers to data reflecting the condition of the cycling tool itself, such as power of an electric vehicle, tire pressure, etc. The environment information refers to external data related to the cycling environment, including real-time road conditions, weather conditions, surrounding obstacles, road slope, etc.

[0103] The interaction information is information displayed by the extended reality device in the riding scene mode, which is determined from real-time data of the target terminal device, state information of the riding vehicle, or environmental information. Among them, the interaction information is presented in a way that does not block the riding field of view, for example, navigation information is superimposed on the actual road surface in the form of an AR arrow, safety warning information is prompted by a prominent dynamic icon, and the layout of the interaction information can be automatically adjusted according to the riding posture (such as moving the information up when looking down at the road).

[0104] Taking AR glasses as an example, the user wears AR glasses and rides an electric vehicle (switches to the riding scene mode through sensor recognition of the riding posture), and the AR glasses automatically connect the riding watch and the mobile phone and obtain vehicle state information. At this time, the display interface of the AR glasses pops up a candidate information list, including navigation information (such as straight ahead for 2 kilometers and then turn left), life service information (such as a convenience store 1 kilometer ahead, available for charging), and safety warning information (such as a motor vehicle approaching 10 meters behind). The user selects at least one item in the candidate information list as the interaction information. In the case where the user does not select, the default selected information in the riding scene mode can be used as the interaction information.

[0105] In the riding scene mode, the embodiments of the present application determine the interaction information through real-time data of the target terminal device, state information of the riding vehicle, and environmental information, focusing on navigation guidance, life services, and safety warnings, so that the rider can master the route and vehicle conditions (such as remaining power and turning prompts) in real time, and can avoid risks in advance (such as obstacle warnings) through environmental perception, thereby improving the safety and convenience of the riding process.

[0106] In case six, in the case where the interaction scene mode is a work scene mode, the interaction information displayed by the extended reality device is determined based on at least one of real-time data of the target terminal device, environmental information, or selection operation information of the user, and the interaction information includes at least one of schedule information, meeting information, document information, communication and collaboration information, or task progress information. The meeting information includes at least one of meeting time, meeting theme, meeting participants, or meeting minutes.

[0107] Exemplarily, the real-time data of the target terminal device refers to device synchronization data related to the work process, including document editing progress of an office computer, message updates of an enterprise collaboration platform, and check-in status of a smart attendance device. The environmental information refers to external data related to the work scene, such as the current office location, indoor temperature and humidity, and the online status of colleagues. The selection operation information of the user refers to work instructions initiated by the user through voice, gesture, or touch.

[0108] The interaction information is information displayed by the extended reality device in the work scenario mode, which is determined from real-time data of the target terminal device, environment information, or user selection operation information.

[0109] Taking AR glasses as an example, the user wears the AR glasses into a company office (identified as a work scenario through environment information), and the AR glasses automatically connect to an office computer and an enterprise collaboration platform and switch to a work scenario mode. At this time, the display interface of the AR glasses pops up a candidate information list, including schedule information, meeting information, document information, communication collaboration information, and task progress information. The user selects at least one item in the candidate information list as the interaction information. In the case where the user does not select, the default selection information of the work scenario mode can be used as the interaction information.

[0110] In the work scenario mode, the embodiments of the present application determine the interaction information through real-time data of the target terminal device, environment information, or user selection operation, focus on office and collaboration needs, let the user master the work rhythm (such as meeting arrangement and task progress) in real time, and can quickly retrieve the required information (such as documents and communication content) through active selection. The information presentation mode that fits the office scene reduces operation interference and helps to improve work concentration and collaboration efficiency, optimizing the office experience in the work scenario.

[0111] It should be noted that cases one to six are exemplary descriptions of application scenarios of the extended reality device, and the extended reality device can also be applied in other scenarios, which are not limited by the present application.

[0112] Optionally, when the reference scenario of the extended terminal device changes, the target terminal device is automatically determined and connected. For example, the current reference scenario is a driving scenario, the target terminal device is a vehicle terminal, and the extended terminal device is connected to the vehicle terminal. When the vehicle is turned off, the target terminal device is re-determined as a mobile phone terminal, and the extended terminal device is automatically connected to the mobile phone terminal.

[0113] In the exemplary embodiments of the present application, an APP for interacting with the extended reality device is installed in the terminal device, and the interaction information displayed by the extended reality device is set through the APP.

[0114] In an embodiment, the terminal device is a vehicle-mounted terminal, and the APP is a vehicle-machine version APP. The vehicle-machine version APP can read vehicle-machine information through a local Controller Area Network (CAN). For example, vehicle speed, fuel level, power level, navigation, vehicle location, and the like. When the extended reality device is connected to the vehicle-machine APP through wireless means such as Wi-Fi or Bluetooth, the vehicle-machine information is obtained in real time, and the display content of the information, i.e., the interactive information, can be customized through the vehicle-machine version APP. Among them, after the vehicle is turned off, the vehicle location recorded by the vehicle-machine version APP in real time can be used for the user to find the vehicle in the subsequent process. Alternatively, the vehicle-machine version APP prompts the user whether to start the video recording function. After the video recording function is started, the surrounding environment video can be recorded. The video contains key spatial features in the garage, such as unique column markers, wall graffiti, parking space numbers, passage directions, and other environmental details, which facilitates finding the vehicle in a signal-free underground garage.

[0115] In another embodiment, the terminal device is a mobile phone, and the APP is a mobile phone version APP. The mobile phone version APP obtains mobile phone information in real time through a local system interface (Application Programming Interface, API). For example, phone, message, navigation, and the like. When the extended reality device is connected to the mobile phone APP through wireless means such as Wi-Fi or Bluetooth, the mobile phone information is obtained in real time, and the display content of the information, i.e., the interactive information, can be customized through the mobile phone version APP.

[0116] In another embodiment, the terminal device is a computer, and the computer can be installed with a computer version APP or a cloud web page capable of interacting with the extended reality device. After the user logs in through a unified account, a wireless connection between the computer and the extended reality device is established, and the extended reality device can obtain computer information in real time. For example, document data, photos, videos, and the like. The display content of the information, i.e., the interactive information, can be customized through the computer version APP or the cloud web page.

[0117] In the exemplary embodiments of the present application, the target terminal device can also be a control system of a smart terminal device. Exemplarily, the current office building scene or home scene is determined through reference information such as location information, environmental information, and time information; the target control system of the smart terminal device (such as an office building elevator control system or a home elevator terminal) is determined from the candidate control systems of the smart terminal device (such as an elevator control system and an access control device control system) based on the reference scene, and a connection is automatically established; then a corresponding interactive scene mode (such as an office scene mode or a home scene mode) is matched; the interactive information displayed by the extended reality device (such as elevator reservation options, waiting time, target floor prompts, and the like) is determined based on the interactive scene mode, and the user can complete operations such as elevator reservation through voice or gestures.

[0118] Optionally, the extended reality device is associated with at least one virtual object, and upon receiving a wake-up operation for the virtual object, the extended reality device invokes a virtual object corresponding to the interaction scene mode based on the extended reality device; upon receiving an interaction operation based on the virtual object, the virtual object is controlled to perform a target action based on the interaction operation and the reference scene.

[0119] In the exemplary embodiments of the present application, the extended reality device pre-binds at least one virtual object (such as an AI assistant, a virtual tool, etc.) with different interaction scene modes, and each scene mode corresponds to an adapted virtual object image. When the user initiates a wake-up operation for the virtual object, the extended reality device invokes the virtual object associated with the currently activated interaction scene mode and causes it to appear in the field of view, wherein the wake-up operation of the virtual object includes but is not limited to voice instructions, specific gestures, or clicking a virtual button to wake up the virtual object.

[0120] In the terminal device, the user can log in to an application that can interact with the extended reality device, and after logging in to the application, the user can wake up the virtual object (such as an AI assistant). For example, the user wakes up the virtual object by specifying a wake-up word, and interacts with the virtual object (such as body movements, instruction issuance) through voice, gestures, touch, etc., and the extended reality device recognizes the interaction operation and controls the virtual object to perform a target action (such as action feedback, function invocation, content display, etc.) that is adapted to the operation intent and the current reference scene, while ensuring that the action logic conforms to the scene characteristics (such as the interaction in the cockpit does not exceed the spatial range, and the action in the office scene does not interfere with the work field of view).

[0121] Optionally, the virtual object can choose whether to record personal information according to the user privacy agreement, and if the user agrees to record personal information, the virtual object stores all the user's personal APP login accounts and passwords that have been logged in, and when the user opens the same APP in different terminal devices, the virtual object will remind the user whether to use the stored personal account and password to log in.

[0122] Take the virtual object as an AI assistant as an example. For example, in the passenger scenario mode, the virtual object associated with the passenger scenario mode is a cabin AI assistant, and the user initiates a wake-up operation through voice or gesture. The AR glasses call the cabin AI assistant and make it appear in the cabin (for example, in the space in front of the co-pilot) in a three-dimensional form. When the user makes a "pop" gesture (quickly pushing the fingers forward), the cabin AI assistant performs a "spin away from near to far" motion effect, which conforms to the physical logic in the cabin space. When the user's voice instruction is "open the movie application (APP)", the cabin AI assistant "pulls out" the icon of the APP from the virtual pocket, gradually enlarges it to the user's preset size and plays the content, and then "shrinks and puts it back into the pocket" when it is closed. The action is adapted to the entertainment needs of the passenger. When the user touches the cabin AI assistant, the cabin AI assistant presents a "cute and comfortable" expression, which responds to the emotional logic of interpersonal interaction.

[0123] For another example, in the work scenario mode, the virtual object associated with the leisure scenario mode is a work AI assistant. The user wakes up through voice when working from home, and the AR glasses call the work AI assistant to make it float in the air or on the top of the computer. When the user is focused on work, the user can "make the assistant quiet" through gestures, and the work AI assistant performs a "stand still in the corner" action to avoid interference. When the user's instruction is "tell a story", the work AI assistant makes gestures and changes expressions in the air, and cooperates with voice narration, which meets the relaxation needs of the work scenario. When the user selects "only voice interaction", the work AI assistant hides the virtual image and only responds to instructions through voice, which adapts to the scene needs of focused work.

[0124] It should be noted that when the extended reality device or the terminal device meets the matching condition, the multiple extended reality devices or terminal devices are matched, and after the matching is completed, multiple virtual objects are displayed in the extended reality device. The multiple virtual objects are virtual objects corresponding to the terminal devices or the extended reality devices that complete the matching. The matching condition includes but is not limited to that the extended reality device or the terminal device supports a cross-platform interconnection protocol and is in a network connection state.

[0125] Take AR glasses as an example, and take an AI assistant as an example. A user initiates a request through a voice instruction, an interface operation, or a preset rule, for example, calls an AI assistant of a colleague, selects an add matching device in an AI assistant menu, and automatically matches a device for a team meeting every week three. The AI assistant transmits information including a user identifier, a terminal type, and a current scene to a cloud server. The cloud server verifies the legality of the request (for example, whether the other party is a friend or whether the other party is in a meeting invitation list), and pushes a matching invitation to a terminal device or an AI glass of a user to be matched. After the terminal device or the AI glass of the user to be matched confirms the matching invitation, the server establishes an encrypted data channel, and completes online matching of two or more terminal devices or AI glasses. At this time, the AI glass synchronously acquires an AI assistant of the other party. That is, the user can see a virtual image of the AI assistant of the other party (for example, a 3D cartoon image, a digital avatar, or the like) in the AR glass, and realizes an interactive function through the AI assistant.

[0126] Interactive function one, online meeting. The AI assistant simulates a real meeting scene, for example, AI assistants of all participants sit around a virtual conference room, an AI assistant of a speaker makes a hand-raising and nodding action, and cooperates with voice to synchronously deliver a user emotion (for example, an image expression is adjusted when a speech speed changes).

[0127] Interactive function two, voice chat and social interaction. The AI assistant simulates a real social scene, for example, AI assistants of all participants are present in a virtual space in a standing side by side or sitting opposite each other posture, an AI assistant of a speaker makes a waving hand and smiling action, and cooperates with voice to synchronously deliver a user emotion.

[0128] Interactive function three, file transmission and collaboration. The AI assistant simulates a real office scene, for example, AI assistants of all collaborators are distributed around a virtual workbench, an AI assistant of an initiator of transmission makes a taking and delivering action, and cooperates with a file state to synchronously deliver an operation progress.

[0129] It should be noted that the AI assistant realizes an interactive function as an example, and the AI assistant can also realize other interactive functions, which are not limited in the present application.

[0130] Embodiments of the present application bind a virtual object and an interactive scene mode, combine a wake-up operation and a scene-based interaction response, adapt an image and an action of the virtual object to user demand and a reference scene, enhance immersion and interest of interaction, and improve efficiency of function calling through scene-based logic.

[0131] The application identifies the reference scene where the extended reality device is located through reference information, automatically connects the target terminal device interacting with the extended reality device, and does not need the user to manually switch the target terminal device interacting with the extended reality device, thereby improving the interaction efficiency of the extended reality device and the terminal device. The application determines the adaptive interaction scene mode based on the reference scene and the target terminal device, does not need the user to manually switch the interaction scene mode, realizes seamless fusion and cooperation of multi-device information, and improves the convenience of cross-scene use of the extended reality device. The application determines the interaction information of the extended reality device based on the interaction scene mode, matches the interaction information and the interaction scene mode, avoids invalid information interference, and thereby improves the information display efficiency.

[0132] The application further provides an interaction device of an extended reality device. Figure 3 is a schematic diagram of an interaction device of an extended reality device provided by an embodiment of the application, as Figure 3 shown, the device comprises:

[0133] A determination module 301 is configured to determine the reference scene where the extended reality device is located based on reference information, and the reference information comprises at least one of position information, environment information, device connection signals, or time information.

[0134] A control module 302 is configured to control the extended reality device to automatically connect with a target terminal device by using the reference scene, and the target terminal device is determined based on the reference scene and candidate terminal devices, and the candidate terminal devices are terminal devices capable of interacting with the extended reality device.

[0135] The determination module 301 is further configured to determine an interaction scene mode based on the reference scene and the target terminal device, and the interaction scene mode comprises any one of a driving scene mode, a passenger scene mode, a leisure scene mode, a sports scene mode, or a riding scene mode.

[0136] The determination module 301 is further configured to determine interaction information displayed by the extended reality device based on the interaction scene mode.

[0137] In a possible implementation, the determination module 301 is further configured to determine the candidate terminal device corresponding to the reference scene in the case where the extended reality device in the reference scene meets a connection condition, determine the candidate terminal device as the target terminal device in the case where the number of the candidate terminal devices is one, or determine the connection priority of each candidate terminal device and the extended reality device in the case where the number of the candidate terminal devices is multiple, and determine the target terminal device among the candidate terminal devices based on the connection priority.

[0138] In a possible implementation, the determining module 301 is configured to acquire scene feature information corresponding to a reference scene, the scene feature information being used to distinguish an interactive scene mode in the reference scene; match the scene feature information with a scene mode mapping strategy corresponding to the target terminal device to obtain a matching result, the scene mode mapping strategy representing a corresponding relationship between the scene feature information and the interactive scene mode; and determine the interactive scene mode based on the matching result.

[0139] In a possible implementation, the extended reality device is associated with at least one virtual object, and the control module 302 is further configured to receive a wake-up operation for the virtual object, invoke the virtual object corresponding to the interactive scene mode based on the extended reality device; and in a case where an interactive operation based on the virtual object is received, control the virtual object to perform a target action based on the interactive operation, the target action being adapted to the interactive operation and the reference scene.

[0140] In a possible implementation, the determining module 301 is configured to, in a case where the interactive scene mode is a driving scene mode, determine interactive information displayed by the extended reality device based on driving parameters and environment information of a vehicle, the interactive information being information related to a driving process of the vehicle, the interactive information including at least one of vehicle parameter information, driving assistance information, or environment information, the driving assistance information including at least one of lane departure warning information, collision warning information, lane guiding information, or navigation information, the vehicle parameter information being directly obtained based on the driving parameters, and the driving assistance information being indirectly obtained based on the driving parameters.

[0141] In a possible implementation, the determining module 301 is configured to, in a case where the interactive scene mode is a passenger scene mode, determine interactive information displayed by the extended reality device based on at least one of environment information or selection operation information of a user, the interactive information including at least one of life service information or entertainment information.

[0142] In a possible implementation, the determining module 301 is configured to, in a case where the interactive scene mode is a leisure scene mode, determine interactive information displayed by the extended reality device based on at least one of real-time data of the target terminal device, environment information, or selection operation information of a user, the interactive information including at least one of basic information, social communication information, life service information, media entertainment information, or navigation information, the basic information including at least one of time information, location information, or weather information.

[0143] In a possible implementation, the determining module 301 is configured to, in a case where the interactive scene mode is a sports scene mode, determine interactive information of the extended reality device based on at least one of real-time data of the target terminal device and selection operation information of a user, the interactive information including at least one of a sports type identifier or real-time sports data.

[0144] In a possible implementation, the determining module 301 is configured to determine, in a case where the interaction scene mode is the riding scene mode, the interaction information displayed by the extended reality device based on at least one of the real-time data of the target terminal device, the state information of the riding vehicle, or the environmental information, the interaction information including at least one of navigation information, life service information, or safety warning information.

[0145] The present application identifies the reference scene in which the extended reality device is located by referring to information, automatically connects the target terminal device interacting with the extended reality device, and does not need the user to manually switch the target terminal device interacting with the extended reality device, thereby improving the interaction efficiency of the extended reality device and the terminal device. In addition, the adaptive interaction scene mode is determined based on the reference scene and the target terminal device, and the user does not need to manually switch the interaction scene mode, thereby realizing seamless fusion and cooperation of multi-device information and improving the convenience of cross-scene use of the extended reality device. The interaction information of the extended reality device is determined based on the interaction scene mode, the interaction information is matched with the interaction scene mode, invalid information interference is avoided, and the information display efficiency is improved.

[0146] It should be understood that the above-provided apparatus, when implementing functions thereof, is only exemplified by the above division of functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the above-described functions. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be described here.

[0147] Figure 4 The server provided in the embodiments of the present application has a structure diagram, the server 400 can have great differences due to different configurations or performances, and can include one or more processors 401 and one or more memories 402, wherein the one or more memories 402 store at least one program code, the at least one program code is loaded and executed by the one or more processors 401 to implement the interaction method of the extended reality device provided in each method embodiment. Of course, the server 400 can also have a wired or wireless network interface, a keyboard, and an input and output interface, and other components for realizing the functions of the apparatus, and details are not described here.

[0148] In an exemplary embodiment, a computer readable storage medium is also provided, the storage medium stores at least one program code, the at least one program code is loaded and executed by a processor to enable a computer to implement any of the above-mentioned interaction methods of the extended reality device.

[0149] Optionally, the computer readable storage medium described above can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0150] In the example embodiments, a computer program or a computer program product is also provided, which stores at least one computer instruction loaded and executed by a processor to enable a computer to implement any of the above-mentioned interaction methods of the extended reality device.

[0151] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the reference information, interaction scene mode and interaction information involved in the present application are all obtained under full authorization.

[0152] It should be understood that "multiple" referred to herein means two or more. "And / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship.

[0153] The above only describes the example embodiments of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of interaction of an extended reality device, characterized in that, The method comprises: determining a reference scene in which an extended reality device is located based on reference information, the reference information comprising at least one of position information, environment information, device connection signals, or time information; controlling the extended reality device to automatically connect with a target terminal device based on the reference scene, the target terminal device being determined based on the reference scene and a candidate terminal device, the candidate terminal device being a terminal device capable of interacting with the extended reality device; determining an interaction scene mode based on the reference scene and the target terminal device, the interaction scene mode comprising any one of a driving scene mode, a passenger scene mode, a leisure scene mode, a sports scene mode, or a cycling scene mode; determining interaction information displayed by the extended reality device based on the interaction scene mode.

2. The method of claim 1, wherein, Before the extended reality device is controlled to automatically connect with the target terminal device based on the reference scene, the method further comprises: in a case where the extended reality device in the reference scene meets a connection condition, determining a candidate terminal device corresponding to the reference scene; in a case where the number of candidate terminal devices is one, determining the candidate terminal device as the target terminal device; or, in a case where the number of candidate terminal devices is multiple, determining a connection priority of each candidate terminal device with the extended reality device, and determining the target terminal device among the candidate terminal devices based on the connection priority.

3. The method of claim 1, wherein, The determination of the interaction scene mode based on the reference scene and the target terminal device comprises: obtaining scene feature information corresponding to the reference scene, the scene feature information being used to distinguish the interaction scene mode in the reference scene; matching the scene feature information with a scene mode mapping strategy corresponding to the target terminal device to obtain a matching result, the scene mode mapping strategy representing a corresponding relationship between the scene feature information and the interaction scene mode; determining the interaction scene mode based on the matching result.

4. The method of claim 1, wherein, The extended reality device is associated with at least one virtual object, and the method further comprises: receiving a wake-up operation for the virtual object, and invoking a virtual object corresponding to the interaction scene mode based on the extended reality device; in a case where an interaction operation based on the virtual object is received, controlling the virtual object to perform a target action based on the interaction operation, the target action being adapted to the interaction operation and the reference scene.

5. The method according to any one of claims 1 to 4, characterized in that, The determination of the interaction information displayed by the extended reality device based on the interaction scene mode comprises: In a case where the interaction scenario mode is the driving scenario mode, the interaction information displayed by the extended reality device is determined based on the driving parameter of the vehicle and the environment information, the interaction information being information related to a driving process of the vehicle, the interaction information including at least one of vehicle parameter information, driving assistance information, or the environment information, the driving assistance information including at least one of lane departure warning information, collision warning information, lane guidance information, or navigation information, the vehicle parameter information being directly obtained based on the driving parameter, and the driving assistance information being indirectly obtained based on the driving parameter.

6. The method according to any one of claims 1 to 4, characterized in that, The determining of the interaction information displayed by the extended reality device based on the interaction scenario mode includes: In a case where the interaction scenario mode is the passenger scenario mode, the interaction information displayed by the extended reality device is determined based on at least one of the environment information or selection operation information of a user, the interaction information including at least one of life service information or entertainment information.

7. The method according to any one of claims 1 to 4, characterized in that, The determining of the interaction information displayed by the extended reality device based on the interaction scenario mode includes: In a case where the interaction scenario mode is the leisure scenario mode, the interaction information displayed by the extended reality device is determined based on at least one of real-time data of the target terminal device, the environment information, or selection operation information of a user, the interaction information including at least one of basic information, social communication information, life service information, media entertainment information, or navigation information, the basic information including at least one of time information, location information, or weather information.

8. The method according to any one of claims 1 to 4, characterized in that, The determining of the interaction information displayed by the extended reality device based on the interaction scenario mode includes: In a case where the interaction scenario mode is the sports scenario mode, the interaction information of the extended reality device is determined based on at least one of real-time data of the target terminal device and selection operation information of a user, the interaction information including at least one of sports type identification or real-time sports data.

9. The method according to any one of claims 1 to 4, characterized in that, The determining of the interaction information displayed by the extended reality device based on the interaction scenario mode includes: In a case where the interaction scenario mode is the cycling scenario mode, the interaction information displayed by the extended reality device is determined based on at least one of real-time data of the target terminal device, state information of a cycling vehicle, or the environment information, the interaction information including at least one of navigation information, life service information, or safety warning information.

10. An interaction device of an extended reality device, characterized in that, The apparatus includes: A determination module configured to determine a reference scenario in which an extended reality device is located based on reference information, the reference information including at least one of location information, environment information, device connection signals, or time information; A control module configured to control the extended reality device to automatically connect with a target terminal device based on the reference scenario, the target terminal device being determined based on the reference scenario and candidate terminal devices, the candidate terminal devices being terminal devices capable of interacting with the extended reality device. The determining module is further configured to determine an interaction scene mode based on the reference scene and the target terminal device, the interaction scene mode including any one of a driving scene mode, a passenger scene mode, a leisure scene mode, a sports scene mode, or a cycling scene mode. The determining module is further configured to determine interaction information displayed by the extended reality device based on the interaction scene mode.

Images