Augmented reality information notification method and device and electronic equipment

Abstract

The embodiment of the invention discloses an augmented reality information notification method and device and electronic equipment, and the method comprises the steps: recognizing target terminal equipment from a current real scene image of the augmented reality equipment if the augmented reality equipment receives target information, and obtaining an equipment recognition result; and controlling output of notification information for the target information according to the equipment identification result. The output of the notification information of the target information is controlled by identifying whether the real scene picture contains the target terminal equipment or not, so that the output control of the notification information of the target information is realized. If the target terminal device is contained in the real scene picture, whether the user has the condition of directly viewing the target information through the target terminal device is indicated, and on the basis of controlling the output of the notification information corresponding to the target information, the reasonability of outputting the notification information is improved, and the power consumption of the device caused by unnecessary notification is reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of extended reality technology, and specifically to an extended reality information notification method, device, and electronic device. Background Art

[0002] Extended reality technology expands human perception and interaction methods. Extended reality (XR) is a comprehensive term that covers virtual reality (VR), augmented reality (AR) and mixed reality (MR). Through digital technology, it interacts with the real world to create virtual, augmented and mixed reality experiences. Extended reality provides a more intuitive and convenient content presentation experience through virtual reality scenes or extended reality scenes. It has developed rapidly in the fields of entertainment, games, education, medical care, enterprise production and social communication, showing great potential and influence.

[0003] With the promotion of extended reality devices, more and more applications can be used on extended reality devices, and it is possible for extended reality devices and other terminal devices to form a multi-terminal login scenario for the same application (including a dual-terminal login scenario for the same application). For example, extended reality devices and other terminal devices simultaneously run call applications, text messaging applications, WeChat applications, or video conferencing applications. After a user logs in to the same account for the same application on two or more terminal devices including the extended reality device, the user can receive, send, or process information on the two devices respectively.

[0004] However, since the extended reality device and other terminal devices are logged in to the same account at the same time, when the account receives information, notification information for the information is usually output on the two terminal devices at the same time, resulting in notification waste and additional power consumption. Summary of the invention

[0005] The embodiments of the present application provide an extended reality information notification method, device, and electronic device, which can improve the rationality of information notification, avoid notification waste, and reduce device power consumption.

[0006] In a first aspect, an embodiment of the present application provides an extended reality information notification method, which is applied to an extended reality device, and the method includes:

[0007] If the target information is received, the target terminal device is identified from the current real scene image of the extended reality device to obtain a device identification result;

[0008] The output of notification information for the target information is controlled according to the device identification result.

[0009] In a second aspect, an embodiment of the present application further provides an extended reality information notification device, which is applied to an extended reality device, and the device includes:

[0010] An identification module, configured to identify a target terminal device from the current real scene image of the extended reality device if target information is received, and obtain a device identification result;

[0011] A control module is used to control the output of notification information for the target information according to the device identification result.

[0012] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps in the above-mentioned extended reality information notification method when executed by the processor.

[0013] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned extended reality information notification method are implemented.

[0014] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0015] If the extended reality device of the embodiment of the present application receives target information, it identifies the target terminal device from the current real scene screen of the extended reality device, obtains a device identification result, and controls the output of notification information for the target information based on the device identification result.

[0016] Among them, after the extended reality device receives the target information, it controls the output of the notification information of the target information by identifying whether the real scene screen contains the target terminal device, thereby realizing the output control of the notification information of the target information. Whether the real scene screen contains the target terminal device indicates whether the user has the conditions to directly view the target information through the target terminal device. Based on this, the output of the notification information corresponding to the target information is controlled, the rationality of the notification information output is improved, and the power consumption of the device caused by unnecessary notifications is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a scene diagram of the extended reality information notification method provided in an embodiment of the present application;

[0019] Figure 2 It is a flowchart of the extended reality information notification method provided in an embodiment of the present application;

[0020] Figure 3 It is a framework diagram of the extended reality call reminder system provided in an embodiment of the present application;

[0021] Figure 4 is another flowchart of the extended reality information notification method provided in an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of the structure of an extended reality information notification device provided in an embodiment of the present application;

[0023] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0025] The embodiments of the present application provide an extended reality information notification method, device and electronic device. Specifically, the embodiments of the present application provide an extended reality information notification device suitable for an electronic device (extended reality device), which is used to output and control the notification information between the extended reality device and the target terminal device, and the target terminal device is a terminal device other than the extended reality device, and the extended reality device includes but is not limited to a head-mounted display and wearable glasses. The extended reality device can be an integrated extended reality device built into the computing processing unit, or a split extended reality device external to the computing processing unit. Among them, the extended reality device includes but is not limited to an airborne optical display system, i.e., a head-up display system, applied to vehicles such as aircraft, automobiles, and ships, for example, an AR-HUD (Augmented reality hud-up display) mounted on an intelligent networked car, a handheld mobile device such as a mobile phone, a desktop computer, a laptop computer, a tablet, etc., and a wearable near-eye display system such as a head-mounted display, smart glasses, etc. When the extended reality device is a wearable head-mounted display or smart glasses, it can be an integrated extended reality device built into the computing processing unit, or a split extended reality device external to the computing processing unit. The target terminal device may be a mobile phone, tablet computer, laptop computer, desktop computer or TV and other devices.

[0026] See also Figure 1 , Figure 1 : is a schematic diagram of a scenario in which an extended reality device according to an embodiment of the present application performs the extended reality information notification method, wherein the specific execution process of the extended reality device performing the extended reality information notification method is as follows:

[0027] If the extended reality device 10 receives the target information, it identifies the target terminal device 11 from the current real scene screen of the extended reality device 10, obtains a device identification result, and controls the output of notification information for the target information according to the device identification result.

[0028] like Figure 1 As shown, taking the extended reality device as optical see-through smart glasses as an example, the extended reality device at least includes a glasses body, an optical display module, electronic circuit components, sensors, etc.

[0029] It is understandable that after the extended reality device receives the target information, it controls the output of the notification information of the target information by identifying whether the target terminal device is included in the real scene screen, thereby realizing the output control of the notification information of the target information. Whether the target terminal device is included in the real scene screen indicates whether the user has the conditions to directly view the target information through the target terminal device. Based on this, the output of the notification information corresponding to the target information is controlled to improve the rationality of the output of the notification information, which not only reduces the power consumption loss caused by the output of unnecessary notification information of the extended reality device, but also can ensure that the notification information will not be missed to the greatest extent.

[0030] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0031] See also Figure 2 , Figure 2 A flowchart of an extended reality information notification method provided in an embodiment of the present application. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings. Specifically, the extended reality call reminder specifically includes:

[0032] 101. If target information is received, the target terminal device is identified from the current real scene image of the extended reality device to obtain a device identification result.

[0033] The target information is information sent by other terminal devices to the extended reality device. For example, other devices send information to the SMS application or WeChat message of the current extended reality device through the SMS application or WeChat message. It can be understood that the target information can be text information, video information, or incoming call request information, wherein the incoming call request refers to a request for telephone communication, which is a call request initiated by the initiator of the telephone communication to the current extended reality device side. The embodiment of the present application mainly describes how to improve the rationality of the output of device notification information from the side receiving the telephone request.

[0034] Among them, the target terminal device and the extended reality device are in the same application multi-terminal login scenario. It should be noted that the same application multi-terminal login scenario includes the same application dual-terminal login scenario, that is, the same account is logged in on two, three or more devices at the same time, such as one number and dual terminals. The target terminal device can be a mobile phone, tablet or other device that supports the same application multi-terminal login and is other than the current extended reality device.

[0035] In some embodiments, the target terminal device may also be an extended reality device, that is, two, three or more extended reality devices control the output of notification information through the extended reality information notification method.

[0036] It should be noted that an extended reality device is a device that combines real content with virtual content, such as an augmented reality device (AR) in an extended reality device, which is a device that superimposes virtual content on real content for display. The real scene picture in the embodiment of the present application is a scene picture corresponding to the real content, such as a picture of a real scene viewed through the screen of an augmented reality device.

[0037] It should be noted that the device recognition result includes one of the two types: the target terminal device is included in the real scene picture and the target terminal device is not included.

[0038] It should be noted that since the extended reality device and the target terminal device are in the same application multi-terminal login scenario, the extended reality device and the target terminal device will receive the target information sent by other users or other terminals at the same time. However, if the notification information of the target information is output on two terminal devices at the same time, there is a problem of notification waste and extra power consumption. For example, if the extended reality device and the target terminal device simultaneously output call reminders for incoming call requests, the user can only respond to the call reminder on one terminal device, resulting in an invalid call reminder on the other terminal device. For extended reality devices, which already have demanding battery life, reducing their invalid information notifications is of great significance to extending their battery life.

[0039] 102. Control output of notification information for the target information according to the device identification result.

[0040] Among them, the device recognition result represents whether the target terminal device can be browsed through the field of view of the extended reality device. If the target terminal device can be browsed through the extended reality device, it means that the user can notice the notification information on the target terminal device, and in order to reduce or control the power consumption of the extended reality device, the notification information corresponding to the target information can be output through the target terminal device in priority. Conversely, if the target terminal device cannot be browsed through the extended reality device, in order to avoid the user missing the target information, the notification information corresponding to the target information can be directly output on the extended reality device.

[0041] Therefore, in the embodiment of the present application, the step of “controlling the output of notification information for the target information according to the device identification result” includes:

[0042] If the device recognition result indicates that the real scene image does not include the target terminal device, outputting notification information for the target information;

[0043] If the device recognition result indicates that the target terminal device is included in the real scene image, the output of notification information for the target information is canceled, and the target terminal device is controlled to output the notification information.

[0044] Among them, in the embodiment of the present application, when the extended reality device outputs notification information, the target terminal device no longer outputs notification information, that is, only one terminal device outputs notification information at the same time.

[0045] In summary, after receiving the target information, the extended reality device of the embodiment of the present application controls the output of the notification information of the target information by identifying whether the target terminal device is included in the real scene screen, thereby realizing the output control of the notification information of the target information. Whether the target terminal device is included in the real scene screen indicates whether the user has the conditions to directly view the target information through the target terminal device. Based on this, the output of the notification information corresponding to the target information is controlled to improve the rationality of the output of the notification information, which not only reduces the power consumption loss caused by the output of unnecessary notification information of the extended reality device, but also can ensure that the notification information will not be missed to the greatest extent.

[0046] Among them, in the embodiments of the present application, in order to improve the accuracy of the output of the notification information, it is also possible to determine whether the extended reality device outputs the notification information corresponding to the target information according to the wearing state of the extended reality device, that is, optionally, in some embodiments of the present application, the step of "if the target information is received, identifying the target terminal device from the current real scene screen of the extended reality device to obtain the device identification result" includes:

[0047] If the target information is received, identifying the wearing state of the extended reality device;

[0048] If the wearing state indicates that the extended reality device is currently being worn, identifying a target terminal device from a current real scene screen of the extended reality device to obtain a device identification result;

[0049] If the wearing state indicates that the extended reality device is not currently being worn, then controlling the output of notification information for the target information according to the device recognition result includes:

[0050] The output of notification information for the target information is canceled, and the target terminal device is directly controlled to output notification information for the target information.

[0051] That is, if the extended reality device is in a worn state, then when the extended reality device outputs the notification information, the user can directly and quickly perceive the notification information. Conversely, if the user is not wearing the extended reality device, even if the notification information is output through the extended reality device, the user will not be able to perceive the notification information in time, resulting in an invalid notification.

[0052] Therefore, in order to avoid invalid power consumption of the extended reality device, when the extended reality device is not worn, the generation of notification information and the output of notification information are directly canceled, and the target terminal device is controlled to output the notification information. When the extended reality device is worn, if the user can still conveniently perceive the notification information from the target terminal device, the notification information can still be output through the target terminal device. Therefore, in the embodiment of the present application, when the extended reality device is in a worn state, the notification information corresponding to the target information is not directly output through the extended reality device, but it is analyzed whether the target terminal device can be browsed through the extended reality device. If it can be browsed, the notification information is still output to the user by outputting the notification information through the target terminal device. If not, the notification information is output through the extended reality device. Fully control or fully reduce the power consumption of the extended reality device, and ensure that at least one device and a device that is easy for the user to perceive outputs the notification information to avoid users missing information.

[0053] Optionally, although the extended reality device and the target terminal device support the same application multi-terminal login scenario, in some cases, the same account is not logged in to multiple terminals at the same time. Therefore, in order to improve the accuracy of notification information output control and avoid users missing notification information, when performing output control on notification information, it is also possible to determine whether the target terminal device is currently in a logged-in state, and select one of the extended reality device and the target terminal device according to the login status to output the notification information. That is, optionally, in some embodiments of the present application, the step of "if the wearing state indicates that the extended reality device is currently being worn, identifying the target terminal device from the current real scene screen of the extended reality device to obtain a device identification result" includes:

[0054] If the wearing state indicates that the extended reality device is currently being worn, identifying an application scenario of the extended reality device and the target terminal device;

[0055] If the application scenario is a multi-terminal login scenario for the same application, identifying the target terminal device from the current real scene screen of the extended reality device to obtain a device identification result;

[0056] If the application scenario is not a same-application multi-terminal login scenario, notification information for the target information is directly output.

[0057] That is, when the extended reality device is in a worn state, the notification information is not output directly by the extended reality device, but further analysis is performed to determine whether there are target terminal devices with multiple terminals logged in to the same application, so as to further analyze which device is more suitable for outputting notification messages when the extended reality device is in a worn state.

[0058] The wearing status can be obtained by analyzing data collected by sensors arranged on the extended reality device. In some cases, it can also be determined based on the user's voice, for example, by indicating through voice that the extended reality device is currently in a wearing state.

[0059] Among them, when the application scenario of the extended reality device and the target terminal device is a multi-terminal login scenario for the same application, the target terminal device can be used as a potential device suitable for outputting notification information. Then, by identifying whether the target terminal device is included in the real scene screen, it is determined whether the target terminal device is suitable as the device for finally outputting notification information, thereby achieving a more refined analysis of the output of notification information, improving the rationality of the output of notification information, and helping to reduce the ineffective power consumption of the device.

[0060] Optionally, in some embodiments of the present application, if the real scene screen includes a target terminal device, but the target terminal device is far away from the current extended reality device, the user cannot conveniently view or timely perceive the notification information output by the target terminal device. Therefore, when the real scene includes a target terminal device, it is further possible to further analyze which terminal device is more suitable to output the notification information based on the distance between the target terminal device and the current extended reality device. That is, optionally, in some embodiments of the present application, the step of "controlling the output of the notification information for the target information according to the device recognition result" includes:

[0061] If the device recognition result indicates that the real scene image does not include the target terminal device, outputting notification information for the target information;

[0062] If the device recognition result indicates that the real scene image includes the target terminal device, obtaining the distance between the extended reality device and the target terminal device;

[0063] Furthermore, if the distance is less than a preset distance threshold, the output of notification information for the target information is canceled, and the target terminal device is controlled to output the notification information; if the distance is greater than or equal to the preset distance threshold, the notification information for the target information is output.

[0064] It is understandable that the distance between the target terminal device and the current extended reality device actually reflects the distance between the target terminal device and the current user, which is usually a straight-line distance or a horizontal distance between the user and the target terminal device.

[0065] Among them, in the embodiments of the present application, whether the real scene screen contains the target terminal device can be analyzed and determined by image recognition technology, that is, optionally, in some embodiments of the present application, the step of "identifying the target terminal device from the current real scene screen of the extended reality device to obtain a device recognition result" includes:

[0066] Inputting the scene image corresponding to the real scene picture into a target recognition model, and outputting a device recognition result for the target terminal device in the scene image through the target recognition model; and / or,

[0067] Based on the template image of the target terminal device, the scene image corresponding to the real scene picture is matched to obtain a matching result, and a device recognition result is generated according to the matching result.

[0068] Among them, the scene image is obtained by capturing the real scene picture, for example, the real scene picture is photographed by using a camera set on the extended reality device to obtain the scene image, wherein the real scene picture or scene image is the real content browsed through the optical lens (such as a physical screen) of the extended reality device. When the user wears the extended reality device, the real scene picture or scene image corresponds to the content currently browsed by the user when wearing the extended reality device.

[0069] Among them, the template image is the image corresponding to the target terminal device, which refers to the image of interest. By matching the template image with the scene image, the similarity between each area in the scene image and the template image can be identified, and then it can be analyzed whether the scene image contains the target terminal device, that is, whether the real scene picture contains the target terminal device.

[0070] It should be noted that the target recognition model is a machine learning model, which is a model that can perform image recognition processing after training the basic network. The basic network of the target recognition model includes but is not limited to convolutional neural network (CNN), regional convolutional neural network (R-CNN) or single-stage target detection network (such as YOLO (You Only LookOnce), SSD (Single Shot MultiBox Detector), etc.).

[0071] Before inputting the scene image into the target recognition model, the scene image may be preprocessed to improve the recognition accuracy of the target terminal device, for example, binarization, denoising, detail enhancement, etc. may be performed on the scene image.

[0072] Among them, in the embodiments of the present application, in order to further improve the rationality of the output of the notification information, that is, to control the power consumption of the device and prevent the user from missing the notification information, it is possible to analyze which device is more suitable for outputting the notification information in combination with the power information of the device, the carrying condition of the target terminal device, etc., that is, optionally, in some embodiments of the present application, the step of "controlling the output of the notification information for the target information according to the device recognition result" includes:

[0073] Acquire the wearing state information and power information of the extended reality device and the portable information of the target terminal device;

[0074] The output of notification information for the target information is controlled according to the device recognition result, the wearing state information, the power information and the carried information.

[0075] The carried-with information refers to information about whether the user carries the target terminal device with him / her, including one of the two situations: carrying it with him / her and not carrying it with him / her.

[0076] Among them, a comprehensive analysis of the initial switch state based on the wearing status, power information and carry-on information helps to remind the accuracy of the initial switch state.

[0077] Specifically, the initial switch state can be analyzed based on the wearing state, power information and carry-on information, including:

[0078] If the wearing state indicates that the extended reality device is not currently being worn, directly canceling the output of the notification information for the target information, and controlling the target terminal device to output the notification information;

[0079] If the wearing state indicates that the extended reality device is currently being worn, and the carried-on information indicates that the target terminal device is not carried on the body, directly outputting notification information for the target information;

[0080] If the wearing state indicates that the extended reality device is currently being worn, and the power information represents that the current power of the extended reality device exceeds the preset power threshold, directly outputting notification information for the target information;

[0081] If the wearing status indicates that the extended reality device is currently being worn, and the power information represents that the current power of the extended reality device does not exceed the preset power threshold, then directly cancel outputting the notification information for the target information, and control the target terminal device to output the notification information.

[0082] That is, regardless of whether the target terminal device is carried with you or not, notification information can be directly output when the extended reality device is worn and the available power is high.

[0083] When the extended reality device is worn and the target terminal device is not carried with the user, the notification information should be output regardless of the battery level of the extended reality device to prevent the user from missing the received target information. When the battery level is zero, the extended reality device cannot provide call reminders, so the target terminal device outputs the call reminders. Specifically, since the target terminal device is not carried with the user at this time, and in order to avoid the impact of the call reminder on other users or to ensure the protection of user privacy, the call reminder can be output on the target terminal device through SMS reminders or other methods.

[0084] When wearing an extended reality device and carrying a target terminal device, you can control whether to output a call reminder according to the battery level. For example, when the battery level of the extended reality device is high, you can control the extended reality device to output notification information. When the battery level of the extended reality device is low, you can control the extended reality device not to output notification information, and the target terminal device will output the notification information.

[0085] In summary, after the extended reality device receives the target information, the embodiment of the present application controls the output of the notification information of the target information by identifying whether the target terminal device is included in the real scene screen, thereby realizing the output control of the notification information of the target information. Whether the target terminal device is included in the real scene screen indicates whether the user has the conditions to directly view the target information through the target terminal device, and based on this, the output of the notification information corresponding to the target information is controlled, thereby improving the rationality of the output of the notification information and reducing the power consumption of the device caused by unnecessary notifications.

[0086] That is, by utilizing whether the real scene screen of the extended reality device includes the target terminal device, that is, analyzing whether the user can currently conveniently view notifications from the target terminal device, to select a device that is more suitable for outputting notification information, thereby reducing the ineffective power consumption of the device and avoiding users from missing information, and outputting notification information to users in a timely, convenient, and user-friendly manner.

[0087] Among them, by combining the wearing status, power information, and personal information to comprehensively analyze which device outputs the notification information, the rationality of the notification information output is improved and the user experience is improved.

[0088] Among them, in the embodiment of the present application, whether to output the notification information can be controlled by the state of the notification switch, that is, whether or not the notification information can be output is controlled by the on or off state of the notification switch. For example, taking the target information as a call reminder as an example, the notification switch is a call reminder switch. When the extended reality device cancels the output of the call reminder, the call reminder switch of the extended reality device is turned off, and the call reminder switch of the target terminal device is turned on. Conversely, when the extended reality device outputs the call reminder and the target terminal device cancels the output of the call reminder, the call reminder switch of the extended reality device is turned on, and the call reminder switch of the target terminal device is turned off.

[0089] It is understandable that the extended reality device maintains a communication connection with the target terminal device, such as connecting via Bluetooth, wifi or the operator network. When the extended reality device determines the current call reminder switch state, it can inform the target terminal device of its own situation through the communication connection, so as to prompt the target terminal device to adjust its own call reminder state based on the call reminder switch state of the extended reality device, so as to ensure that only one terminal device outputs the call reminder at the same time. For example, when the call reminder switch state of the extended reality device is to output the call reminder, the target terminal device turns off the call reminder. Conversely, when the call reminder switch state of the extended reality device is not to output the call reminder, the target terminal device turns on the call reminder.

[0090] Among them, it should be noted that when the real scene picture includes the target terminal device, it means that the user can currently browse the target device in the real scene through the extended reality device, and the user can directly answer the call through the target terminal device; conversely, when the real scene picture does not include the target terminal device, it means that the user cannot browse the target device in the real scene through the extended reality device, which to a large extent indicates that the user cannot directly answer the call through the target terminal device, or cannot easily and quickly answer the call through the target terminal device, such as the target terminal device is far away from the user or the target terminal device is blocked by other objects.

[0091] Therefore, determining the call reminder switch status based on the device identification result can improve the rationality of the device's call reminder. For example, when the target terminal device is not included in the real scene picture, the call reminder is output through the extended reality device; when the target terminal device is included in the real scene picture, the call reminder is turned off, that is, the call reminder corresponding to the call request is not output.

[0092] In order to more clearly illustrate the extended reality information notification method of the embodiment of the present application, the extended reality information notification method of the embodiment of the present application is described in combination with the target terminal device. For example, the extended reality device includes XR glasses, the target terminal device is a mobile phone, the notification information is a call reminder, and the same application multiple terminal login is a one-number dual terminal. For details, please refer to Figure 3 As shown, Figure 3 1 is a framework diagram of an extended reality call reminder system provided in an embodiment of the present application, wherein the extended reality call reminder system includes XR glasses and a mobile phone, the XR glasses include a glasses detection module 111, a glasses setting module 112, a glasses call reminder module 113 and a glasses communication module 114, and the mobile phone includes a mobile phone call reminder module 115, a mobile phone setting module 116 and a mobile phone communication module 117, specifically:

[0093] Glasses detection module 111: XR glasses can detect whether the user is wearing or taking off glasses through distance sensors or TP touch pads. XR glasses can periodically detect the power status of the glasses themselves and obtain the remaining power of the glasses.

[0094] Glasses setting module 112:

[0095] Preferably, a call reminder optimization switch is set in the glasses setting module. When the call reminder optimization switch is turned on, the XR glasses are set to turn on or off the call reminder function of the XR glasses according to the glasses detection module 111, and notify the mobile phone end; when the call reminder optimization switch is turned off, the XR glasses and the mobile phone of the one-number dual-terminal keep receiving the call reminder at the same time. That is, the call reminder optimization switch refers to a control switch for whether to optimize the call reminder output of the XR glasses and the mobile phone.

[0096] Preferably, a low-battery call reminder switch can be set in the glasses setting module 112, and a low-battery threshold value (such as less than 30% battery is low battery) can be defined. When the low-battery call reminder switch is turned on, the XR glasses detect that the user is wearing the XR glasses, and the XR glasses turn on the call reminder function; the XR glasses detect that the user takes off the XR glasses, and the glasses turn off the call reminder function.

[0097] When the low-battery call reminder switch is turned off: when the XR glasses detect that the user is wearing the XR glasses and the battery level of the XR glasses is greater than a threshold, the XR glasses turn on the call reminder function; when the XR glasses detect that the user is wearing the XR glasses and the battery level of the XR glasses is less than a threshold, the XR glasses turn off the call reminder function; among them, regardless of whether the battery level is higher than the threshold, as long as the XR glasses detect that the user has taken off the XR glasses, the XR glasses turn off the call reminder function.

[0098] Glasses incoming call reminder module 113: When the incoming call reminder function of the XR glasses is turned on, the XR glasses will remind you of incoming calls; when the incoming call reminder function is turned off, the glasses will not remind you of incoming calls.

[0099] Glasses communication module 114 and mobile phone communication module 117: XR glasses and mobile phones establish connections and communicate through low-power Bluetooth, WIFI, operator networks and other connection technologies. Usually, when the two terminals are close, they will use low-power Bluetooth or WIFI to maintain pairing connection, so they can communicate through low-power Bluetooth or WIFI; when they are far away, such as wearing XR glasses and leaving the phone at home, they can communicate through the operator network.

[0100] When the XR glasses described in the glasses setting module 112 turn on or off the incoming call function, the mobile phone setting module 116 will be informed through the glasses communication module 114 and the mobile phone communication module 117. After the mobile phone setting module 116 obtains the status of the glasses' incoming call reminder function, it will make corresponding settings.

[0101] Mobile phone setting module 116: The mobile phone setting module will save the incoming call reminder mode set in the mobile phone system. When obtaining information that the XR glasses have turned on the incoming call reminder function, the mobile phone setting module preferably sets the mobile phone incoming call reminder to be silent; when obtaining information that the XR glasses have turned off the incoming call reminder function, the mobile phone setting module will restore the previously saved incoming call reminder mode.

[0102] Mobile phone incoming call reminder module 115: The mobile phone performs the incoming call reminder function.

[0103] Specifically, the method for notifying extended reality information according to an embodiment of the present application is described below with examples of several scenarios, specifically:

[0104] Application scenario 1: The mobile phone and XR glasses are both nearby and can be used normally.

[0105] Preferably, the call reminder optimization switch in the glasses setting module of the XR glasses will be turned on to avoid the mobile phone and the XR glasses ringing at the same time when a call comes in.

[0106] Optionally, the low-battery call reminder switch in the glasses setting module of the XR glasses is turned on: when the user puts on the glasses, the glasses turn on the call reminder function, and the phone call reminder is muted. The user can see the call reminder at a glance in the XR glasses and will not miss the call. The phone also remains muted to avoid ringing at the same time and reduce interference to others. When the user takes off the XR glasses, the XR glasses turn off the call reminder function to reduce the power consumption of the XR glasses, and the phone restores the original call reminder so that the call will not be missed.

[0107] Optionally, the low-battery call reminder switch in the glasses setting module of the XR glasses is turned off: when the user wears the XR glasses and the battery level of the XR glasses is higher than the threshold, the XR glasses turn on the call reminder function, and the phone call reminder is muted. When the user wears the XR glasses but the battery level of the XR glasses is lower than the threshold, the XR glasses turn off the call reminder function to reduce the power consumption of the glasses, and the phone restores the original call reminder, so that calls will not be missed. When the user takes off the XR glasses, the XR glasses turn off the call reminder function to reduce the power consumption of the XR glasses, and the phone restores the original call reminder, so that calls will not be missed.

[0108] Application scenario 2: The user is wearing XR glasses, but the mobile phone is not with him, such as going out for a run without carrying the mobile phone.

[0109] Preferably, the call reminder optimization switch in the glasses setting module of the XR glasses will be turned on to avoid the mobile phone and the XR glasses ringing at the same time when a call comes in.

[0110] Preferably, the low-battery call reminder switch in the glasses setting module of the XR glasses is turned on. Since the mobile phone is not around, it is preferred that the XR glasses can also receive call reminders when the battery is low, so as not to miss calls. When the user wears the XR glasses, the XR glasses turn on the call reminder function, so that calls will not be missed, and the mobile phone call reminder is muted, which can avoid the ringing of the mobile phone that is not around to disturb others and improve personal privacy.

[0111] Application scenario three: The mobile phone is carried with you, but the XR glasses are not with you.

[0112] Preferably, the call reminder optimization switch in the glasses setting module of the XR glasses will be turned on. When the user takes off the XR glasses, the XR glasses will turn off the call reminder function to reduce the power consumption of the XR glasses, and notify the mobile phone to restore the original call reminder so that calls will not be missed.

[0113] Among them, the XR glasses of the embodiments of the present application include AR glasses, VR glasses, MR glasses, etc.

[0114] In summary, the extended reality call reminder scheme of the embodiment of the present application realizes that even if the extended reality device is in a worn state, a more refined conditional judgment is used to determine whether to push the call reminder to the extended reality device. This is different from the existing method of directly pushing the call reminder to the extended reality device as long as the extended reality device is in a worn state. The embodiment of the present application improves the rationality of the device call reminder through a comprehensive analysis of the initial switch state, device recognition results, etc.

[0115] In order to more clearly illustrate the extended reality information notification method of the embodiment of the present application, the following will describe the extended reality information notification method from the overall process. For details, please refer to Figure 4 , Figure 4: is another flowchart of the extended reality information notification method provided in an embodiment of the present application, wherein the extended reality information notification method specifically includes:

[0116] 121. Detection of receiving target information;

[0117] 122. Obtaining the wearing status of the extended reality device. If the wearing status indicates that the extended reality device is not worn, executing step 123; otherwise, executing step 124;

[0118] 123. Control the extended reality device to cancel the notification information corresponding to the output target information;

[0119] 124. Determine whether the target terminal device and the extended reality device are in the same application multi-terminal login scenario. If not, execute step 125. Otherwise, execute step 126.

[0120] 125. Control the extended reality device to output notification information corresponding to the target information;

[0121] 126. Determine whether the target terminal device is included in the real scene image of the extended reality device. If not, execute step 127. Otherwise, execute step 128.

[0122] 127. Controlling the extended reality device to output notification information corresponding to the target information, and controlling the target terminal device not to output the notification information;

[0123] 128. Calculate the distance between the target terminal device and the current extended reality device;

[0124] 129. Determine whether the distance is greater than or equal to a preset distance threshold. If so, execute step 130. Otherwise, execute step 131.

[0125] 130. Controlling the extended reality device to output notification information corresponding to the target information, and controlling the target terminal device not to output the notification information;

[0126] 131. Control the extended reality device not to output notification information corresponding to the target information, and control the target terminal device to output the notification information.

[0127] In summary, the embodiments of the present application utilize whether the real scene screen of the extended reality device contains the target terminal device, that is, analyzes whether the user can currently conveniently view notifications from the target terminal device to screen out devices that are more suitable for outputting notification information, thereby reducing the ineffective power consumption of the device and avoiding users from missing information, and outputting notification information to users in a timely, convenient, and user-friendly manner.

[0128] Among them, by combining the wearing status, power information, and personal information to comprehensively analyze which device outputs the notification information, the rationality of the notification information output is improved and the user experience is improved.

[0129] In order to better implement the extended reality information notification method of the present application, the present application also provides an extended reality information notification device based on the above extended reality information notification method. The meanings of the terms are the same as those in the above extended reality information notification method, and the specific implementation details can refer to the description in the method embodiment.

[0130] See also Figure 5 , Figure 5 : is a schematic diagram of the structure of an extended reality information notification device provided in an embodiment of the present application, wherein the extended reality information notification device is applied to an extended reality device, and the extended reality information notification device can be specifically as follows:

[0131] The identification module 201 is used to identify the target terminal device from the current real scene screen of the extended reality device if the target information is received, and obtain a device identification result;

[0132] The control module 202 is used to control the output of notification information for the target information according to the device identification result.

[0133] Optionally, in some embodiments of the present application, controlling the output of notification information for the target information according to the device recognition result includes:

[0134] If the device recognition result indicates that the real scene image does not include the target terminal device, outputting notification information for the target information;

[0135] If the device recognition result indicates that the target terminal device is included in the real scene image, the output of notification information for the target information is canceled, and the target terminal device is controlled to output the notification information.

[0136] Optionally, in some embodiments of the present application, if the target information is received, identifying the target terminal device from the current real scene screen of the extended reality device to obtain a device identification result includes:

[0137] If the target information is received, identifying the wearing state of the extended reality device;

[0138] If the wearing state indicates that the extended reality device is currently being worn, identifying a target terminal device from a current real scene screen of the extended reality device to obtain a device identification result;

[0139] If the wearing state indicates that the extended reality device is not currently being worn, then controlling the output of notification information for the target information according to the device recognition result includes:

[0140] The output of notification information for the target information is canceled, and the target terminal device is directly controlled to output notification information for the target information.

[0141] Optionally, in some embodiments of the present application, if the wearing state indicates that the extended reality device is currently being worn, identifying a target terminal device from a current real scene screen of the extended reality device to obtain a device identification result includes:

[0142] If the wearing state indicates that the extended reality device is currently being worn, identifying an application scenario of the extended reality device and the target terminal device;

[0143] If the application scenario is a multi-terminal login scenario for the same application, identifying the target terminal device from the current real scene screen of the extended reality device to obtain a device identification result;

[0144] If the application scenario is not a same-application multi-terminal login scenario, notification information for the target information is directly output.

[0145] Optionally, in some embodiments of the present application, controlling the output of notification information for the target information according to the device recognition result includes:

[0146] If the device recognition result indicates that the real scene image does not include the target terminal device, outputting notification information for the target information;

[0147] If the device recognition result indicates that the real scene image includes the target terminal device, obtaining the distance between the extended reality device and the target terminal device;

[0148] Furthermore, if the distance is less than a preset distance threshold, the output of notification information for the target information is canceled, and the target terminal device is controlled to output the notification information; if the distance is greater than or equal to the preset distance threshold, the notification information for the target information is output.

[0149] Optionally, in some embodiments of the present application, the step of identifying a target terminal device from a current real scene screen of the extended reality device to obtain a device identification result includes:

[0150] Inputting the scene image corresponding to the real scene picture into a target recognition model, and outputting a device recognition result for the target terminal device in the scene image through the target recognition model; and / or,

[0151] Based on the template image of the target terminal device, the scene image corresponding to the real scene picture is matched to obtain a matching result, and a device recognition result is generated according to the matching result.

[0152] Optionally, in some embodiments of the present application, controlling the output of notification information for the target information according to the device recognition result includes:

[0153] Obtaining the wearing status and power information of the extended reality device and the portable information of the target terminal device;

[0154] The output of notification information for the target information is controlled according to the device recognition result, the wearing state information, the power information and the carried information.

[0155] In the embodiment of the present application, if the identification module 201 receives the target information, it will identify the target terminal device from the current real scene screen of the extended reality device to obtain the device identification result. Then, the control module 202 controls the output of the notification information for the target information according to the device identification result.

[0156] Among them, after the extended reality device receives the target information, the embodiment of the present application controls the output of the notification information of the target information by identifying whether the real scene screen contains the target terminal device, thereby realizing the output control of the notification information of the target information. Whether the real scene screen contains the target terminal device indicates whether the user has the conditions to directly view the target information through the target terminal device, and based on this, the output of the notification information corresponding to the target information is controlled, thereby improving the rationality of the output of the notification information and reducing the power consumption of the device caused by unnecessary notifications.

[0157] In addition, the present application also provides an electronic device, such as Figure 6 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0158] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will appreciate that Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0159] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0160] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0161] The electronic device also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0162] The electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0163] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application program stored in the memory 302, thereby implementing the steps in any one of the augmented reality information notification methods provided in the embodiments of the present application.

[0164] If the extended reality device of the embodiment of the present application receives target information, it identifies the target terminal device from the current real scene screen of the extended reality device, obtains a device identification result, and controls the output of notification information for the target information based on the device identification result.

[0165] Among them, after the extended reality device receives the target information, it controls the output of the notification information of the target information by identifying whether the real scene screen contains the target terminal device, thereby realizing the output control of the notification information of the target information. Whether the real scene screen contains the target terminal device indicates whether the user has the conditions to directly view the target information through the target terminal device. Based on this, the output of the notification information corresponding to the target information is controlled, the rationality of the notification information output is improved, and the power consumption of the device caused by unnecessary notifications is reduced.

[0166] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0167] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0168] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any of the extended reality information notification methods provided in the present application.

[0169] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0170] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0171] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the extended reality information notification methods provided in the present application, the beneficial effects that can be achieved by any of the extended reality information notification methods provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0172] The above is a detailed introduction to an extended reality information notification method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for notifying extended reality information, characterized in that: Applied to an extended reality device, the method comprises: If the target information is received, the target terminal device is identified from the current real scene image of the extended reality device to obtain a device identification result; The output of notification information for the target information is controlled according to the device identification result.

2. The method for notifying extended reality information according to claim 1, characterized in that: The controlling the output of notification information for the target information according to the device identification result includes: If the device recognition result indicates that the real scene image does not include the target terminal device, outputting notification information for the target information; If the device recognition result indicates that the target terminal device is included in the real scene image, the output of notification information for the target information is canceled, and the target terminal device is controlled to output the notification information.

3. The method for notifying extended reality information according to claim 1, characterized in that: If the target information is received, the target terminal device is identified from the current real scene screen of the extended reality device to obtain a device identification result, including: If the target information is received, identifying the wearing state of the extended reality device; If the wearing state indicates that the extended reality device is currently being worn, identifying a target terminal device from a current real scene screen of the extended reality device to obtain a device identification result; If the wearing state indicates that the extended reality device is not currently being worn, then controlling the output of notification information for the target information according to the device recognition result includes: The output of notification information for the target information is canceled, and the target terminal device is directly controlled to output notification information for the target information.

4. The method for notifying extended reality information according to claim 3, characterized in that: If the wearing state indicates that the extended reality device is currently being worn, identifying a target terminal device from a current real scene screen of the extended reality device to obtain a device identification result includes: If the wearing state indicates that the extended reality device is currently being worn, identifying an application scenario of the extended reality device and the target terminal device; If the application scenario is a multi-terminal login scenario for the same application, identifying the target terminal device from the current real scene screen of the extended reality device to obtain a device identification result; If the application scenario is not a same-application multi-terminal login scenario, notification information for the target information is directly output.

5. The method for notifying extended reality information according to claim 1, characterized in that: The controlling the output of notification information for the target information according to the device identification result includes: If the device recognition result indicates that the real scene image does not include the target terminal device, outputting notification information for the target information; If the device recognition result indicates that the real scene image includes the target terminal device, obtaining the distance between the extended reality device and the target terminal device; Furthermore, if the distance is less than a preset distance threshold, the output of notification information for the target information is canceled, and the target terminal device is controlled to output the notification information; if the distance is greater than or equal to the preset distance threshold, the notification information for the target information is output.

6. The method for notifying extended reality information according to claim 1, characterized in that: The step of identifying a target terminal device from the current real scene screen of the extended reality device to obtain a device identification result includes: Inputting the scene image corresponding to the real scene picture into a target recognition model, and outputting a device recognition result for the target terminal device in the scene image through the target recognition model; and / or, Based on the template image of the target terminal device, the scene image corresponding to the real scene picture is matched to obtain a matching result, and a device recognition result is generated according to the matching result.

7. The method for notifying extended reality information according to claim 1, characterized in that: The controlling the output of notification information for the target information according to the device identification result includes: Obtaining the wearing status and power information of the extended reality device and the portable information of the target terminal device; The output of notification information for the target information is controlled according to the device recognition result, the wearing state information, the power information and the carried information.

8. An extended reality information notification device, characterized in that: Applied to an extended reality device, the device comprises: An identification module, configured to identify a target terminal device from the current real scene image of the extended reality device if target information is received, and obtain a device identification result; A control module is used to control the output of notification information for the target information according to the device identification result.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the extended reality information notification method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the extended reality information notification method according to any one of claims 1 to 7 are implemented.

Images