Display interaction method, head-mounted display device and computer readable storage medium

Abstract

The embodiment of the invention discloses a display interaction method, head-mounted display equipment and a computer readable storage medium. The method comprises the following steps: acquiring a notification sequence sent by a terminal device, generating a first broadcast instruction based on the notification sequence, and sending the first broadcast instruction to the terminal device, so that the terminal device synthesizes a first target notification into audio content and broadcasts the audio content of the first target notification, and in response to a notification display instruction sent by the terminal device when broadcasting the audio content, displaying the first target notification on the head-mounted display device. According to the embodiment of the invention, the situation that the terminal equipment broadcasts the audio content disorderly can be improved, the consistency between the currently broadcasted notification and the currently displayed notification is improved, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of data interaction technology, and in particular to a display interaction method, a head-mounted display device, and a computer-readable storage medium. Background Technology

[0002] In related technologies, when a head-mounted display device receives multiple notifications from a terminal device, it displays the notifications sequentially according to their order of appearance. Simultaneously, the terminal device converts the notifications into audio content for synchronous playback. However, the terminal device requires a certain amount of time to convert the notifications into audio content, resulting in a delay in audio playback. For example, the head-mounted display device may have finished displaying the current notification, but the terminal device may still be playing the corresponding audio content. Furthermore, while the terminal device is playing the audio content for the current notification, the head-mounted display device may display the next notification, leading to inconsistencies between the notification being played by the terminal device and the notification being displayed on the head-mounted display, thus affecting the user experience. Summary of the Invention

[0003] One objective of this application is to provide a display interaction method, a head-mounted display device, and a computer-readable storage medium to improve the situation in related technologies where the notification audio broadcast by the terminal device is inconsistent with the notification displayed by the head-mounted display device.

[0004] In a first aspect, embodiments of this application provide a display interaction method applied to a head-mounted display device, comprising: acquiring a notification sequence sent by a terminal device, the notification sequence including multiple notifications; generating a first broadcast instruction based on the notification sequence, the first broadcast instruction being used to instruct the terminal device to broadcast audio content corresponding to a first target notification, the first target notification being a notification in the notification sequence; sending the first broadcast instruction to the terminal device, causing the terminal device to synthesize the first target notification into audio content and broadcast the audio content of the first target notification; and displaying the first target notification on the head-mounted display device in response to a notification display instruction sent by the terminal device when broadcasting the audio content.

[0005] Optionally, generating the first broadcast instruction based on the notification sequence includes: selecting a notification that meets a preset filtering condition as a first target notification in the notification sequence; and generating the first broadcast instruction based on the first target notification.

[0006] Optionally, selecting a notification that meets preset filtering conditions as the first target notification in the notification sequence includes: selecting the notification that is in a state of waiting to be broadcast and has the earliest timestamp in the notification sequence; determining that the notification that is in a state of waiting to be broadcast and has the earliest timestamp meets the preset filtering conditions; and setting the notification that is in a state of waiting to be broadcast and has the earliest timestamp as the first target notification.

[0007] Optionally, displaying the first target notification on the head-mounted display device includes: determining a target display duration based on the text content of the first target notification; and displaying the first target notification on the head-mounted display device based on the target display duration.

[0008] Optionally, determining the target display duration based on the text content of the first target notification includes: determining the number of characters in the text content; and determining the target display duration based on the number of characters in the text content.

[0009] Optionally, determining the target display duration based on the number of words in the content includes: obtaining a preset display schedule, the display schedule including multiple word count ranges and a display duration corresponding to each word count range; traversing the word count range corresponding to the number of words in the display schedule; and determining the display duration of the word count range corresponding to the number of words in the content as the target display duration.

[0010] Optionally, in response to the notification display instruction sent by the terminal device when playing the audio content, after the head-mounted display device displays the first target notification, the method further includes: in response to the head-mounted display device ending the display of the first target notification and receiving a completion playback instruction sent by the terminal device, generating a second playback instruction, wherein the second playback instruction is used to instruct the terminal device to play the audio content corresponding to the second target notification, the second target notification being the next notification after the first target notification, and the completion playback instruction being generated after the terminal device has finished playing the audio content of the first target notification; sending the second playback instruction to the terminal device so that the terminal device synthesizes the second target notification into audio content and plays the audio content of the second target notification.

[0011] Optionally, the head-mounted display device is equipped with a touch module, and the display interaction method further includes: responding to a touch operation received by the touch module, stopping the display of the current notification and generating a pause broadcast command; sending the pause broadcast command to the terminal device so that the terminal device stops broadcasting the audio content corresponding to the current notification.

[0012] In a second aspect, embodiments of this application provide a head-mounted display device, including a memory and a processor. The memory is connected to the processor, and the processor is configured to execute one or more computer programs stored in the memory. When the processor executes the one or more computer programs, it causes the controller to implement the above-described display interaction method.

[0013] In a third aspect, embodiments of this application provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the aforementioned display interaction method.

[0014] The embodiments of this application can achieve the following technical effects: In this embodiment, upon receiving a notification sequence from a terminal device, a first broadcast instruction is generated based on the notification sequence. The first broadcast instruction instructs the terminal device to broadcast the audio content corresponding to the first target notification. Sending the first broadcast instruction to the terminal device enables the terminal device to synthesize the first target notification into audio content and broadcast the audio content of the first target notification. Therefore, the audio content to be broadcast by the terminal device is controlled by the head-mounted display device. The head-mounted display device controls the notification object broadcast by the terminal device, which can prevent the terminal device from broadcasting other content on its own before the head-mounted display device issues the first broadcast instruction. This provides a working basis for ensuring consistency between the display of the notification and the broadcast of the audio content.

[0015] When the terminal device synthesizes audio content and plays the audio content of the first target notification, the terminal device actively sends a notification display command to the head-mounted display device. The head-mounted display device then displays the first target notification based on the notification display command. This method can offset the time it takes for the terminal device to synthesize the audio content, ensuring that the first target notification displayed by the head-mounted display device is synchronized with and consistent with the audio content played by the terminal device, thus improving the user experience. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure of a head-mounted display device provided in an embodiment of this application; Figure 2 This application provides a schematic diagram of the circuit structure of a head-mounted display device according to an embodiment of the present application. Figure 3 This is a schematic diagram of the system architecture between the head-mounted display device and the cloud server provided in an embodiment of this application; Figure 4 This is a schematic diagram of the system architecture between the head-mounted display device and the terminal device provided in the embodiments of this application; Figure 5 A flowchart illustrating a display interaction method provided in an embodiment of this application; Figure 6 This is a schematic diagram illustrating the application scenario between a head-mounted display device and a terminal device provided in an embodiment of this application; Figure 7 This is a schematic diagram illustrating the communication between a head-mounted display device and a terminal device provided in an embodiment of this application. Figure 8 This is a schematic diagram of the structure of a display interaction device according to an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a controller provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0019] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0020] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0021] In the process of implementing the embodiments of this application, the inventors discovered a related technology, the process of which in implementing notification display of a head-mounted display device is as follows: Step 1: The user is already wearing the head-mounted display device. Proceed to Step 2. Step 2: The user uses a mobile phone and establishes a communication connection with the head-mounted display device via Bluetooth to proceed to Step 3; Step 3: The user grants the phone's notification list permission and allows notifications to be sent to the head-mounted display device, so that the head-mounted display device can perform audio playback based on the notifications, and proceeds to Step 4; Step 4: The phone receives multiple notifications, saves them locally, and sends them to the head-mounted display device via Bluetooth. Proceed to Step 5 and Step 6 respectively. Step 5: The phone synthesizes the first notification into audio content according to the time sequence of the notifications. After the synthesis of the audio content of the first notification is completed, the phone starts to play the audio content of the first notification and proceeds to step 8. Step 6: The head-mounted display device displays the first notification in chronological order, then proceeds to Step 7; Step 7: After the first notification is displayed on the head-mounted display for 5 seconds, the first notification disappears from the head-mounted display and the head-mounted display begins to display the second notification. At this time, the phone has not yet finished playing the audio content of the first notification, that is, the phone is still playing the audio content of the first notification. Proceed to Step 8. Step 8: When the user slides the touch module of the head-mounted display, the head-mounted display cancels the display of the second notification and controls the mobile phone to stop broadcasting the first notification, proceeding to step 9; Step 9: The head-mounted display shows the third notification, and the mobile phone synthesizes the audio content corresponding to the second notification and plays the audio content of the second notification in the manner described above.

[0022] In addition to the technical problems pointed out in the background, the related technology also has the following technical problems: When the notification currently being played by the terminal device is inconsistent with the notification currently being displayed by the head-mounted display device, the user's operation of the touch module cannot change the inconsistency between the currently played and currently displayed notifications. On the contrary, it is easy to deepen the misalignment between the currently played and currently displayed notifications. That is, it takes a certain amount of time to convert the notification into audio content. When the user applies multiple swipe operations to the touch module, the misalignment between the currently played and currently displayed notifications continues to accumulate, which in turn causes a large discrepancy in the sequence number between the currently played and currently displayed notifications. For example, the currently displayed notification is the 6th notification, while the currently played notification is the 3rd notification, thus causing information confusion for the user.

[0023] The embodiments of this application can improve the situation where terminal devices broadcast audio content out of order, improve the consistency between the currently broadcast notification and the currently displayed notification, and enhance the user experience.

[0024] The present application provides a head-mounted display device. The head-mounted display device includes augmented reality (AR) devices, virtual reality (VR) devices, visual aids and medical devices, motion and photography devices, etc. Among them, the augmented reality device can be AR glasses, the virtual reality device can be VR glasses, the visual aids and medical devices can be head-mounted electronic vision aids, and the motion and photography devices can be head-mounted projectors.

[0025] Please refer to the following: Figure 1 and Figure 2 The head-mounted display device 100 includes a frame 11, two lenses 12, two temples 13, a controller 14, an optical engine 15, a communication module 16, and a touch module 17.

[0026] Two lenses 12 are embedded in the frame 11. One of the two lenses 12 is a waveguide lens, and the other is a refractive lens or a plano lens; or both lenses 12 are waveguide lenses.

[0027] Two temples 13 are respectively located at opposite ends of the frame 11, one temple being the first temple 131 and the other temple being the second temple 132. For example, the first temple 131 is located on the right side of the frame 11, and the second temple 132 is located on the left side of the frame 11.

[0028] An optical engine 15 is housed within the first temple 131 and is positioned corresponding to the optical waveguide. Under the control of the controller 14, the optical engine 15 generates a display signal. This signal is transmitted through the optical waveguide and coupled out at a preset position on the waveguide, thus projecting onto the user's eye. For example, the controller 14 controls the optical engine 15 to generate a display signal corresponding to a notification. This display signal is transmitted through the optical waveguide, coupled out at a preset position on the waveguide, and then projected onto the user's eye, allowing the user to receive the text content corresponding to the notification.

[0029] The communication module 16 is communicatively connected to the controller 14 for communicating with external devices. Specifically, the controller 14 receives data sent by external devices through the communication module 16, or the external device sends data to the communication module 16, which then forwards the data to the controller 14. The communication module 16 may include Bluetooth, Wi-Fi, 6G, 5G, 4G, 3G, or 2G modules, etc.

[0030] Please see Figure 3The external device is a cloud server 300, and the head-mounted display device 100 communicates with the cloud server 300 via a communication module 16. The head-mounted display device 100 interacts with the cloud server 300 via the communication module 16. For example, the cloud server 300 can send control commands to the head-mounted display device 100, causing the head-mounted display device 100 to perform related operations; for example, the head-mounted display device 100 responds to a control command and enters a power-off state. For example, the head-mounted display device 100 can send data to the cloud server 300, causing the cloud server 300 to save the data in the cloud.

[0031] Please see Figure 4 The external device is the terminal device 400. The head-mounted display device 100 establishes a Bluetooth or Wi-Fi connection with the terminal device 400 through the communication module 16. The head-mounted display device 100 interacts with the terminal device 400 through the communication module 16. For example, the terminal device 400 can send terminal information to the head-mounted display device 100 for display. The terminal information includes text information or audio information, such as notifications, chat content, news information, and articles. The audio information includes music, video, and other multimedia content. The head-mounted display device 100 can send switching commands to the terminal device 400, causing the terminal device 400 to perform related operations based on the switching commands, such as controlling the terminal device 400 to play the next song. The terminal device 400 can be a mobile phone, desktop computer, tablet computer, smartwatch, etc.

[0032] A touch module 17 is disposed at the end of one of the first temple 131 and the second temple 132 and is electrically connected to the controller 14. It is used to detect touch operations input by the user. When the user needs to change the operating state of the head-mounted display device 100 and the terminal device 400, the user inputs a touch operation on the touch module 17, such as sliding on the touch module 17. The head-mounted display device 100 detects the touch operation, executes the control operation corresponding to the touch operation, and generates a working control command. This working control command is sent to the terminal device 400 via the communication module 16, causing the terminal device 400 to change its current operating state based on the working control command.

[0033] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 5 , Figure 5 A flowchart of a display interaction method provided in this application embodiment is shown below. This application embodiment implements the display interaction method through steps S51 to S54: Step S51: Obtain the notification sequence sent by the terminal device. The notification sequence includes multiple notifications.

[0034] A notification sequence is a collection of multiple notifications that a terminal device needs to send to a head-mounted display device. Notifications include long notifications and short notifications. Long notifications are those with a content length exceeding a preset character count threshold, while short notifications are those with a content length less than or equal to the preset character count threshold. The preset character count threshold is customized by the designer based on business needs; for example, the preset character count threshold is 120. Notifications can be WeChat messages, email reminders, calendar notifications, etc.

[0035] A notification includes a notification identifier, sender identification information, a receipt timestamp, text content, and a priority identifier. The notification identifier is a unique identification code for the notification. The sender identification information identifies the sender. The receipt timestamp is the time when the terminal device or head-mounted display device successfully received the notification. The text content is the text information carried in the notification. The priority identifier is a classification identifier used to define the importance of the notification. Priority identifiers include high priority, medium priority, and low priority. For example, work-related notifications have a high priority identifier, chat-related notifications have a medium priority identifier, and advertising-related notifications have a low priority identifier.

[0036] Please see Figure 6 The terminal device 400's notification list 41 receives multiple notifications, which are arranged in chronological order, such as... Figure 6 As shown, the first notification 42 is "Go home early for dinner tonight," and the second notification 43 is the company's team building arrangements for Saturday.

[0037] The terminal device 400 supports the installation of various applications (APPs), including an AR application 44 for interacting with the head-mounted display device. The user operates the terminal device 400 to grant the AR application permission to read the notification list 41. When the user clicks to open the AR application 44, the AR application 44 runs on the operating system of the terminal device 400, reads all notifications in the notification list 41, and packages all notifications in the notification list 41 into a notification sequence via the Bluetooth connection between the terminal device 400 and the head-mounted display device 100, sending the notification sequence to the head-mounted display device 100.

[0038] The ways in which head-mounted display devices acquire notification sequences sent by terminal devices include active acquisition and passive acquisition.

[0039] In some embodiments, obtaining a notification sequence sent by a terminal device includes the following steps: in response to the establishment of a Bluetooth connection between the head-mounted display device and the terminal device, a notification acquisition request is sent to the terminal device, so that the terminal device packages all notifications in the notification list into a notification sequence based on the notification acquisition request and returns the notification sequence to the head-mounted display device.

[0040] In other embodiments, obtaining the notification sequence sent by the terminal device includes the following steps: obtaining the notification sequence sent by the terminal device at a preset frequency, and storing the notification sequence locally on the head-mounted display device.

[0041] When the head-mounted display device receives a notification sequence from the terminal device, the head-mounted display device returns a reception success message to the terminal device. This message includes the notification identifier of each notification in the sequence. The terminal device parses the reception success message and checks whether the head-mounted display device has successfully received each notification. If it detects that the head-mounted display device has successfully received each notification, the terminal device enters the stage of preparing to send the next notification sequence. If it detects that the head-mounted display device has not successfully received a relevant notification, the terminal device resends the notification to the head-mounted display device until the head-mounted display device successfully receives the notification. Thus, this embodiment of the application ensures that the head-mounted display device can receive a complete and reliable notification sequence.

[0042] Step S52: Generate the first broadcast instruction based on the notification sequence.

[0043] The first broadcast instruction is used to instruct the terminal device to broadcast the audio content corresponding to the first target notification, which is a notification in a notification sequence. For example... Figure 6 As shown, the first target notification is either the first notification 42 or the second notification 43.

[0044] In this embodiment of the application, a first broadcast instruction is generated based on the notification sequence through steps S521 and S522, as detailed below: Step S521: Select the notification that meets the preset filtering conditions from the notification sequence as the first target notification.

[0045] The preset filtering criteria are the conditions for selecting the first target notification from the notification sequence. The components of the preset filtering criteria can be customized by the designer based on engineering experience.

[0046] In some embodiments, selecting a notification that meets preset filtering conditions as the first target notification in the notification sequence includes the following steps: determining the priority identifier of each notification in the notification sequence, obtaining notifications that are in the state of waiting to be broadcast and have a high priority identifier, determining that the notifications that are in the state of waiting to be broadcast and have a high priority identifier meet the preset filtering conditions, and setting the notifications that are in the state of waiting to be broadcast and have a high priority identifier as the first target notification.

[0047] Determining the priority identifier of each notification in the notification sequence includes the following steps: parsing the type of each notification in the notification sequence to obtain the notification type corresponding to the notification, and determining the priority identifier corresponding to the notification based on the notification type.

[0048] In some embodiments, type parsing of each notification in the notification sequence to obtain the notification type corresponding to the notification includes the following steps: obtaining type hint words, inputting the type hint words and the notification sequence into a pre-trained large language model, so that the large language model, guided by the type hint words, performs type parsing of each notification in the notification sequence to obtain the notification type corresponding to each notification.

[0049] In other embodiments, type parsing of each notification in the notification sequence to obtain the notification type corresponding to the notification includes the following steps: inputting the notification sequence into a pre-trained artificial intelligence model, so that the artificial intelligence model performs type parsing on each notification in the notification sequence to obtain the notification type corresponding to each notification.

[0050] In other embodiments, performing type parsing on each notification in the notification sequence to obtain the notification type corresponding to the notification includes the following steps: performing type parsing on each notification in the notification sequence based on a preset semantic parsing algorithm to obtain the notification type corresponding to each notification.

[0051] Determining the priority identifier corresponding to a notification based on the notification type includes the following steps: obtaining a priority table, which includes multiple notification types and a priority identifier corresponding to each notification type, and traversing the priority table to find the priority identifier corresponding to the notification type.

[0052] Please refer to Table 1, the priority table is shown in Table 1: Table 1

[0053] As shown in Table 1, the first notification 42 is a chat notification, and in this embodiment, the priority identifier of the first notification 42 is set to medium priority. The second notification 43 is a work notification, and in this embodiment, the priority identifier of the second notification 43 is set to high priority.

[0054] The "pending broadcast" status indicates that the terminal device has not yet synchronously broadcast the notification. For example, when the first notification 42 has been synchronously displayed by the head-mounted display device and the audio content of the first notification 42 has been synchronously broadcast by the terminal device 400, the broadcast status of the first notification 42 is updated to the "broadcast" status. However, the second notification 43 has not yet been synchronously displayed by the head-mounted display device and the audio content of the second notification 43 has not yet been synchronously broadcast by the terminal device 400. Therefore, the broadcast status of the second notification 43 is the "pending broadcast" status.

[0055] Since the broadcast status of the second notification 43 is pending broadcast and its priority identifier is high priority, this embodiment sets the second notification 43 as the first target notification.

[0056] This application embodiment can prioritize the display of high-priority notifications on the head-mounted display device based on the priority of each notification, and control the terminal device to broadcast the audio content of high-priority notifications in a limited manner. This allows users to promptly grasp the information conveyed by high-priority notifications, avoids users missing high-priority notifications, and improves the user experience.

[0057] In other embodiments, selecting a notification that meets preset filtering conditions as the first target notification in the notification sequence includes the following steps: selecting the notification that is in a state to be broadcast and has the earliest timestamp in the notification sequence, determining that the notification that is in a state to be broadcast and has the earliest timestamp meets the preset filtering conditions, and setting the notification that is in a state to be broadcast and has the earliest timestamp as the first target notification.

[0058] Selecting the notification with the earliest timestamp that is in the pending broadcast state in the notification sequence includes the following steps: performing a traversal operation on the notification sequence to put the notifications in the pending broadcast state into a preset queue, sorting all the notifications in the preset queue according to a preset time order to obtain the sorting result, and extracting the notification with the earliest timestamp from the sorting result as the first target notification.

[0059] The preset time order can be either from earliest to latest or from latest to earliest. When the preset time order is from earliest to latest, the notification ranked first in the sorting results is extracted as the first target notification. When the preset time order is from latest to earliest, the notification ranked last in the sorting results is extracted as the first target notification.

[0060] In this embodiment, the head-mounted display device is controlled to display the notifications in chronological order, and the terminal device is controlled to broadcast them, so that the user can learn about the content of each notification in chronological order.

[0061] Step S522: Generate a first broadcast instruction based on the first target notification.

[0062] In this embodiment of the application, a notification identifier corresponding to the first target notification is obtained, and the target notification identifier corresponding to the first target notification is encapsulated in a first broadcast instruction.

[0063] In this embodiment, a first broadcast instruction is generated based on a first target notification. The first broadcast instruction is used to instruct the terminal device to broadcast the audio content corresponding to the first target notification. Therefore, the audio content to be broadcast by the terminal device is controlled by the head-mounted display device. The head-mounted display device controls the notification object broadcast by the terminal device, which can prevent the terminal device from broadcasting other content on its own before the head-mounted display device issues the first broadcast instruction. This improves the situation where the terminal device broadcasts audio content in an unordered manner and provides a working basis for ensuring consistency between the display of the notification and the broadcast of the audio content.

[0064] Step S53: Send a first broadcast instruction to the terminal device so that the terminal device synthesizes the first target notification into audio content and broadcasts the audio content of the first target notification.

[0065] When the head-mounted display sends the first broadcast command to the terminal device, the terminal device parses the target notification identifier from the first broadcast command. Next, the terminal device searches its local notification list for the notification corresponding to the target notification identifier, sets the notification corresponding to the target notification identifier as the first target notification, and performs audio synthesis processing on the first target notification to obtain the corresponding audio content. After the terminal device synthesizes the audio content of the first target notification, it automatically broadcasts the audio content of the first target notification and sends a notification display command to the head-mounted display device.

[0066] The head-mounted display device provided in this application embodiment does not encapsulate the first target notification in the first broadcast instruction, but only encapsulates the lightweight target notification identifier in the first broadcast instruction. This makes the data volume of the first broadcast instruction small, without occupying a large bandwidth, and can be quickly sent to the terminal device. This avoids the situation where the data volume of the first broadcast instruction is too large and it is easy to fail to send, thereby improving the success rate of the terminal device receiving the first broadcast instruction and improving the reliability of the terminal device synchronously broadcasting the audio content of the first target notification.

[0067] Step S54: In response to the notification display instruction sent by the terminal device when playing audio content, the first target notification is displayed on the head-mounted display device.

[0068] The notification display command is the instruction that triggers the head-mounted display device to display the first target notification. The terminal device actively sends the notification display command to the head-mounted display device only when it synthesizes and plays the audio content of the first target notification. The head-mounted display device then displays the first target notification based on the notification display command. This method can offset the time it takes for the terminal device to synthesize the audio content, ensuring that the first target notification displayed by the head-mounted display device is synchronized with and consistent with the audio content played by the terminal device, thus improving the user experience.

[0069] In this embodiment of the application, a first target notification is displayed on the head-mounted display device through steps S541 and S542, as detailed below: Step S541: Determine the target display duration based on the text content of the first target notification.

[0070] Step S542: Display the first target notification on the head-mounted display device based on the target display duration.

[0071] In step S541, the text content is the collection of text in the first target notification, and the target display duration is the duration for which the head-mounted display device displays the first target notification. As mentioned earlier, the first target notification can be a long notification or a short notification, and the text content lengths of long and short notifications differ. Short notifications can be read quickly, while long notifications require a longer viewing time. Related technologies control the display of notifications on the head-mounted display device according to a fixed display duration. When the notification is a short notification, the user has already finished reading the notification on the head-mounted display device, but the notification still remains on the head-mounted display device, occupying the display space of the head-mounted display device. When the notification is a long notification, the user has not finished reading the notification on the head-mounted display device, but the notification has automatically disappeared, resulting in the user not being able to obtain complete information about the notification. The embodiments of this application adaptively adjust the display duration according to the text content of the first target notification. The adjusted target display duration can both ensure that the user can reliably and completely obtain complete information about the first target notification and avoid the first target notification from lingering meaninglessly on the head-mounted display device.

[0072] Determining the target display duration based on the text content of the first target notification includes the following steps: determining the number of words in the text content, and determining the target display duration based on the number of words.

[0073] The word count refers to the number of valid characters in the text content. Valid characters are those that can convey information to the user, while invalid characters are those that cannot. For example, in the first target notification: "Please arrange your time accordingly, and feel free to communicate if you have any questions," neither the comma (,) nor the period (.") conveys any information to the user. Therefore, both the comma (,) and period (.") are invalid characters. All other characters in the first target notification are valid characters except for the comma (,) and period (.").

[0074] Determining the word count of the text content includes the following steps: removing invalid characters from the text content to obtain the removed text content, determining the total number of valid characters contained in the removed text content, and setting the total number of characters as the content word count.

[0075] In some embodiments, determining the target display duration based on the number of words in the content includes the following steps: determining the reading duration per word, where the reading duration per word is the reading duration of one valid character, and determining the target display duration based on the reading duration per word and the number of words in the content.

[0076] Determining the target display duration based on single-character reading time and content word count includes the following steps: obtaining the preset base display duration, multiplying the single-character reading time and content word count to obtain the effective display duration, and adding the base display duration and the effective display duration to obtain the target display duration.

[0077] In this embodiment, a basic display duration is used as the minimum guaranteed duration, and then an effective display duration is added on top of that. The resulting target display duration can ensure that the user can obtain complete information about the first target notification, while also avoiding meaningless lingering of the first target notification on the head-mounted display device.

[0078] In other embodiments, determining the target display duration based on the number of words in the content includes the following steps: obtaining a preset display schedule, which includes multiple word count ranges and a display duration corresponding to each word count range; traversing the word count ranges corresponding to the number of words in the content within the display schedule; and determining the display duration of the word count range corresponding to the number of words in the content as the target display duration.

[0079] Please refer to Table 2, which shows the timeline as follows: Table 2

[0080] As shown in Table 2, the display duration corresponding to the character count range [1, 30] is the first display duration, the display duration corresponding to the character count range [31, 50] is the second display duration, the display duration corresponding to the character count range [51, 100] is the third display duration, and the display duration corresponding to the character count range (100, +∞) is the fourth display duration. The order of first display duration < second display duration < third display duration < fourth display duration. The first, second, third, and fourth display durations are customized by the designer based on engineering experience. For example, the first, second, third, and fourth display durations are 3s, 5s, 10s, and 15s, respectively. When the content of the first target notification has 20 characters, the target display duration is 3s. When the content of the first target notification has 40 characters, the target display duration is 5s. When the content of the first target notification has 80 characters, the target display duration is 10s.

[0081] In step S542, for example, when the number of characters in the content of the first target notification is less than or equal to 30, the target display duration of the first target notification is matched to a first duration, that is, the first target notification is displayed on the head-mounted display device 100 for 3 seconds, which can prevent the first target notification from occupying too much display screen of the head-mounted display device 100. When the number of characters in the content of the first target notification is greater than 30 and less than or equal to 50, the target display duration of the first target notification is matched to a second duration, that is, the first target notification is displayed on the head-mounted display device 100 for 5 seconds, which meets the need for quick reading of the notification. When the number of characters in the content of the first target notification is greater than 30 and less than or equal to 50, the target display duration of the first target notification is matched to a third duration, that is, the first target notification is displayed on the head-mounted display device 100 for 10 seconds, ensuring that the user has enough time to read the complete content of the notification. When the number of characters in the content of the first target notification is greater than 100, the target display duration of the first target notification is matched to a fourth duration, that is, the first target notification is displayed on the head-mounted display device 100 for 15 seconds, ensuring that the user has enough time to read the complete content of the notification.

[0082] The embodiments of this application adaptively adjust the target display duration based on the number of characters in the first target notification, which can directly reduce the display time of the head-mounted display device, reduce the overall power consumption of the head-mounted display device, and also avoid the user repeatedly triggering the display notification operation due to insufficient target display duration.

[0083] In some embodiments, after the head-mounted display device displays the first target notification, the display interaction method further includes: in response to the head-mounted display device ending the display of the first target notification and receiving a completion broadcast instruction sent by the terminal device, generating a second broadcast instruction, sending the second broadcast instruction to the terminal device, so that the terminal device synthesizes the second target notification into audio content and broadcasts the audio content of the second target notification.

[0084] The completion broadcast instruction is generated after the terminal device broadcasts the audio content of the first target notification. When the terminal device finishes broadcasting the audio content of the first target notification, it generates a completion broadcast instruction and sends it to the head-mounted display device, causing the head-mounted display device to generate a second broadcast instruction. The second broadcast instruction instructs the terminal device to broadcast the audio content corresponding to the second target notification, which is the next notification after the first target notification.

[0085] For example, the second notification 43 is the first target notification. After the head-mounted display device stops displaying the second notification 43 and the terminal device stops broadcasting the second notification 43, the terminal device sends a completion broadcast instruction to the head-mounted display device. The head-mounted display device responds to the completion broadcast instruction, selects the third notification as the second target notification, generates a second broadcast instruction based on the third notification, and sends the second broadcast instruction to the terminal device. This enables the terminal device to synthesize the audio content of the third notification based on the second broadcast instruction and automatically broadcast the audio content of the third notification.

[0086] After displaying the first target notification, the head-mounted display device needs to receive a completion playback command from the terminal device before it can proceed to the display processing flow of the second target notification. Therefore, the head-mounted display device's handling of the second target notification is controlled by the terminal device, and the terminal device only sends the completion playback command after completing the playback of the audio content of the first target notification. This avoids the head-mounted display device entering the display processing flow of the second target notification out of order, ensuring that the display and playback of the first target notification are synchronized before entering the display and playback processing flow of the second target notification. This guarantees that the display and playback of the same notification are always consistent and aligned, which is beneficial to improving the user experience.

[0087] In some embodiments, the head-mounted display device is provided with a touch module, and the display interaction method further includes: responding to a touch operation received by the touch module, stopping the display of the current notification and generating a pause broadcast command, and sending the pause broadcast command to the terminal device so that the terminal device stops broadcasting the audio content corresponding to the current notification.

[0088] In response to a touch operation received by the touch module, stopping the display of the current notification and generating a pause broadcast command includes the following steps: in response to a touch operation received by the touch module, determining the touch trajectory of the touch operation; in response to the touch trajectory matching a preset trajectory, stopping the display of the current notification and generating a pause broadcast command.

[0089] The preset trajectory is customized by the designer based on engineering experience. For example, the preset trajectory is a trajectory applied to the surface of the touch module in a first direction, such as left or right.

[0090] The current notification can be the first target notification, the second target notification, or any notification in the notification sequence. The pause broadcast command is an instruction to the terminal device to stop broadcasting the current notification.

[0091] For example, when a user swipes left on the surface of the touch module, the head-mounted display device detects that the user's touch trajectory on the surface of the touch module matches a preset trajectory, stops displaying the current notification, and generates a pause broadcast command. This embodiment of the application, while maintaining consistency between notification display and notification broadcast, allows for flexible switching of notifications between the head-mounted display device and the terminal device via the touch module, which improves the user's flexibility in switching notifications.

[0092] To illustrate in detail the display interaction method provided in the embodiments of this application, the embodiments of this application are combined with... Figure 7 This will be explained in detail below: S71, terminal device 400 sends a notification sequence to head-mounted display device 100; S72, the head-mounted display device 100 generates a first broadcast instruction based on the notification sequence; S73, the head-mounted display device 100 sends a first broadcast command to the terminal device 400; S74, the terminal device 400 synthesizes and broadcasts the audio content of the first target notification based on the first broadcast instruction; S75, terminal device 400 sends a notification display command to head-mounted display device 100; S76, The head-mounted display device 100 displays a first target notification based on a notification display instruction; S77, Head-mounted display device 100 ends display of first target notification; S78, terminal device 400 sends a completion broadcast command to head-mounted display device 100; S79, the head-mounted display device 100 returns to step S72 based on the completed broadcast instruction.

[0093] S710, the head-mounted display device 100 stops displaying the current notification and generates a pause broadcast command based on touch operation.

[0094] S711, the head-mounted display device 100 sends a pause broadcast command to the terminal device 400.

[0095] S712, terminal device 400 stops broadcasting the audio content corresponding to the current notification based on the pause broadcast command.

[0096] In summary, the embodiments of this application have at least the following technical effects: 1) The embodiments of this application can improve the situation of disordered audio content broadcast by terminal devices, improve the consistency between the currently broadcast notification and the currently displayed notification, and enhance the user experience.

[0097] 2) The embodiments of this application can improve the situation of disordered display of notifications in head-mounted display devices, improve the consistency between the currently broadcast notification and the currently displayed notification, and enhance the user experience.

[0098] 3) In this embodiment of the application, the head-mounted display device is controlled to display the notifications in chronological order, and the terminal device is controlled to broadcast them, so that the user can understand the content of each notification in chronological order.

[0099] 4) According to the text content of the first target notification, the display duration is adaptively adjusted. The adjusted target display duration can ensure that the user can reliably and completely obtain complete information about the first target notification, and can also avoid the first target notification from lingering meaninglessly on the head-mounted display device.

[0100] 5) Under the premise that the notification display and notification broadcast are consistent, the present application embodiment controls the head-mounted display device and the terminal device to flexibly switch notifications through the touch module, which is beneficial to improve the flexibility of users switching notifications.

[0101] It should be noted that in the above embodiments, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0102] As another aspect of the embodiments of this application, this application provides a display interaction device. The display interaction device can be a software module, which includes several instructions stored in a memory. A processor can access the memory, call the instructions, and execute them to complete the head-mounted display method described in the various embodiments above.

[0103] In some embodiments, the display interaction device can also be constructed from hardware devices. For example, the display interaction device can be constructed from one or more chips, which can work together to complete the head-mounted display method described in the various embodiments above. As another example, the display interaction device can also be constructed from various logic devices, such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, ARM (Acorn RISC Machine) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0104] Please see Figure 8 The interactive display device 800 includes a notification acquisition module 81, an instruction generation module 82, an instruction sending module 83, and a notification display module 84.

[0105] The notification acquisition module 81 is used to acquire the notification sequence sent by the terminal device. The notification sequence includes multiple notification instructions. The generation module 82 is used to generate a first broadcast instruction based on the notification sequence. The first broadcast instruction is used to instruct the terminal device to broadcast the audio content corresponding to the first target notification. The first target notification is one of the notifications in the notification sequence. The notification instruction sending module 83 is used to send the first broadcast instruction to the terminal device so that the terminal device can synthesize the first target notification into audio content and broadcast the audio content of the first target notification. The notification display module 84 is used to respond to the notification display instruction sent by the terminal device when broadcasting the audio content and display the first target notification on the head-mounted display device.

[0106] In some embodiments, the instruction generation module 82 is specifically used to: select a notification that meets a preset filtering condition from the notification sequence as a first target notification, and generate a first broadcast instruction based on the first target notification.

[0107] In some embodiments, the instruction generation module 82 is specifically used to: select the notification in the notification sequence that is in the state of waiting to be broadcast and has the earliest timestamp; determine that the notification in the state of waiting to be broadcast and has the earliest timestamp meets the preset filtering conditions; and set the notification in the state of waiting to be broadcast and has the earliest timestamp as the first target notification.

[0108] In some embodiments, the notification display module 84 is specifically configured to: determine a target display duration based on the text content of the first target notification, and display the first target notification on a head-mounted display device based on the target display duration.

[0109] In some embodiments, the notification display module 84 is specifically used to: determine the number of characters in the text content and determine the target display duration based on the number of characters.

[0110] In some embodiments, the notification display module 84 is specifically used to: obtain a preset display schedule, the display schedule including multiple word count ranges and a display duration corresponding to each word count range; traverse the word count range corresponding to the content word count within the display schedule; and determine the display duration of the word count range corresponding to the content word count as the target display duration.

[0111] In some embodiments, the notification display module 84 is specifically configured to: generate a second broadcast instruction in response to the head-mounted display device ending the display of the first target notification and receiving a completion broadcast instruction sent by the terminal device, wherein the second broadcast instruction is used to instruct the terminal device to broadcast the audio content corresponding to the second target notification, the second target notification being the next notification after the first target notification, the completion broadcast instruction being generated by the terminal device after broadcasting the audio content of the first target notification, and sending the second broadcast instruction to the terminal device so that the terminal device synthesizes the second target notification into audio content and broadcasts the audio content of the second target notification.

[0112] The head-mounted display device is equipped with a touch module. The notification display module 84 is specifically used to: respond to the touch operation received by the touch module, stop displaying the current notification and generate a pause broadcast command to send a pause broadcast command to the terminal device so that the terminal device stops broadcasting the audio content corresponding to the current notification.

[0113] As another aspect of this application, this application provides a controller. See also... Figure 9 , Figure 9 This is a schematic diagram of a controller provided in an embodiment of this application. The controller 14 includes one or more processors 141 and a memory 142. The memory 142 is connected to one or more processors 141, for example, via a bus.

[0114] Processor 141 is configured to support controller 14 in performing the corresponding functions in the methods described in the above method embodiments. Processor 141 may be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0115] Memory 142 is used to store program code, etc. Memory 142 may include volatile memory (VM), such as random access memory (RAM); memory 142 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 142 may also include combinations of the above types of memory 142.

[0116] The memory 142 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the head-mounted display method in the embodiments of this application. The processor 91 executes various functional applications and data processing of the head-mounted display method and head-mounted display device by running the non-volatile software programs, instructions, and modules stored in the memory 142, thereby realizing the functions of the head-mounted display method and the various modules or units of the head-mounted display device provided in the above method embodiments.

[0117] The memory 142 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function. The data storage area may store data created based on the use of the head-mounted display device. In some embodiments, the memory 142 may optionally include remotely located memories 142 relative to the processor 141, which can be connected to the head-mounted display device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0118] The one or more modules are stored in the memory 142. When executed by the one or more processors 141, they perform the head-mounted display method in any of the above method embodiments. For example, they perform the method steps described in the above method embodiments to realize the functions of the modules described in the above device embodiments.

[0119] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a controller, cause the controller to perform the method described in the foregoing embodiments.

[0120] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0121] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A display interaction method applied to a head-mounted display device, characterized in that, include: Obtain a notification sequence sent by the terminal device, wherein the notification sequence includes multiple notifications; A first broadcast instruction is generated based on the notification sequence. The first broadcast instruction is used to instruct the terminal device to broadcast the audio content corresponding to the first target notification. The first target notification is a notification in the notification sequence. Send the first broadcast instruction to the terminal device so that the terminal device synthesizes the first target notification into audio content and broadcasts the audio content of the first target notification; In response to a notification display instruction sent by the terminal device when playing the audio content, the first target notification is displayed on the head-mounted display device.

2. The display interaction method according to claim 1, characterized in that, The step of generating the first broadcast instruction based on the notification sequence includes: In the notification sequence, notifications that meet the preset filtering conditions are selected as the first target notification; A first broadcast instruction is generated based on the first target notification.

3. The display interaction method according to claim 2, characterized in that, The step of selecting notifications that meet preset filtering conditions as the first target notification in the notification sequence includes: Select the notification that is in the pending broadcast state and has the earliest timestamp from the notification sequence; The notification that is in the pending broadcast state and has the earliest timestamp meets the preset filtering criteria; Set the notification that is in the pending broadcast state and has the earliest timestamp as the first target notification.

4. The display interaction method according to claim 1, characterized in that, The step of displaying the first target notification on the head-mounted display device includes: The target display duration is determined based on the text content of the first target notification; The first target notification is displayed on the head-mounted display device based on the target display duration.

5. The display interaction method according to claim 4, characterized in that, Determining the target display duration based on the text content of the first target notification includes: Determine the number of characters in the text content; The target display duration is determined based on the number of words in the content.

6. The display interaction method according to claim 5, characterized in that, The step of determining the target display duration based on the number of words in the content includes: Obtain a preset display schedule, which includes multiple character count ranges and a display duration corresponding to each character count range; Traverse the range of word counts corresponding to the word count of the content within the displayed time schedule; The target display duration is determined by the display duration of the range of characters corresponding to the number of characters in the content.

7. The display interaction method according to claim 1, characterized in that, In response to the notification display instruction sent by the terminal device when playing the audio content, after the head-mounted display device displays the first target notification, the method further includes: In response to the head-mounted display device ending the display of the first target notification and receiving a completion broadcast instruction sent by the terminal device, a second broadcast instruction is generated, wherein the second broadcast instruction is used to instruct the terminal device to broadcast the audio content corresponding to the second target notification, the second target notification being the next notification after the first target notification, and the completion broadcast instruction is generated after the terminal device has finished broadcasting the audio content of the first target notification; The second broadcast instruction is sent to the terminal device so that the terminal device synthesizes the second target notification into audio content and broadcasts the audio content of the second target notification.

8. The display interaction method according to any one of claims 1-7, characterized in that, The head-mounted display device is equipped with a touch module, and the display interaction method further includes: In response to the touch operation received by the touch module, the current notification is stopped from being displayed and a pause broadcast command is generated; Send the pause broadcast command to the terminal device so that the terminal device stops broadcasting the audio content corresponding to the current notification.

9. A head-mounted display device, characterized in that, The device includes a memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, and the processor, when executing the one or more computer programs, causing the controller to implement the display interaction method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the display interaction method as described in any one of claims 1-8.

Images