Display method, device, system and electronic device for head-mounted display device

Abstract

The application provides a display method, device, system and electronic equipment for a head-mounted display device. The head-mounted display device comprises a first processing unit and a second processing unit. The display method comprises: in response to receiving a target operation instruction of a user on the head-mounted display device, controlling the first processing unit and the second processing unit to run in parallel, and calling first image data stored locally, processing the first image data through the first processing unit to obtain a first image; obtaining second image data, processing the second image data through the second processing unit to obtain a second image; performing synthesis processing on the first image and the second image to obtain a third image, wherein the third image can display at least part of the first image and at least part of the second image; and displaying the third image on a display unit of the head-mounted display device.

Description

Technical Field

[0001] This application relates to the field of head-mounted display device technology, and in particular to a display method, apparatus, system and electronic device for head-mounted display devices. Background Technology

[0002] Head-mounted displays (HUDs) create images in the user's eyes by sending optical signals, enabling various display effects such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). With advancements in modern technology, especially the increasing maturity of HUD technology, its applications have become more widespread. To meet diverse user needs in different usage scenarios, HUDs can offer multiple display and interaction methods, thus at least partially satisfying user requirements. Summary of the Invention

[0003] This application provides a display method, apparatus, system, and electronic device for head-mounted display devices.

[0004] According to a first aspect of the embodiments of this application, a display method for a head-mounted display device is provided. The head-mounted display device includes a first processing unit and a second processing unit. The method includes: responding to receiving a target operation command from a user to the head-mounted display device, controlling the first processing unit and the second processing unit to operate in parallel; calling locally stored first image data, processing the first image data through the first processing unit to obtain a first image; acquiring second image data, processing the second image data through the second processing unit to obtain a second image; performing composite processing on the first image and the second image to obtain a third image, wherein the third image can display at least a portion of the first image and at least a portion of the second image; and displaying the third image on a display unit of the head-mounted display device.

[0005] According to a second aspect of the embodiments of this application, a display device for a head-mounted display device is provided, comprising: a first processing module, configured to, in response to receiving a target operation command from a user for the head-mounted display device, control a first processing unit and a second processing unit to operate in parallel, and call locally stored first image data, process the first image data through the first processing unit to obtain a first image; a second processing module, configured to acquire second image data, process the second image data through the second processing unit to obtain a second image; a compositing processing module, configured to composite the first image and the second image to obtain a third image, wherein the third image is capable of displaying at least a portion of the first image and at least a portion of the second image; and a display module, configured to display the third image on a display unit of the head-mounted display device.

[0006] According to a third aspect of the embodiments of this application, a display system is provided, comprising: a first processing unit, configured to, in response to receiving a target operation command from a user for a head-mounted display device, process locally stored first image data to obtain a first image; a second processing unit, configured to, in response to acquiring second image data, operate in parallel with the first processing unit to process the second image data to obtain a second image; and a display unit, configured to display a third image synthesized from the first image and the second image, wherein the third image is capable of displaying at least a portion of the first image and at least a portion of the second image.

[0007] According to a fourth aspect of the embodiments of this application, an electronic device is provided, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store a computer program; and the processor is used to execute the method of any one of the first aspects by running the computer program stored in the memory.

[0008] According to a fifth aspect of the embodiments of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method as described in any of the first aspects.

[0009] According to a sixth aspect of the embodiments of this application, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the method as described in any of the first aspects. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0011] Figure 1A This is a schematic diagram of some optional display systems in the embodiments of this application.

[0012] Figure 1B This is a schematic diagram of some optional head-mounted display devices and host devices in the embodiments of this application.

[0013] Figure 2 This is a flowchart illustrating some optional display methods for head-mounted display devices in embodiments of this application.

[0014] Figure 3 This is a schematic diagram of combining the first and second images to obtain the third image.

[0015] Figure 4 An optional flowchart for combining the first and second images to obtain a third image.

[0016] Figure 5 This is a schematic diagram of an optional overall process of the display method according to an embodiment of this application.

[0017] Figure 6 This is a structural block diagram of some optional display devices for head-mounted display devices in the embodiments of this application.

[0018] Figure 7 This is a structural block diagram of some optional electronic devices in the embodiments of this application.

[0019] Explanation of reference numerals in the attached figures:

[0020] 100. Display system; 102. First processing unit; 104. Second processing unit; 106. Display unit; 200. Head-mounted display device; 300. Host terminal; 600. Display device for head-mounted display device; 602. First processing module; 604. Second processing module; 606. Synthesis processing module; 608. Display module; 700. Electronic device; 702. Processor; 704. Communication interface; 706. Memory; 708. Communication bus; 710. Computer program. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of this application. It should be understood that the various steps described in the method implementation of this disclosure can be performed in different orders and / or in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0022] In some alternative embodiments, head-mounted display devices often require displaying other images simultaneously with a given image. Such a scenario could be, for example, displaying a settings screen for configuring parameters of the head-mounted display device while it is displaying a source image. Optionally, the head-mounted display device can run a companion application on a connected host device, which can then generate the source image and settings screen and send them to the head-mounted display device for display.

[0023] This application provides an optional technical solution that can improve the convenience and efficiency of displaying multiple images simultaneously on a head-mounted display device, thereby enhancing the user experience.

[0024] To better illustrate the display solution for head-mounted display devices in this application, an exemplary description is provided below, using a display system suitable for this solution as an example. Figure 1A As shown, the display system 100 may include a first processing unit 102, a second processing unit 104, and a display unit 106. The display system 100 can be applied in a head-mounted display device 200. The head-mounted display device 200 may include, but is not limited to, AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, etc.

[0025] In some optional embodiments, the first processing unit 102 is used to process the first image data stored locally in response to receiving a target operation command from a user on the head-mounted display device 200, so as to obtain a first image.

[0026] Optionally, the first image data can be image data of the parameter setting interface stored locally. The first image can be an image of the parameter setting interface obtained by processing the first image data and then used for display on the screen. To improve the user experience of the head-mounted display device 200, the first image data can be processed by an appropriate algorithm to obtain the first image before being used for subsequent display on the screen, avoiding direct display of the locally stored first image data. Here, the processing method for the first image data is not specifically limited and can be selected as needed.

[0027] In some optional embodiments, the target operation instruction can be any type of operation instruction on the head-mounted display device 200. For example, it may include, but is not limited to, at least one of button operation instructions, remote control operation instructions, voice operation instructions, gesture operation instructions, and remote operation instructions. In some feasible implementations, physical buttons can be installed on the head-mounted display device 200, allowing the user to operate according to a pre-defined method, such as double-clicking the button, thereby invoking the target operation instruction.

[0028] In some alternative embodiments, the second processing unit 104 may be used to process the second image data in parallel with the first processing unit 102 in response to the acquisition of the second image data, thereby obtaining the second image.

[0029] Optionally, the second image data can be source image data for display on the head-mounted display device 200 (e.g., it can be independent image data for display, or video frame image data from a video being played). The second image can be an image obtained by rendering the second image data for display on the screen. As an example, the second image can be a video frame image of the video currently being played on the head-mounted display device 200. To meet the user experience of using the head-mounted display device 200, the second image data can be processed by a suitable algorithm to obtain the second image before being used for subsequent display on the screen. Here, the processing method for the second image data is not specifically limited and can be selected as needed.

[0030] It is understood that the first processing unit 102 and the second processing unit 104 can process different image data independently. Optionally, the first processing unit 102 and the second processing unit 104 can be two different processing units in the same processing chip. Alternatively, the first processing unit 102 and the second processing unit 104 can also be processing units in different processing chips. Optionally, the first processing unit 102 can be, but is not limited to, a CPU (Central Processing Unit), an MCU (Microcontroller Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), etc. Here, the first processing unit 102 can be understood as a CPU as an example. Optionally, the second processing unit 104 can be an image processing unit for processing the second image data, which can facilitate the improvement of image data processing speed.

[0031] In some optional embodiments, the display unit 106 is used to display a third image obtained by combining the first and second images, etc. Here, the third image can display at least a portion of the first image and at least a portion of the second image. As an example, the third image can display the first image and a portion of the second image.

[0032] Optionally, the display unit 106 may include one or more displays. In some optional embodiments, the display unit 106 may include a left-eye display and a right-eye display, respectively corresponding to the user's binocular display. Both the left-eye display area and the right-eye display can display a corresponding third image to ensure the user's binocular viewing experience.

[0033] In some optional embodiments, the head-mounted display device 200 may not need to be used in conjunction with a host device. That is, the second processing unit 104 of the head-mounted display device 200 does not rely on obtaining the second image data through a host device. In this optional case, the second image data may be stored in the local storage of the head-mounted display device 200, and the second processing unit 104 may also obtain the second image data from the local storage; alternatively, the second processing unit 104 may obtain the second image data through other means such as a network.

[0034] In some other alternative embodiments, such as Figure 1B As shown, the head-mounted display device 200 in this application can be used in conjunction with the host terminal 300. That is, the head-mounted display device 200 can be connected to the host terminal 300, and the host terminal 300 can provide second image data to the second processing unit 104 of the head-mounted display device 200. The host terminal 300 can be various electronic devices capable of providing source image data, including but not limited to: handheld terminals (such as mobile phones, handheld screen projectors, etc.), tablet computers, algorithm boxes, computers, vehicle-mounted systems, servers, data storage devices (such as portable hard drives, etc.), and other electronic devices. The head-mounted display device 200 and the host terminal 300 can be connected via wired or wireless means, without specific limitations.

[0035] It is understandable that, using the above-mentioned display system 100, when a target operation instruction is received from the head-mounted display device 200, the first processing unit 102 is invoked to run in parallel with the second processing unit 104. This allows the first processing unit 102 to process the locally stored first image data while the second processing unit 104 renders the second image data. As a result, the head-mounted display device 200 can render the first and second images in parallel when needed by the user, improving the efficiency of image processing and enhancing the user experience of the head-mounted display device.

[0036] Optionally, when the head-mounted display device 200 is a separate device, the head-mounted display device 200 does not need to rely on the host to run the driver program for the head-mounted display device. The head-mounted display device only needs to obtain the second image data from the host to realize the parallel rendering of the first and second images, as well as the composite display of the images. There is no need to consider the compatibility issues between the driver program and different host devices, which further improves the versatility of the head-mounted display device.

[0037] It is understood that this solution can be adapted to any usage scenario where other images are displayed simultaneously with the source image. For example, the first image mentioned above can be a settings screen for parameter settings of the head-mounted display device, and the second image data can be the source image displayed on the head-mounted display device. Therefore, using the solution disclosed in this application, the first processing unit 102 and the second processing unit 104 process different image data in parallel, which can provide users with options for adjusting the settings screen of the head-mounted display device while ensuring the display effect of the source image such as video, effectively improving the efficiency of setting and adjusting the head-mounted display device and enhancing the user experience.

[0038] In some optional embodiments, after obtaining the first image, the first processing unit 102 is further configured to transmit the first image to the first display layer of the head-mounted display device 200; after obtaining the second image, the second processing unit 104 is further configured to transmit the second image to the second display layer of the head-mounted display device 200, wherein the second display layer is a different display layer from the first display layer; the third image is synthesized by the first image obtained from the first cache corresponding to the first display layer and the second image obtained from the second cache corresponding to the second display layer.

[0039] In some optional embodiments, the first processing unit 102 is specifically used to transmit a first image to a first display layer of the head-mounted display device 200 at a first frame rate; the second processing unit 104 is specifically used to transmit a second image to a second display layer of the head-mounted display device 200 at a second frame rate; wherein the second frame rate is greater than or equal to the first frame rate; and the display unit 106 is specifically used to display a third image transmitted to the display unit 106 at the second frame rate.

[0040] In some optional embodiments, the first cache and the second cache are different storage units, and the read and write speed supported by the second cache is higher than that supported by the first cache.

[0041] In some alternative embodiments, the third image is obtained by overlaying the first image on top of the second image, and the third image is able to display the complete first image and at least a portion of the first image.

[0042] In some alternative embodiments, the transparency of the first image displayed in the third image can be adjusted based on received transparency adjustment instructions for the first image.

[0043] In some optional embodiments, the display unit 106 includes a left-eye display and a right-eye display respectively corresponding to the user's binocular display; the first processing unit 102 is specifically used to: perform binocular image merging processing on the first image data to obtain two first images respectively corresponding to the left-eye display and the right-eye display; the second processing unit 104 is specifically used to: perform spatial anchoring processing and binocular image merging processing on the second image data to obtain two second images respectively corresponding to the left-eye display and the right-eye display.

[0044] In some optional embodiments, the target operation instruction includes one of: a setting operation instruction for setting parameters, and a wake-up operation instruction for waking up the parameter setting interface.

[0045] The various optional embodiments described above will be specifically described in the method embodiments below. The specific content and beneficial effects can be understood by referring to the following text, and will not be repeated here.

[0046] According to one aspect of the embodiments of this application, a display method for a head-mounted display device is provided. The head-mounted display device 200 may include a first processing unit 102 and a second processing unit 104. Figure 2 A flowchart illustrating some optional display methods for head-mounted display devices in embodiments of this application is shown. (Refer to...) Figure 2 As shown, the method includes steps S202, S204, S206, and S208, specifically:

[0047] S202: In response to receiving a user's target operation command for the head-mounted display device, control the first processing unit and the second processing unit to operate in parallel, and call the first image data stored locally, process the first image data through the first processing unit to obtain the first image.

[0048] In some optional embodiments, the first image data can be pre-stored locally in the head-mounted display device 200. Optionally, the first processing unit 102 and the second processing unit 104 can process different image data independently and without interference. Optionally, as described above, the first processing unit 102 and the second processing unit 104 can be different processing units in the processing chip of the head-mounted display device 200, and this display method can be executed by the processing chip.

[0049] In some optional embodiments, upon receiving a user's target operation instruction for the head-mounted display device 200, the first processing unit 102 can be invoked, allowing the first processing unit 102 to operate in parallel with the second processing unit 104. It is understood that upon receiving a user's target operation instruction for the head-mounted display device 200, locally stored first image data can be invoked, enabling the first processing unit 102 to perform image rendering on the first image data to obtain a first image.

[0050] Optionally, the first image data may be image data of the parameter setting interface stored locally on the head-mounted display device. The first image may be an image of the parameter setting interface obtained by processing the first image data and then used for display on the screen. To meet the user experience of using the head-mounted display device 200, the first image data may be processed by a suitable algorithm to obtain the first image before being used for subsequent display on the screen. Here, the processing method of the first image data is not specifically limited and can be selected as needed. For example, in one example, the first image data may be processed by binocular merging to obtain the first images displayed on the left and right eye displays of the head-mounted display device respectively.

[0051] As mentioned above, the target operation command can be any type of operation command for the head-mounted display device 200. For example, it can be at least one of the following: button operation command, remote control operation command, voice operation command, gesture operation command, and remote operation command.

[0052] In some feasible implementations, physical button components can be set on the head-mounted display device 200, allowing users to operate according to the set button method, thereby realizing the input of target operation commands.

[0053] The specific function of the target operation instruction can be set as needed. In some optional embodiments, the target operation instruction may include either a setting operation instruction for setting parameters or a wake-up operation instruction for waking up the parameter setting interface.

[0054] The parameters that can be set by the setting operation command can be any parameter of the head-mounted display device 200. As an example, the setting operation command may include, but is not limited to, at least one of the following related parameters of the head-mounted display device 200: display parameters (e.g., brightness, resolution, saturation, screen size, screen distance, display mode (e.g., widescreen mode or narrowscreen mode), 2D / 3D display switching parameters), audio parameters (e.g., volume, volume switch, audio output channel), power parameters (e.g., charging, power supply, power output), etc.

[0055] The parameter setting interface that can be activated by the wake-up operation command can be a setting interface that can be used to set any one or more of the aforementioned parameters.

[0056] In some optional embodiments, the difference between setting operation commands and wake-up operation commands can be understood as follows: For example, a user can first input a wake-up operation command to wake up the parameter setting interface, and then set the parameters through the parameter setting interface. That is, the wake-up operation command will not change the parameters of the head-mounted display device before the parameter setting interface is woken up; or, for another example, a user can input a setting operation command to set the parameters of the head-mounted display device, and at the same time wake up the parameter setting interface.

[0057] It should be understood that in this embodiment, by responding to either a setting operation command for setting parameters or a wake-up operation command for waking up the parameter setting interface, the first processing unit 102 can be invoked, thereby enabling the first processing unit 102 and the second processing unit 104 of the head-mounted display device 200 to run in parallel, and to call the first image data stored locally. The first processing unit 102 processes the first image data to obtain a first image, and then combines the first image with the second image obtained by the second processing unit 104 to form a third image, which is then displayed on the display unit 106. When the first image is the image of the parameter setting interface, it is beneficial to meet the user's need to set parameters for the head-mounted display device while ensuring the display effect of the second image, thereby improving the user's experience with the head-mounted display device.

[0058] To facilitate the explanation of the embodiments of this application, the relevant optional content of step S204 will be described below.

[0059] S204: Acquire the second image data, process the second image data through the second processing unit, and obtain the second image.

[0060] In some optional embodiments, the second image data may be source image data for display on the head-mounted display device 200. The second image may be an image obtained by processing the second image data for display on the screen. As an example, the second image may be a video frame image of the video currently being played on the head-mounted display device 200. Optionally, to meet the user's experience using the head-mounted display device 200, the second image data may be processed by a suitable algorithm to obtain the second image before being used for subsequent display on the screen. The specific processing method is not limited here and can be selected as needed. For example, in one example, spatial anchoring algorithm processing and binocular image merging processing may be performed. Optionally, during the operation of the head-mounted display device 200, the second processing unit 104 may continuously run to process the acquired second image data and display the corresponding second image on the screen according to the refresh rate of the head-mounted display device 200.

[0061] In some optional embodiments, the second image data may be image data obtained by the second processing unit 104 from local storage, or image data obtained by the second processing unit 104 from other means such as a network. In other embodiments, the second image data may also be source image data provided by the host terminal 300 to the second processing unit 104 of the head-mounted display device 200.

[0062] In some optional application scenarios, the target operation command can be a wake-up operation command for waking up the parameter setting interface. When a user plays video using a head-mounted display device, assuming the operation method is physical button operation, the screen parameter setting interface of the head-mounted display device 200 can be brought up. When the user performs a double-click operation on the target physical button of the head-mounted display device 200, the corresponding operation command can be received to wake up the parameter setting interface. Then, the first processing unit 102 of the head-mounted display device 200 is invoked to control the first processing unit 102 and the second processing unit 104 for processing each frame of video image data to run in parallel. Processing unit 102 can retrieve and process first image data from local storage to obtain an image of the parameter setting interface (i.e., the first image). Simultaneously, processing unit 104, operating in parallel with processing unit 102, can retrieve and process second image data (e.g., frame data from a video obtained from host unit 300) to obtain a second image. This second image is then combined with the first image to obtain a third image, which is displayed by display unit 106. Thus, the parameter setting interface image (i.e., the first image) can be displayed simultaneously with video playback (i.e., while the second image is being displayed). The first and second images can represent at least a portion of various images, facilitating user settings of the parameters of the head-mounted display device 200 via the parameter setting interface.

[0063] This application does not limit the specific method of processing the first image data and the second image data. For example, if the display unit 106 includes a left-eye display and a right-eye display corresponding to the user's binocular display, then in order to satisfy the binocular display effect, the first image data and the second image data can be adaptively processed.

[0064] Optionally, the first processing unit 102 and the second processing unit 104 can each run different algorithms to process the image data they have processed. As an example, the first processing unit 102 and the second processing unit 104 can run different algorithms of the same type (which may correspond to different codes), or they can run different types of algorithms. For instance, the first processing unit 102 can run a binocular fusion processing algorithm to process the first image data, and the second processing unit 104 can run a different binocular fusion processing algorithm to process the second image data. Although the types of algorithms they run are the same, the two algorithms are implemented through different code segments, thereby ensuring that the first processing unit 102 and the second processing unit 104 can process the image data in parallel without interfering with each other.

[0065] In some optional embodiments, the first image data can be processed by the first processing unit 102 to obtain two first images corresponding to the left and right eye displays, respectively. Optionally, after the first image data is processed by binocular merging, the merged first image can be displayed at a specified position within the field of view of the head-mounted display device (e.g., the middle position within the field of view of the head-mounted display device).

[0066] In this application, the first image data is processed by binocular fusion processing to obtain two first images, so that the subsequent synthesis processing corresponds to two third images on the left and right eye displays. When viewed by the user's binoculars, the first image displayed in the two third images can achieve the effect of binocular stereoscopic display, so as to improve the user's experience of using the head-mounted display device.

[0067] In some optional embodiments, the second processing unit 104 can perform spatial anchoring processing and binocular merging processing on the second image data to obtain two second images corresponding to the left and right eye displays, respectively. Optionally, the above-mentioned spatial anchoring processing can be understood as monitoring the spatial changes of the head-mounted display device in the 3DOF (degrees of freedom) or 6DOF (degrees of freedom) directions, and calculating the position of the image in space based on the detected spatial changes, so as to achieve the display effect of anchoring the image displayed by the head-mounted display device at a fixed position in space.

[0068] In this application, the second image data is processed by spatial anchoring and binocular merging to obtain two second images. This results in two third images corresponding to the left and right eye displays in the subsequent compositing process. When viewed by the user's binoculars, the second images displayed in the two third images can be presented in the same position in space and can be anchored to a specified position in space without rotating with the user's head, thereby improving the user's experience with the head-mounted display device.

[0069] It should be understood that the order in which the second processing unit 104 performs binocular merging processing and spatial anchoring processing on the second image data can be set as needed. Binocular merging processing can be performed first, followed by spatial anchoring processing, or vice versa; there is no single limitation.

[0070] In some optional embodiments of this application, when the first processing unit 102 processes the first image data, it uses binocular image merging processing to obtain the first image, ensuring that the display position of the first image relative to the head-mounted display device remains unchanged and is displayed at a fixed position within the user's field of vision. This avoids image loss problems that may occur due to the small size of the first image. This also helps to accommodate the user's needs for setting parameters on the head-mounted display device when the first image is an image from a parameter setting interface, thereby further improving the user experience.

[0071] Optionally, since the second image data can be processed in the second processing unit 104, which runs in parallel with the first processing unit 102, the second processing unit 104 can quickly render a second image that is different from the first image display mode, thus meeting the spatial anchoring display requirements of the second image and ensuring the display effect of the head-mounted display device.

[0072] In other embodiments, when the first processing unit 102 processes the first image data, it may employ spatial anchoring processing in addition to binocular merging processing; there is no single limitation. It should be understood that the order of binocular merging processing and spatial anchoring processing can be set as needed. Binocular merging processing and spatial anchoring processing can be performed first, or spatial anchoring processing can be performed first and then binocular merging processing; there is no single limitation.

[0073] It should be understood that any existing or innovative binocular merging algorithm can be used to implement binocular merging processing, and any existing or innovative spatial anchoring algorithm can be used to implement spatial anchoring processing. This application does not make any specific limitations here.

[0074] In this application, a first image is obtained by performing binocular merging processing on the first image data, and a second image is obtained by performing spatial anchoring processing and binocular merging processing on the second image data. When the display unit 106 displays the third image obtained by merging the first image and the second image, the user can simultaneously see the second image anchored at a specified position in space and the first image that is always at the specified position in the field of vision and floats above the second image through the head-mounted display device 200.

[0075] S206: Combine the first image and the second image to obtain a third image, wherein the third image can display at least a portion of the first image and at least a portion of the second image.

[0076] After obtaining the first image and the second image, they can be combined to obtain the third image. By displaying the third image, at least a portion of the first image and at least a portion of the second image can be shown, thus achieving simultaneous display of the first image and the second image.

[0077] Optionally, as described above, the first processing unit 102 and the second processing unit 104 can be different processing units in the processing chip of the head-mounted display device 200, and the above-mentioned step S206 can be executed by the processing chip. In one implementation, the processing chip can run a synthesizer, which is a software module, and the synthesizer can perform composite processing of the first image and the second image.

[0078] This application does not limit the specific implementation of the compositing process. In one embodiment, the first image can be superimposed on top of the second image to obtain a third image that can display the complete first image and at least part of the second image.

[0079] Based on this, the synthesis of the first and second images can be effectively achieved. Since the third image synthesized in this way can display the complete first image and at least a portion of the second image, users can focus more on the content of the first image displayed in the third image. This also facilitates user adjustments to the head-mounted display device's settings when the first image is the parameter setting interface, further enhancing the user experience.

[0080] For example, a first image can be acquired and then superimposed on a second image at a specified position to achieve an overlay operation, thereby synthesizing a third image that can display the complete first image and at least part of the second image.

[0081] Optionally, the third image in this application displays at least a portion of the second image. If in some cases the second image is not fully displayed, it may be because the first image occludes a portion of the second image.

[0082] As an example, such as Figure 3 The diagram shown above, the first image is an image of the parameter setting interface (e.g., Figure 3 For example, this parameter setting interface can be used for display settings and audio settings (though it is not limited to this in practice). If the second image is a video frame image from a playing video, the first image can be overlaid on top of the second image at a specified position to achieve a superposition operation, thus synthesizing a third image. In the third image, the first image occludes the portion of the second image that overlaps with it, allowing the third image to display the complete first image and a portion of the second image. It should be understood that... Figure 3 This is not intended to limit the embodiments of this application.

[0083] As an optional embodiment, the display method of this application may also adjust the transparency of the first image displayed in the third image in response to receiving a transparency adjustment instruction for the first image.

[0084] It should be understood that by adjusting the transparency of the first image displayed in the third image, one can flexibly change whether the second image is fully displayed, flexibly change the style of the first image displayed in the third image, and effectively meet different user needs.

[0085] The aforementioned adjustment of the transparency of the first image displayed in the third image can be a default adjustment made during the mass production stage of the head-mounted display device in response to transparency adjustment commands input by staff; it can also be a flexible, custom adjustment made during the usage stage of the head-mounted display device in response to transparency adjustment commands input by the user when actively setting parameters of the head-mounted display device; it can also be an upgrade adjustment made in response to transparency adjustment commands included in the upgrade package when the head-mounted display device undergoes a system or firmware upgrade. Alternatively, it can take other forms of adjustment.

[0086] Optionally, the transparency adjustment command can be input in any operation method, such as at least one of the following: button operation command, remote control operation command, voice operation command, gesture operation command, remote operation command, interface operation command, etc.

[0087] In some optional embodiments, the transparency of the first image can be adjusted to the value corresponding to a completely opaque state (e.g., transparency set to 0). In this case, the portion of the second image overlapping the first image cannot be displayed in the third image, allowing the third image to display both the complete first image and a portion of the second image. In other optional embodiments, the transparency of the first image can be adjusted to the value corresponding to a transparent state. In this case, the portion of the second image overlapping the first image can be displayed in the third image, allowing the third image to display both the complete first and second images. It is understood that the greater the transparency of the first image, the clearer the portion of the second image overlapping the first image will be displayed.

[0088] In some optional embodiments, the first image data locally stored in the head-mounted display device 200 may include transparency information of the first image. Therefore, after retrieving the locally stored first image data, the first processing unit 102 can perform image processing according to the included transparency information, thereby obtaining a first image with corresponding transparency. It is understood that the transparency information included in the corresponding first image data may also differ for different frames of the first image, so as to generate first images with different transparency.

[0089] Optionally, the third image in this application may be an image with depth information, and the depth information of the second image shown in the third image is greater than the depth information of the first image shown in the third image.

[0090] Optionally, if the display unit 106 described above includes a left-eye display and a right-eye display corresponding to the user's binoculars, and two first images and two second images corresponding to the left-eye and right-eye displays are obtained respectively, then the first image corresponding to the left-eye display and the second image corresponding to the left-eye display can be combined to obtain a third image corresponding to the left-eye display; and the first image corresponding to the right-eye display and the second image corresponding to the right-eye display can be combined to obtain a third image corresponding to the right-eye display. This provides two third images corresponding to the left-eye and right-eye displays respectively, facilitating subsequent display on the left-eye and right-eye displays to meet the requirements of the head-mounted display device for binocular display.

[0091] In some optional embodiments, after obtaining the first image in step S202, the display method of this application may further include: transmitting the first image to the first display layer of the head-mounted display device.

[0092] Optionally, the first display layer can be referred to as an Overlay layer, and may correspond to the first buffer of the head-mounted display device 200. The first processing unit 102 described above can transmit the processed first image to the first display layer, thereby realizing the transmission of the first image to the first display layer of the head-mounted display device 200. The first buffer can buffer the image in the first display layer.

[0093] Optionally, the display unit 106 may include a left-eye display and a right-eye display. Then, the first display layer may include two first display layers corresponding to the left-eye display and the right-eye display, respectively. The two first images corresponding to the left-eye display and the right-eye display can be transmitted to the two first display layers respectively.

[0094] In some optional embodiments, after obtaining the second image, the display method of this application may further include: transmitting the second image to a second display layer of a head-mounted display device, wherein the second display layer is a different display layer from the first display layer.

[0095] Optionally, the second display layer can be referred to as the Video layer, and may correspond to the second buffer of the head-mounted display device 200. The second processing unit 104 described above can transmit the processed second image to the second display layer, thereby realizing the transmission of the second image to the second display layer of the head-mounted display device 200. The second buffer can cache the image in the second display layer.

[0096] Optionally, the display unit 106 may include a left-eye display and a right-eye display. The second display layer may include two second display layers corresponding to the left-eye display and the right-eye display, respectively. The two second images corresponding to the left-eye display and the right-eye display can be transmitted to the two second display layers respectively.

[0097] In some optional embodiments, a first image may be transmitted to a first display layer of the head-mounted display device at a first frame rate, and a second image may be transmitted to a second display layer of the head-mounted display device at a second frame rate, wherein the second frame rate is greater than or equal to the first frame rate. Optionally, in the case of compositing a third image, the first image may be obtained from a first buffer, and the second image may be obtained from a second buffer. It is understood that, since the first frame rate is less than the second frame rate, there may be cases where the first image is the same in adjacent frames of the third image.

[0098] It is understandable that when the first image is transmitted to the first display layer of the head-mounted display device at a first frame rate, the first display layer can be refreshed at a corresponding first speed. When the second image is transmitted to the second display layer of the head-mounted display device at a second frame rate, the second display layer can be refreshed at a corresponding second speed. The second image and the first image are combined at a second speed to form a third image, and the head-mounted display device also refreshes the screen at a second speed. Therefore, the third image can be refreshed at a high speed on the head-mounted display device to ensure the stability of the image display. It is evident that in this scheme, different processing units process images in parallel, ensuring the independence of the first and second display layers receiving different images. This allows the first and second display layers to be refreshed at different speeds, ensuring the display effect of the head-mounted display device while reducing the computational load on the head-mounted display device. Taking the transmission of the second image data from the host terminal 300 to the head-mounted display device 200 as an example, the generation of the second image data is generally faster than that of the first image data (for example, the reason could be that the host terminal 300 generates the second image data faster). Therefore, the latency for processing the second image data to obtain the second image and displaying it on the screen must be low. Thus, a faster processing method is suitable for processing the second image data to obtain the second image, to accommodate the high-speed refresh of the second image. However, in some cases, the size of the first image data is smaller (for example, see...). Figure 3 (For example, the first image is processed from local storage only after the user inputs the target operation command, which is relatively slow. Therefore, a slower method can be used to process the first image data to obtain the first image, in order to adapt to the slower refresh rate of the first image. As an example, this can also be understood in the context of parameter settings. In this case, the first image displayed in the third image is the image of the parameter settings interface, and the second image is the image of the video frame being played. Generally, the first image refreshes more slowly, and the second image refreshes more quickly.)

[0099] Therefore, in this embodiment, by transmitting the first image to the first display layer of the head-mounted display device 200 at a first frame rate, and transmitting the second image to the second display layer of the head-mounted display device 200 at a second frame rate greater than or equal to the first frame rate, the display unit 106 of the head-mounted display device 200 can refresh and display the third image at the second frame rate. This ensures that the second image has a higher refresh rate to improve the stability of the image displayed by the head-mounted display device 200, and also reduces the computational load of the head-mounted display device 200 by using the low refresh rate of the first image corresponding to the first image, thereby improving the user experience of using the head-mounted display device.

[0100] The first and second frame rates can be set according to actual needs. For example, the first frame rate can be less than or equal to 30Hz. For example, the second frame rate can be greater than or equal to 90Hz.

[0101] For example, in one example, the first frame rate is 30Hz and the second frame rate is 90Hz. Optionally, in this case, when synthesizing the third image using every 3 frames of the second image, the same first image can be used for each frame. For example, assuming that the 3 frames of the second image transmitted at a frame rate of 90Hz are denoted as A1, A2, and A3, and the 1 frame of the first image transmitted at a frame rate of 30Hz is denoted as B, and the synthesized third images are denoted as C1, C2, and C3, then the synthesized third image C1 can use the first image B and the second image A1, the synthesized third image C2 can use the first image B and the second image A2, and the synthesized third image C3 can use the first image B and the second image A3.

[0102] Optionally, the first cache and the second cache can be different storage units, and the read and write speed supported by the second cache is higher than that supported by the first cache.

[0103] Therefore, by using caches with different read and write speeds, it is easy to transmit the first and second images at different frame rates to accommodate the need for faster processing of the second image data and relatively slower processing of the first image data. This ensures that when the display unit 106 of the head-mounted display device 200 displays the third image, it can guarantee the stability of the high-speed refreshed second image display while reducing the computational load of the head-mounted display device through the low refresh rate of the first image, thereby improving the user experience of the head-mounted display device.

[0104] Optionally, the second cache can be a high-speed cache. The second display layer can be connected to the second processing unit 104 via the high-speed cache. Optionally, the first cache can be general-purpose memory. The first display layer can be connected to the first processing unit 102 via general-purpose memory. The second cache and the first cache can be independent of each other, which can meet the "fast and slow" transmission requirements of the second image and the first image, and avoid data transmission conflicts.

[0105] Optionally, refer to Figure 4 The flowchart shown illustrates how steps S2062 to S2066 can be used to synthesize the first and second images to obtain the third image. Specifically:

[0106] S2062: Retrieve the first image from the first cache corresponding to the first display layer.

[0107] For example, the first image can be obtained from the general memory corresponding to the first display layer (overlay layer). This can be understood by example as an image of the parameter setting interface of the head-mounted display device 200.

[0108] Optionally, the display unit 106 may include a left-eye display and a right-eye display, and then two first images corresponding to the left-eye display and the right-eye display respectively can be obtained from the first cache.

[0109] S2064: Retrieve the second image from the second cache corresponding to the second display layer.

[0110] For example, the second image can be obtained from the cache corresponding to the second display layer (Video layer). This can be understood by taking the second image as a video frame image of the video being played on the head-mounted display device 200.

[0111] Optionally, the display unit 106 may include a left-eye display and a right-eye display, so that two second images corresponding to the left-eye display and the right-eye display respectively can be obtained from the second buffer.

[0112] S2066: Combine the first image and the second image to obtain the third image.

[0113] As mentioned earlier, the first image can be superimposed on top of the second image to obtain a third image that fully displays the first image and at least partially displays the second image. This has already been explained in detail above and will not be repeated here.

[0114] Optionally, the display unit 106 may include a left-eye display and a right-eye display. By superimposing the first image corresponding to the left-eye display onto the second image corresponding to the left-eye display, a third image corresponding to the left-eye display can be obtained; similarly, by superimposing the first image corresponding to the right-eye display onto the second image corresponding to the right-eye display, a third image corresponding to the right-eye display can be obtained.

[0115] Based on this, the optional implementation schemes of steps S2062 to S2066 in this application can effectively acquire the first image and the second image, so as to effectively realize the synthesis processing of the first image and the second image and obtain the third image for subsequent display.

[0116] S208: Display the third image on the display unit of the head-mounted display device.

[0117] After the third image is synthesized, it can be transmitted to the display unit 106 so that the display unit 106 can display the third image to show the user at least part of the first image and at least part of the second image.

[0118] For example, if the display unit 106 includes a left-eye display and a right-eye display, then two third images corresponding to the left-eye display and the right-eye display respectively are obtained. The two third images can be transmitted to the left-eye display and the right-eye display respectively for display to meet the user's binocular display requirements.

[0119] Based on this, the optional technical solution in this application can, in response to receiving a user's target operation command for the head-mounted display device, control the first processing unit and the second processing unit in the head-mounted display device to operate in parallel. This allows the first processing unit to process the first image data to obtain a first image, and the second processing unit to process the second image data to obtain a second image. The first and second images are then combined to form a third image, enabling the head-mounted display device to display the third image, thus showing the user at least a portion of the first image and at least a portion of the second image. Therefore, if this solution is adopted, when the user needs to display the first and second images simultaneously, it is not necessary to run an application program compatible with the head-mounted display device on the host side. The user can perform corresponding operations on the head-mounted display device when needed, allowing multiple processing units to process the first and second images in parallel on the head-mounted display device side, and then combine and display the first and second images. This display method is both convenient and efficient, improving the user experience of the head-mounted display device. Therefore, this solution can be adapted to any usage scenario that displays other images while displaying an image (including but not limited to scenarios such as "displaying a settings screen for setting parameters of the head-mounted display device while the source image is displayed on the head-mounted display device"), ensuring the convenience and efficiency of the display in the corresponding usage scenario and effectively improving the display effect.

[0120] It is understood that by adopting the technical solution of this application, when a target operation instruction is received from the head-mounted display device 200, the first processing unit 102 is invoked to run in parallel with the second processing unit 104. This allows the first processing unit 102 to process the locally stored first image data while the second processing unit 104 renders the second image data. As a result, the head-mounted display device 200 can render the first and second images in parallel when needed by the user, which improves the efficiency of image processing and enhances the user's experience with the head-mounted display device.

[0121] Optionally, when the head-mounted display device 200 is a separate device, the head-mounted display device 200 does not need to rely on the host to run the driver program for the head-mounted display device. The head-mounted display device only needs to obtain the second image data from the host to realize the parallel rendering of the first and second images, as well as the composite display of the images. There is no need to consider the compatibility issues between the driver program and different host devices, which further improves the versatility of the head-mounted display device.

[0122] It is understood that this solution can be adapted to any usage scenario where other images are displayed simultaneously with the source image. For example, the first image mentioned above can be a settings screen for parameter settings of the head-mounted display device, and the second image data can be the source image displayed on the head-mounted display device. Therefore, using the solution disclosed in this application, the first processing unit 102 and the second processing unit 104 process different image data in parallel, which can provide users with options for adjusting the settings screen of the head-mounted display device while ensuring the display effect of the source image such as video, effectively improving the efficiency of setting and adjusting the head-mounted display device and enhancing the user experience.

[0123] In some alternative embodiments, step S208 may include: transmitting the third image at a second frame rate to the display unit of the head-mounted display device for display.

[0124] Based on this, by transmitting the third image to the display unit 106 of the head-mounted display device 200 at the second frame rate, both the display effect of the high-speed refreshed second image and the display effect of the slower refreshed first image can be guaranteed, thereby improving the user experience of the head-mounted display device. Furthermore, this also facilitates user adjustments to the head-mounted display device's parameters when the first image is an image from the parameter setting interface, further enhancing the user experience.

[0125] For example, in one instance, if the first frame rate is 30Hz and the second frame rate is 90Hz, then the third image can be transmitted to the display unit 106 of the head-mounted display device 200 for display at the higher 90Hz frame rate.

[0126] Optionally, if the display unit 106 includes a left-eye display and a right-eye display, then the third image corresponding to the left-eye display can be transmitted to the left-eye display of the display unit 106 at a second frame rate for display; and the third image corresponding to the right-eye display can be transmitted to the right-eye display of the display unit 106 at a second frame rate for display.

[0127] The following is combined with Figure 5 The complete process of an optional display scheme according to an embodiment of this application will be illustrated by example. When understanding the following content, it can be understood in the scenario of "displaying a settings screen for setting parameters of the head-mounted display device when the source screen is displayed on the head-mounted display device".

[0128] like Figure 5As shown, when a user uses the head-mounted display device 200 (taking AR glasses as an example), if it is necessary to adjust the display brightness of the head-mounted display device 200, the user can input a target operation command (taking the wake-up operation command for waking up the parameter setting interface input via physical buttons as an example). Upon receiving the target operation command, the first processing unit 102 and the second processing unit 104 can be controlled to run in parallel: the first processing unit 102 can retrieve the first image data (e.g., the image data of the parameter adjustment interface) from local storage, and perform binocular image merging processing on the first image processing, thereby obtaining two first images (e.g., the image of the parameter adjustment interface) corresponding to the left and right eye displays of the display unit 106, respectively. Then, the two first images are transmitted to the first display layer (overlay layer) corresponding to the left and right eye displays at a first frame rate (e.g., 30Hz). The second processing unit 104, which runs in parallel with the first processing unit 102, can obtain the second image data (e.g., the video frame image data of the played video) from the host terminal 300, and perform spatial anchoring processing and binocular image merging processing on the second image processing, thereby obtaining two second images (e.g., video frame images) corresponding to the left and right eye displays of the display unit 106, respectively. Then, the two second images are transmitted to the second display layer (Video layer) corresponding to the left and right eye displays at a second frame rate greater than the first frame rate (e.g., 90Hz).

[0129] Subsequently, the synthesizer software module running on the processing chip of the head-mounted display device 200 can retrieve two first images from the first cache (which can be general memory) corresponding to the first display layer, and two second images from the second cache (which can be a high-speed cache) corresponding to the second display layer. Then, the first and second images corresponding to the left eye display are synthesized to obtain a third image corresponding to the left eye display, and the first and second images corresponding to the right eye display are synthesized to obtain a third image corresponding to the right eye display (the first image can be overlaid on a specified position above the second image to achieve a superposition operation, resulting in a third image that can display the complete first image and at least part of the second image). Finally, the third images corresponding to the left and right eye displays can be transmitted to the left and right displays of the head-mounted display device at a second frame rate for display. In this way, the head-mounted display device 200 can display the first image while displaying the second image, meeting the user's need for parameter settings on the head-mounted display device 200.

[0130] like Figure 5The implementation scheme shown cleverly utilizes the independent and non-interfering characteristics of the first processing unit 102 and the second processing unit 104 of the head-mounted display device 200 in image data processing. It separately processes the first and second image data using algorithms to obtain the first and second images, and fully considers the "faster and slower" processing characteristics of the second and first image data. The first and second images are output to the first and second display layers at different frame rates. Then, the synthesizer running on the processing chip of the head-mounted display device 200 combines the first and second images to obtain a third image that can simultaneously display at least a portion of the first and second images. This ensures both the spatial anchoring effect and binocular merging effect of the high-speed refreshed second image, as well as the binocular merging effect of the first image. Furthermore, it guarantees both the stability of the second image and the display effect of the slower refreshed first image, thereby improving the user experience of the head-mounted display device. In the above implementation scheme, the host terminal 300 only inputs the second image data as the source image data, while all other processing can be performed on the head-mounted display device 200 side. This facilitates efficient and convenient synthesis and display of the first and second images, improving the user experience of the head-mounted display device. Therefore, this scheme can be adapted to any usage scenario where other images are displayed simultaneously (including but not limited to the example of "displaying a settings screen for parameter settings of the head-mounted display device while displaying the source image"), ensuring convenience and efficiency in the corresponding usage scenario and effectively improving the display effect. It should be understood that... Figure 5 The examples shown are not intended to limit this application.

[0131] It is understood that the above description is only some optional embodiments of the display method of this application and is not intended to limit the embodiments of this application.

[0132] According to another aspect of the embodiments of the application, a display device for a head-mounted display device is provided. The head-mounted display device includes a first processing unit and a second processing unit. (Refer to...) Figure 6 As shown, the display device 600 for a head-mounted display device includes:

[0133] The first processing module 602 is configured to respond to receiving a user's target operation command for the head-mounted display device, control the first processing unit and the second processing unit to operate in parallel, and call the first image data stored locally, process the first image data through the first processing unit, and obtain the first image.

[0134] The second processing module 604 is used to acquire second image data and process the second image data through the second processing unit to obtain a second image.

[0135] The compositing module 606 is used to compose the first image and the second image to obtain a third image, wherein the third image can display at least a portion of the first image and at least a portion of the second image;

[0136] Display module 608 is used to display a third image on the display unit of a head-mounted display device.

[0137] In some optional embodiments, after obtaining the first image, the display device 600 is further configured to: transmit the first image to the first display layer of the head-mounted display device; after obtaining the second image, the display device 600 is further configured to: transmit the second image to the second display layer of the head-mounted display device, wherein the second display layer is a different display layer from the first display layer; the compositing processing module 606 is specifically configured to: obtain the first image from the first cache corresponding to the first display layer; obtain the second image from the second cache corresponding to the second display layer; and perform compositing processing on the first image and the second image to obtain the third image.

[0138] In some optional embodiments, the display device 600 is specifically used to: transmit a first image to a first display layer of the head-mounted display device at a first frame rate; transmit a second image to a second display layer of the head-mounted display device at a second frame rate, wherein the second frame rate is greater than or equal to the first frame rate; and the display module 608 is specifically used to: transmit a third image to a display unit of the head-mounted display device at a second frame rate for display.

[0139] In some optional embodiments, the first cache and the second cache are different storage units, and the read and write speed supported by the second cache is higher than that supported by the first cache.

[0140] In some optional embodiments, the compositing module 606 is specifically used to: superimpose the first image onto the second image to obtain a third image that can display the complete first image and at least part of the second image.

[0141] In some alternative embodiments, the display device 600 is further configured to: adjust the transparency of the first image displayed in the third image in response to receiving a transparency adjustment instruction for the first image.

[0142] In some optional embodiments, the display unit includes a left-eye display and a right-eye display respectively corresponding to the user's binocular display; the first processing module 602 is specifically used to: perform binocular merging processing on the first image data through the first processing unit to obtain two first images respectively corresponding to the left-eye display and the right-eye display; the second processing module 604 is specifically used to: perform spatial anchoring processing and binocular merging processing on the second image data through the second processing unit to obtain two second images respectively corresponding to the left-eye display and the right-eye display.

[0143] In some optional embodiments, the target operation instruction includes one of: a setting operation instruction for setting parameters, and a wake-up operation instruction for waking up the parameter setting interface.

[0144] The display device 600 for a head-mounted display device provided in this application is based on the same inventive concept as the display method for a head-mounted display device described above. It corresponds to the respective display methods for head-mounted display devices in the foregoing multiple method embodiments and has the beneficial effects of the corresponding display method embodiments for head-mounted display devices; therefore, it will not be described again here. Furthermore, the implementation of each module in the display device 600 for a head-mounted display device in this application embodiment can refer to the description of the corresponding parts in the foregoing display method embodiments for head-mounted display devices, and will not be described again here.

[0145] According to another aspect of the embodiments of this application, an electronic device is provided, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store a computer program; the processor is used to execute the display method for a head-mounted display device described above by running the computer program stored in the memory.

[0146] Reference Figure 7 This illustration shows a structural schematic diagram of an electronic device according to an embodiment of this application. The embodiments of this application do not limit the specific implementation of the electronic device. Figure 7 As shown, the electronic device 700 may include: a processor 702, a communications interface 704, a memory 706, and a communications bus 708.

[0147] in:

[0148] The processor 702, communication interface 704, and memory 706 communicate with each other via communication bus 708.

[0149] Communication interface 704 is used to communicate with other electronic devices or servers.

[0150] The processor 702 is used to execute the computer program 710, specifically the relevant steps in the above-described embodiment of the display method for a head-mounted display device.

[0151] Specifically, computer program 710 may include program code that includes computer operation instructions.

[0152] The processor 702 may be a CPU, a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0153] Memory 706 is used to store computer program 710. Memory 706 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0154] The computer program 710 may include multiple computer instructions. Specifically, the computer program 710 may use multiple computer instructions to cause the processor 702 to perform the operation corresponding to the display method for the head-mounted display device described in any of the foregoing multiple method embodiments.

[0155] The specific implementation of each step in computer program 710 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0156] According to another aspect of the embodiments of this application, an embodiment of this application also provides a computer storage medium storing a computer program thereon. When executed by a processor, the computer program implements the display method for a head-mounted display device described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk, etc.

[0157] According to another aspect of the embodiments of this application, the embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the display method for a head-mounted display device described in any of the foregoing method embodiments.

[0158] The display device 600 / electronic device 700 / display system 100 / computer storage medium / computer program product embodiment for head-mounted display device in this application has been described in detail in the foregoing display method embodiment for head-mounted display device. Therefore, its related content and beneficial effects can be understood by referring to the above method embodiment, and will not be repeated here.

[0159] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0160] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application. It should be understood that the various technical features in the technical solutions of the embodiments of this application can be combined in any suitable manner.

[0161] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0162] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for specific applications, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0163] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

[0164] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". It should be noted that the concepts of "first", "second", etc., mentioned in the embodiments of this application are only used to distinguish different devices, modules, or units, and are not used to limit the order of functions performed by these devices, modules, or units or their interdependencies. It should be noted that the modifications of "a" and "a plurality" mentioned in the embodiments of this application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of this application, and are not intended to limit them; although the embodiments of this application have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A display method for a head-mounted display device, the head-mounted display device comprising a first processing unit and a second processing unit, the method comprising: In response to receiving a user's target operation command for the head-mounted display device, the system controls the first processing unit and the second processing unit to operate in parallel, and calls the first image data stored locally, processes the first image data through the first processing unit to obtain the first image; Acquire second image data, process the second image data through the second processing unit, and obtain a second image; The first image and the second image are combined to obtain a third image, wherein the third image can display at least a portion of the first image and at least a portion of the second image; The third image is displayed on the display unit of the head-mounted display device.

2. The method according to claim 1, wherein, After obtaining the first image, the method further includes: transmitting the first image to a first display layer of the head-mounted display device; After obtaining the second image, the method further includes: transmitting the second image to a second display layer of the head-mounted display device, wherein the second display layer is a different display layer from the first display layer; The step of compositing the first image and the second image to obtain the third image includes: Retrieve the first image from the first cache corresponding to the first display layer; Retrieve the second image from the second cache corresponding to the second display layer; The first image and the second image are combined to obtain the third image.

3. The method according to claim 2, wherein, The step of transmitting the first image to the first display layer of the head-mounted display device includes: transmitting the first image to the first display layer of the head-mounted display device at a first frame rate; The step of transmitting the second image to the second display layer of the head-mounted display device includes: transmitting the second image to the second display layer of the head-mounted display device at a second frame rate, wherein the second frame rate is greater than or equal to the first frame rate; The step of displaying the third image on the display unit of the head-mounted display device includes: transmitting the third image to the display unit of the head-mounted display device at the second frame rate for display.

4. The method according to claim 2 or 3, wherein, The first cache and the second cache are different storage units, and the read and write speed supported by the second cache is higher than that supported by the first cache.

5. The method according to claim 1, wherein, The step of compositing the first image and the second image to obtain the third image includes: The first image is superimposed on top of the second image to obtain a third image that displays the complete first image and at least a portion of the second image.

6. The method according to any one of claims 1-3, wherein, The method further includes: In response to receiving a transparency adjustment instruction for the first image, the transparency of the first image displayed in the third image is adjusted.

7. The method according to any one of claims 1-3, wherein, The display unit includes a left-eye display and a right-eye display, respectively corresponding to the user's binocular display; The step of processing the first image data through the first processing unit to obtain the first image includes: The first processing unit performs binocular image merging processing on the first image data to obtain two first images corresponding to the left eye display and the right eye display, respectively. The step of processing the second image data through the second processing unit to obtain the second image includes: The second processing unit performs spatial anchoring processing and binocular merging processing on the second image data to obtain two second images corresponding to the left eye display and the right eye display, respectively.

8. The method according to any one of claims 1-3, wherein, The target operation instruction includes one of the following: a setting operation instruction for setting parameters, and a wake-up operation instruction for waking up the parameter setting interface.

9. A display device for a head-mounted display device, the head-mounted display device comprising a first processing unit and a second processing unit, the device comprising: The first processing module is configured to respond to receiving a target operation command from a user for the head-mounted display device, control the first processing unit and the second processing unit to run in parallel, and call locally stored first image data, process the first image data through the first processing unit to obtain a first image; The second processing module is used to acquire second image data and process the second image data through the second processing unit to obtain a second image. A compositing module is used to compose the first image and the second image to obtain a third image, wherein the third image can display at least a portion of the first image and at least a portion of the second image; The display module is used to display the third image on the display unit of the head-mounted display device.

10. A display system, comprising: The first processing unit is configured to, in response to receiving a target operation command from a user for a head-mounted display device, process locally stored first image data to obtain a first image. The second processing unit is configured to, in response to acquiring the second image data, operate in parallel with the first processing unit to process the second image data and obtain the second image. A display unit is configured to display a third image synthesized from the first image and the second image, wherein the third image is capable of displaying at least a portion of the first image and at least a portion of the second image.

11. An electronic device, comprising: The processor, the communication interface, the memory, and the communication bus are provided, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The memory is used to store computer programs; The processor is configured to perform the method of any one of claims 1-8 by running the computer program stored in the memory.

12. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-8.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method as described in any one of claims 1-8.

Images