Display method, device, system, product, terminal equipment and augmented reality equipment

Abstract

The invention relates to the technical field of display, and provides a display method, device and system, a product, terminal equipment and augmented reality equipment. The display method comprises the steps that a first device splices pictures displayed by at least one virtual display in the first device, and a target picture is obtained; and the first equipment sends the target picture to second equipment, so that the second equipment determines a display picture indicated by the view field range of the second equipment from the target picture and displays the display picture. According to the scheme, the delay of multi-task virtual display can be reduced.

Description

Technical Field

[0001] This disclosure relates to the field of display technology, and more specifically, to a display method, display device, display system, computer program product, terminal equipment, and extended reality device. Background Technology

[0002] Virtual screen display technologies, such as Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), and Extended Reality (XR), are now widely used in various aspects of life, including healthcare, entertainment, and education, providing users with immersive experiences. In these immersive application scenarios, users' need for multitasking is becoming increasingly urgent, leading to the emergence of multitasking virtual screen display methods.

[0003] The multi-task virtual screen display technology in related technologies mainly generates multiple virtual displays on the host side. The head-mounted display device transmits its posture data to the host. The host software obtains the posture data of the head-mounted display device and calculates the position, angle and projection relationship of the virtual screen in three-dimensional space in real time based on the posture changes to determine the display screen of the head-mounted display device. Then, the rendered display screen is transmitted to the head-mounted display device.

[0004] Obviously, existing multi-tasking virtual screen display technology requires multiple interactions between the host and the glasses to display the image, which results in display delay and screen jitter.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a display method and apparatus for a first device, a display method and apparatus for a second device, a display system, a computer program product, a terminal device, and an extended reality device, thereby improving, at least to some extent, the display latency problem existing in multi-tasking virtual screen display technology.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0008] According to a first aspect of this disclosure, a display method is provided, applied to a first device, the display method comprising: stitching together images displayed on at least one virtual display in the first device to obtain a target image; and sending the target image to a second device, so that the second device determines a display image indicated by the field of view of the second device from the target image and displays the display image.

[0009] According to a second aspect of this disclosure, a display method is provided, applied to a second device, the display method comprising: receiving a target image sent by a first device, the target image being obtained by the first device stitching together images displayed by at least one virtual display in the first device; determining the field of view of the second device based on the posture information of the second device, so as to determine a display image indicated by the field of view from the target image; and displaying the display image.

[0010] According to a third aspect of this disclosure, a display system is provided, comprising: a first device for creating at least one virtual display and stitching together the images displayed on the at least one virtual display to obtain a target image; and a second device for determining, from the target image, a display image indicated by the field of view of the second device, and displaying the display image.

[0011] According to a fourth aspect of this disclosure, a display device is provided, applied to a first device, the device comprising: a splicing module configured to splice images displayed on at least one virtual display in the first device to obtain a target image; and a sending module configured to send the target image to a second device, so that the second device determines a display image indicated by the field of view of the second device from the target image and displays the display image.

[0012] According to a fifth aspect of this disclosure, a display device is provided for use in a second device, the device comprising: a receiving module configured to receive a target image sent by a first device, the target image being obtained by the first device stitching together images displayed by at least one virtual display in the first device; a display image determining module configured to determine the field of view of the second device based on the posture information of the second device, so as to determine a display image indicated by the field of view from the target image; and a display module configured to display the display image.

[0013] According to a sixth aspect of this disclosure, a computer program product comprising instructions is provided that, when run on a computer, causes the computer to perform the steps of the display method as described in the first and / or second aspects.

[0014] According to a seventh aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the display methods described in the first and / or second aspects of the above embodiments.

[0015] According to an eighth aspect of this disclosure, a terminal device is provided, comprising: a processor; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the display method as described in the first aspect of the above embodiments.

[0016] According to a ninth aspect of this disclosure, an extended reality device is provided, comprising: a processor; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the display method as described in the first aspect of the above embodiments; and a display device for displaying a display screen under the control of the processor.

[0017] As can be seen from the above technical solutions, the display method, display device, display system terminal equipment, extended reality device, and computer program product implementing the display method in the exemplary embodiments of this disclosure have at least the following advantages and positive effects: In some embodiments of this disclosure, the technical solutions provided include, on the one hand, stitching together the images displayed on at least one virtual display in the first device to obtain a target image, and then sending the target image to a second device, thereby enabling multi-task virtual display; on the other hand, in this disclosure, the first device only needs to send the stitching result of the images corresponding to multiple virtual displays to the second device, and the second device can directly determine the current display image from the stitched images based on its own field of view, without requiring multiple interactions between the first and second devices, which can reduce the display latency of multi-task virtual display; furthermore, in this disclosure, the host only needs to stitch together the images of multiple virtual displays, resulting in low host computing load and easy cross-platform deployment and system capability expansion.

[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0020] Figure 1 This diagram illustrates a flow chart of a display method according to an exemplary embodiment of the present disclosure.

[0021] Figure 2 This diagram illustrates a flowchart of a method for determining a preset positional relationship according to an exemplary embodiment of the present disclosure.

[0022] Figure 3 This illustration shows a flowchart of a method for a second device to determine a display screen according to an exemplary embodiment of the present disclosure.

[0023] Figure 4 This diagram illustrates a flowchart of a method for determining a preset curvature according to an exemplary embodiment of the present disclosure.

[0024] Figure 5 This is a schematic diagram showing a target screen in an exemplary embodiment of the present disclosure.

[0025] Figure 6 This diagram illustrates an arc-shaped screen according to an exemplary embodiment of the present disclosure.

[0026] Figure 7 This diagram illustrates a flowchart of another display method according to an exemplary embodiment of the present disclosure.

[0027] Figure 8 This diagram illustrates a flowchart of another method for determining a display screen according to an exemplary embodiment of the present disclosure.

[0028] Figure 9 This illustration shows a schematic diagram of the architecture of a display system according to an exemplary embodiment of the present disclosure.

[0029] Figure 10 This diagram illustrates the structure of a display device according to an exemplary embodiment of the present disclosure.

[0030] Figure 11 This diagram illustrates the structure of another display device according to an exemplary embodiment of the present disclosure.

[0031] Figure 12 This diagram illustrates the structure of a terminal device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0032] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0033] The terms “a,” “an,” “the,” and “the” are used in this specification to indicate the presence of one or more elements / components / etc.; the terms “including” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first” and “second” are used only as markings and are not a limitation on the number of objects.

[0034] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0035] Multi-tasking virtual screen display technology, a related field, involves deploying multiple virtual screens in a virtual space. During display, the head-mounted display device first sends its pose information to the host computer. The host computer's software then obtains the head-mounted display device's pose data and calculates the position, angle, and projection relationship of the virtual screens in three-dimensional space in real time based on the pose changes. Finally, the rendered image is transmitted to the head-mounted display device for display. This entire process involves multiple interactions between the head-mounted display device and the host computer, resulting in significant display latency on the head-mounted display device and negatively impacting the user's viewing experience.

[0036] To address the aforementioned problems, this disclosure provides a display method. Figure 1This diagram illustrates a flowchart of a display method according to an exemplary embodiment of the present disclosure. The display method provided in this embodiment is applied to a first device. (See reference...) Figure 1 The display method includes: Step S110: The images displayed on at least one virtual display in the first device are stitched together to obtain the target image; Step S120: The target screen is sent to the second device so that the second device can determine the display screen indicated by the field of view of the second device from the target screen and display the display screen.

[0037] exist Figure 1 In the technical solution provided by the illustrated embodiment, on the one hand, the first device stitches together the images displayed on at least one virtual display in the first device to obtain a target image, and sends the target image to the second device, thereby realizing multi-task virtual display; on the other hand, in this disclosure, the first device only needs to send the stitching result of the images corresponding to multiple virtual displays to the second device, and the second device can directly determine the current display image from the stitched image based on its own field of view, without the need for multiple interactions between the first device and the second device, which can reduce the display latency of multi-task virtual display; furthermore, in this disclosure, the host only needs to stitch together the images of multiple virtual displays, resulting in low host computing load and easy cross-platform deployment and system capability expansion.

[0038] The following describes the specific implementation of "step S110, stitching together the images displayed on at least one virtual display in the first device to obtain the target image".

[0039] In one exemplary embodiment, the first device may include computer equipment, such as a desktop computer, laptop, smartphone, smart wearable electronic device (such as a smartwatch, smart neckband), server, etc. The first device may or may not have a physical screen; this exemplary embodiment does not impose any particular limitation on this.

[0040] For example, multiple logically independent virtual display instances can be created in the first device through a virtual display driver or the operating system interface of the first device. Each virtual display is used to display different applications or task windows. For instance, each virtual display can display different applications in full screen, display different task windows of the same application, or display different applications as task windows.

[0041] For example, one exemplary implementation of step S110 may include: stitching together pixel data corresponding to the screen displayed by at least one virtual display in the first device to obtain a target screen.

[0042] For example, the first device can stitch together the content displayed on multiple virtual displays in pixel space to form a continuous ultrawide screen image, and output it in the form of a video stream or display signal.

[0043] For example, stitching together the pixel data corresponding to the images displayed by at least one virtual display in the first device to obtain a target image includes: stitching together the pixel data corresponding to the images displayed by at least one virtual display in the first device according to the preset positional relationship between the virtual displays in the first device to obtain a target image.

[0044] For example, the virtual displays in the first device have a preset positional relationship, and the images displayed by at least one virtual display in the first device can be spliced ​​in pixel space according to the preset positional relationship to obtain the target image.

[0045] For example, Figure 2 This diagram illustrates a flowchart of a method for determining a preset positional relationship according to an exemplary embodiment of this disclosure. (See reference...) Figure 2 The method may include steps S210 to S230, wherein:

[0046] In step S210, a virtual canvas for carrying the target image is generated in advance.

[0047] For example, when the second device establishes a communication connection with the first device, it can send the first device the size of the target screen that meets the transmission protocol requirements of both devices and is needed by the second device. The first device can then pre-generate a virtual canvas that meets the size requirements of the target screen based on the received size information. Alternatively, the first device can directly generate a virtual canvas based on a pre-determined transmission protocol. For instance, if the first and second devices pre-agree that the transmitted screen size is 32:9, the first device can generate a 32:9 virtual canvas.

[0048] In step S220, the virtual canvas is divided into regions to obtain at least one sub-region.

[0049] In one exemplary implementation, each sub-region is used to carry the screen displayed by the virtual display associated with that sub-region.

[0050] For example, the first device can divide the virtual canvas according to the number of virtual displays that need to be generated. If the number of virtual displays that need to be generated is 3, and each virtual display is the same size, the virtual canvas can be divided into 3 equal parts to obtain 3 sub-regions. Each sub-region is associated with a virtual display, and each sub-region is used to display the image of the virtual display associated with that sub-region.

[0051] The number of virtual displays to be generated is determined according to the requirements. The size of each virtual display can be the same or different. This exemplary embodiment does not impose any special limitations on this.

[0052] In one exemplary implementation, the entire area of ​​the virtual canvas can be divided to obtain at least one sub-area, or a portion of the virtual canvas can be divided to obtain at least one sub-area. That is, the image displayed by the virtual monitor in the first device, after being stitched together, can occupy the entire virtual canvas or only a portion of it. Areas of the virtual canvas not occupied by the virtual monitor can display preset color blocks, such as white blocks.

[0053] When the image stitched together by the virtual displays occupies a portion of the virtual canvas, the image displayed on the virtual displays and the preset color blocks in the virtual canvas are stitched together to obtain the target image. That is, regardless of whether the image stitched together by the virtual displays occupies the entire area of ​​the virtual canvas, the size of the target image remains unchanged. For example, if the transmission protocol specifies a 32:9 aspect ratio for the image transmitted between the first and second devices, then the size of the target image will also be 32:9. In other words, the size of the image obtained after stitching together the image displayed on the virtual displays of the first device is less than or equal to the size of the target image.

[0054] In step S230, the preset positional relationship between each virtual display is determined based on the positional relationship between each sub-region.

[0055] For example, the preset positional relationship between each sub-region and each virtual monitor can be determined based on the positional relationship of each sub-region in the virtual canvas. For instance, if sub-region 1 corresponds to virtual monitor 1, sub-region 2 corresponds to virtual monitor 2, and sub-region 3 corresponds to virtual monitor 3, and sub-region 1 is located on the far left of the virtual canvas, sub-region 2 is located in the middle of the virtual canvas, and sub-region 3 is located on the far right of the virtual canvas, then the pixel data of virtual monitor 1 is stitched to the left of the pixel data of virtual monitor 2, and the pixel data of virtual monitor 2 is stitched to the left of the pixel data of virtual monitor 3.

[0056] In another exemplary implementation, instead of generating a virtual canvas, the positional relationships between the virtual displays can be directly customized to obtain a preset positional relationship. For example, by directly customizing virtual display 1 to the left of virtual display 2 and virtual display 2 to the left of virtual display 3, the images can be stitched together directly according to this preset positional relationship to obtain the target image.

[0057] It should be noted that in this disclosure, the target screen output by the first device is a holistic screen that meets the requirements of the existing transmission protocol between the first and second devices. This allows for multi-tasking display of the virtual screen without requiring any modification to the existing transmission protocol between the first and second devices.

[0058] For example, another exemplary implementation of step S110 may include: stitching together the images of all virtual displays in the first device to obtain the target image.

[0059] For example, if the first device has three virtual displays, then the first device can stitch together the images displayed on the three virtual displays to obtain the target image.

[0060] For example, another exemplary implementation of step S110 may include: selecting a target virtual display from the first device virtual display, and stitching the images of the target virtual display together to obtain a target image.

[0061] For example, a total of 5 virtual displays are created in the first device. 3 of the virtual displays show different windows of application 1, and 2 of the virtual displays show different windows of application 2. If the user now selects to browse application 1, then the 3 virtual displays that show the 3 different windows of application 1 can be used as target virtual displays. The images displayed on these 3 target displays are then stitched together to obtain the target image.

[0062] The selection rules for the target virtual display can be determined according to requirements, and this exemplary embodiment does not impose any special limitations on this.

[0063] For example, another exemplary implementation of step S110 may include: stitching together the image displayed on at least one virtual display in the first device and the image displayed on a physical display in the first device to obtain a target image.

[0064] For example, if the first device is equipped with a physical display, the images displayed by all the virtual displays in the first device and the images displayed by the physical display in the first device can be stitched together to obtain the target image. Alternatively, a target virtual display can be selected from the first device, and the images displayed by the target virtual display and the images displayed by the physical display in the first device can be stitched together to obtain the target image.

[0065] Similarly, the images displayed on at least one virtual display in the first device and the images displayed on the physical display in the first device can be spliced ​​together in pixel space according to a preset positional relationship to obtain the target image.

[0066] For example, in step S220 above, before dividing the virtual canvas, the display area corresponding to the physical display in the first device can be determined in the virtual canvas first, and then the other areas can be divided to obtain at least one sub-area, and then the sub-area can be associated with the virtual display.

[0067] Of course, if the first device is equipped with a physical display, it is also possible to stitch together only the images displayed on the virtual display. This exemplary embodiment does not impose any special limitations on this.

[0068] The following describes the specific implementation of "step S120, sending the target screen to the second device so that the second device can determine the display screen indicated by the field of view of the second device from the target screen and display the display screen".

[0069] In one exemplary embodiment, the first device and the second device can communicate via wired or wireless means, and this exemplary embodiment does not impose any particular limitation on this.

[0070] In one exemplary embodiment, the second device may include a wearable display device, such as VR glasses, AR glasses, MR glasses, etc.

[0071] Taking AR glasses as an example, the host device can send the target image to the AR glasses device in the form of a video stream or display signal.

[0072] In one exemplary implementation, the host device can output the complete ultrawide screen image, i.e., the target image obtained after splicing, at a fixed refresh rate to the AR glasses.

[0073] For example, a fixed refresh rate can be understood as the frequency at which the first device updates the target screen. It remains constant during a single use of the second device, that is, it remains constant from the time the second device is turned on until it is turned off. Of course, the user can adjust the refresh rate while using the second device, but it remains constant at least between two adjustments. The specific fixed refresh rate value can be determined according to the performance of the second device, and this exemplary embodiment does not impose any special limitations on it.

[0074] For example, when the first device is performing image stitching, if the update frequencies of the images being stitched, such as those displayed on multiple virtual displays, are inconsistent, the first device can use a frame interpolation algorithm to align the update frequencies of the images displayed on multiple virtual displays, so that the update frequencies of the stitched images are the same, thereby improving the user's viewing experience.

[0075] Of course, the first device may also not perform update frequency alignment processing, and this exemplary embodiment does not impose any special limitations on this.

[0076] For example, the second device determines the display screen indicating the field of view of the second device from the target screen in the following ways: the second device obtains the pose information of the second device according to the pose sensor in the second device, determines the field of view of the second device based on the pose information, and crops the screen within the field of view from the target screen to determine the display screen.

[0077] For example, the wearable display device in this disclosure is equipped with a pose sensor, which can be used to detect the current pose information of the wearable display device in real time. The pose sensor can be a three-degree-of-freedom pose sensor, a six-degree-of-freedom pose sensor, etc., and this exemplary embodiment does not make any special limitation in this regard. The second device can determine the current field of view of the second device based on the current pose detected by the pose sensor in the second device, and then directly crop the image belonging to the current field of view of the second device from the target image, use it as the display image, and display the display image on the second device.

[0078] For example, Figure 3 This diagram illustrates a flowchart of another method for a second device to determine a display screen according to an exemplary embodiment of this disclosure. (See also:) Figure 3 The method may include steps S310 to S320. Wherein: In step S310, the second device maps the target image into an arc-shaped image according to a preset curvature.

[0079] In one exemplary embodiment, the edge region of the curved image is offset outward along the horizontal direction relative to the center of the curved image according to a preset curvature. The preset curvature is determined according to target curvature adaptation parameters configured in the second device, the target curvature adaptation parameters including the target distance between the human eye and the curved image and / or the target size of the curved image.

[0080] For example, a head-mounted display device is worn on a person's eyes. The target distance between the human eye and the curved screen in the target curvature adaptation parameters can be understood as the viewing distance of the human eye when viewing the curved screen, and the target size of the curved screen can be understood as the size of the curved screen itself. Both the viewing distance and the size of the curved screen can be customized by the user according to their needs. Of course, they can also be left as default values ​​if no configuration is performed by the user.

[0081] For example, Figure 4 A flowchart illustrating a method for determining a preset curvature according to an exemplary embodiment of this disclosure is shown. (See reference...) Figure 4 The method may include steps S410 to S420. Wherein: In step S410, in response to the display adjustment operation on the second device, the target arc adaptation parameters indicated by the display adjustment operation are determined.

[0082] For example, after wearing the second device, users can adjust its display according to their viewing habits or needs, such as adjusting the viewing distance and the size of the virtual image. For instance, users can trigger a target control on the second device to adjust its display. After adjusting to their desired curvature adaptation parameters, they can confirm the adjustment. Based on the user's confirmation, the target curvature adaptation parameters are determined.

[0083] In step S420, the target curvature corresponding to the target arc adaptation parameter is determined according to the preset mapping relationship pre-configured in the second device, so as to obtain the preset curvature.

[0084] In one exemplary implementation, a preset mapping relationship is used to indicate the mapping relationship between different arc adaptation parameters and different curvatures.

[0085] For example, the optimal screen curvature for different viewing distances and screen sizes can be determined in advance through experiments or based on experience, thus obtaining a preset mapping relationship, which is then pre-configured in the second device. For instance, the optimal screen curvature is curvature 1 for a viewing distance of 1 meter and a screen size of 32:9, and curvature 2 for a viewing distance of 2 meters and a screen size of 32:9. After obtaining the target curvature adaptation parameters selected by the user, these parameters can be matched with the curvature adaptation parameters in the preset mapping relationship. The preset curvature is obtained based on the optimal screen curvature corresponding to the successfully matched curvature adaptation parameters. The optimal screen curvature can be understood as the curvature that avoids or suppresses trapezoidal distortion at the edge of the field of view.

[0086] Next, continue to refer to Figure 3 In step S320, the second device crops the image within its field of view from the curved image to determine the display image.

[0087] For example, the portion of the image within the current field of view of the second device can be cropped from the curved image to obtain the display image of the second device.

[0088] For example, head-mounted display devices have lenses with a fixed viewing angle. When projecting virtual images, the distance and angle between the edge of the image and the center of the lens are inconsistent. Furthermore, the flat image cannot adapt to the curved field of view of the human eye and the device's lens, resulting in trapezoidal stretching distortion at the image edges—either "narrower at the top and wider at the bottom" or "wider at the top and narrower at the bottom"—which is especially noticeable when observing the edges while rotating the posture. (Reference) Figure 5 This is a schematic diagram of a target image. Figure 6 To be Figure 5 The diagram illustrates the mapping of the target image to an arc-shaped image, i.e. Figure 6 This results in a complete curved virtual image. After mapping the target image as a curved image, the curvature of the curved image can match the field of view of the head-mounted display device's lens and the virtual space coordinates, making the distance and angle from each point on the image to the lens more consistent, avoiding the stretching and offset of the edges of the flat image, and directly offsetting trapezoidal distortion from the image shape.

[0089] Through steps S310 to S320 described above, the target image can be directly mapped into an arc-shaped image, that is, the spliced ​​ultra-wide screen virtual display can be mapped into an arc-shaped display structure. This ensures that the spliced ​​ultra-wide screen target image exists in an arc-shaped structure within the space of the second device. When the user turns their head to view different areas of the arc-shaped image, the second device determines the portion of the arc-shaped image within its current field of view based on its own posture information, crops this portion, and displays it on the second device. This ensures that the image within the second device's field of view is always displayed in a direction approximately directly facing the user's gaze, thus guaranteeing that its projection in the user's field of view remains rectangular and avoiding trapezoidal distortion. Simultaneously, through steps S310 to S320, complex processing such as perspective stretching of the target image is unnecessary, thus avoiding trapezoidal distortion and improving display efficiency while ensuring display accuracy.

[0090] Through the above steps S110 to S120, while realizing multi-task display of virtual screens, the host only needs to be responsible for the unified rendering and splicing output of the content of multiple virtual displays, which reduces the computing load of the host and facilitates cross-platform deployment and system capability expansion.

[0091] For example, Figure 7 A flowchart illustrating another display method according to an exemplary embodiment of this disclosure is shown. (See reference...) Figure 7 The method may include steps S710 to S730. Specifically: in step S710, a target image sent by a first device is received, the target image being obtained by stitching together images displayed on at least one virtual display in the first device; in step S720, the field of view of the second device is determined based on the posture information of the second device, so as to determine the display image indicated by the field of view from the target image; in step S730, the display image is displayed.

[0092] For example, the specific implementation of step S710 can be referred to the relevant content in step S110 above, and will not be repeated here.

[0093] In one exemplary embodiment, determining the field of view of the second device based on the posture information of the second device includes: determining the field of view of the second device based on the posture information of the second device and the preset viewing angle information of the second device.

[0094] The preset viewing angle information of the second device refers to the physical attributes of the second device. It can be understood as a set of fixed viewing angle parameters bound to the hardware characteristics and display requirements of the second device. It does not change with the real-time posture of the device and can be used to constrain the size of the field of view. It can include field of view parameters, such as horizontal field of view or vertical field of view.

[0095] For example, the current line of sight of the second device can be determined based on the current posture information of the second device. Based on the current line of sight of the second device and the preset viewing angle information of the second device, the field of view boundary of the second device can be determined, thereby obtaining the current field of view range of the second device.

[0096] In one exemplary embodiment, the target image in this disclosure has a fixed spatial anchor point. Based on this, exemplaryly, Figure 8 This diagram illustrates a flowchart of another method for determining a display screen according to an exemplary embodiment of the present disclosure. (See also:) Figure 8 The method may include steps S810 to S820. Wherein: In step S810, the visible area of ​​the second device in the target image is determined based on the fixed spatial anchor point and the field of view.

[0097] For example, the second device can use its initial viewpoint as a reference to generate a curved virtual display surface (which can be understood as a virtual display on the second device, specifically an ultra-wide virtual display) in the horizontal direction based on a fixed spatial anchor point. The target image is then mapped onto this curved virtual display surface, thus obtaining a curved image. Since the curved image is generated based on a fixed spatial anchor point, it will not change with the pose of the second device; essentially, the virtual curved display surface is fixed in three-dimensional space.

[0098] The initial viewpoint can be understood as the center of vision corresponding to the pose of the second device when it is powered on. For example, if the target distance is 2 meters and the viewing size is 32:9, the spatial point 2 meters away from the second device at the center of vision corresponding to the pose of the second device when it is powered on can be used as the center point of the virtual curved surface, and a virtual curved surface of size 32:9 can be generated according to the preset curvature.

[0099] Spatial anchor points can anchor the aforementioned field of view to the arc-shaped screen in three-dimensional space, aligning the field of view with the coordinates of the fixed screen, thereby determining the visible area of ​​the second device's field of view within the arc-shaped screen.

[0100] In step S820, the display screen is determined based on the screen within the visible area of ​​the target screen.

[0101] For example, the second device can crop the image within its current visible area from the curved image to obtain the display image.

[0102] When the posture of the second device changes, the visible area of ​​the second device in the curved screen also changes, and the cropped display screen also changes. In this way, the user can see the screens displayed by different virtual displays in the first device in the second device, thereby realizing multi-tasking virtual display.

[0103] For example, other implementation details in steps S710 to S730 described above can be referred to the above description. Figure 1 The specific implementation methods are not described in detail here.

[0104] For example, Figure 9 This diagram illustrates the architecture of a display system according to an exemplary embodiment of the present disclosure. (Reference) Figure 9 The display system may include: a first device 910, used to create at least one virtual display and stitch together the images displayed on the at least one virtual display to obtain a target image; and a second device 920, used to determine the display image indicated by the field of view of the second device from the target image and to display the display image.

[0105] In one exemplary embodiment, determining the display screen indicating the field of view of the second device from the target screen and displaying the display screen includes: mapping the target screen into an arc-shaped screen according to a preset curvature; cropping the screen within the field of view of the second device from the arc-shaped screen to determine the display screen; the edge region of the arc-shaped screen is deflected outward along the horizontal direction relative to the center of the arc-shaped screen according to the preset curvature; the preset curvature is determined according to target arc adaptation parameters configured in the second device, the target arc adaptation parameters including the target distance between the human eye and the arc-shaped screen and / or the target size of the arc-shaped screen.

[0106] Other specific implementations of the first and second devices can refer to the above description. Figure 1 and Figure 7 The specific implementation methods for the corresponding parts are not described in detail here.

[0107] Taking the second device equipped with a three-degree-of-freedom attitude sensor as an example, in this disclosure, multiple logically independent virtual displays can be created on the host side, and windows of different applications can be displayed on different virtual displays. Then, the host will stitch the images displayed on the multiple virtual displays into a complete and continuous ultra-wide screen image, namely the target image mentioned above. The image is transmitted to the glasses end through the native three-degree-of-freedom method. The glasses end then determines its visible image in the target image based on its own attitude information and its own preset viewing angle parameters, thereby cropping the visible image to obtain the display image, and displaying the display image in the glasses.

[0108] The method disclosed herein enables multi-tasking display without multiple interactions between the host and the glasses, reducing latency in multi-tasking display on the virtual screen. This reduced latency leads to more stable visuals, improving both image stability and spatial consistency. Furthermore, the host only needs to handle unified rendering and ultra-widescreen output of content from multiple virtual displays; posture perception, display determination, and screen fixation are all autonomously handled by the glasses based on native 3DoF (three degrees of freedom) capabilities. This reduces the computational load on the host, making the display system easily deployable across platforms and expandable.

[0109] Furthermore, since the ultra-wide screen display content output by the host forms an arc-shaped display effect around the user in space, rather than being concentrated on a single plane, there is no need for perspective stretching, which can avoid trapezoidal distortion and further enhance the user's viewing experience.

[0110] The methods disclosed herein can be applied to extended reality devices such as augmented reality glasses, virtual reality glasses, and mixed reality glasses. Specific application scenarios include, but are not limited to: virtual desktops and productivity systems, multi-screen office and remote collaboration scenarios, immersive entertainment and games, and industrial control and professional display systems.

[0111] Furthermore, it should be noted that the above figures are merely illustrative representations of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0112] Furthermore, exemplary embodiments of this disclosure also provide a display device applied to a first device. (See reference...) Figure 10 As shown, the display device includes the following program modules: a splicing module 1010, configured to splice the images displayed on at least one virtual display in the first device to obtain a target image; and a sending module 1020, configured to send the target image to a second device so that the second device can determine the display image indicated by the field of view of the second device from the target image and display the display image.

[0113] In one exemplary embodiment, the splicing module 1010 can be specifically used to splice the pixel data corresponding to the screen displayed by at least one virtual display in the first device to obtain the target screen.

[0114] In one exemplary embodiment, stitching together pixel data corresponding to the images displayed on at least one virtual display in the first device to obtain a target image includes: stitching together pixel data corresponding to the images displayed on at least one virtual display in the first device according to a preset positional relationship between the virtual displays in the first device to obtain a target image; wherein, the method for determining the preset positional relationship includes at least: pre-generating a virtual canvas for carrying the target image; dividing the virtual canvas into regions to obtain at least one sub-region, wherein each sub-region is used to carry the image displayed on the virtual display associated with the sub-region; and determining the preset positional relationship between the virtual displays according to the positional relationship between the sub-regions.

[0115] In one exemplary embodiment, the splicing module 1010 can also be specifically used to splice the image displayed on at least one virtual display in the first device and the image displayed on a physical display in the first device to obtain a target image.

[0116] In one exemplary embodiment, the method by which the second device determines the display screen indicating the field of view of the second device from the target screen includes: the second device acquiring the pose information of the second device based on the pose sensor in the second device, determining the field of view of the second device based on the pose information, and cropping the screen within the field of view from the target screen to determine the display screen.

[0117] In one exemplary embodiment, the second device includes a head-mounted display device. The second device determines the display screen based on the field of view indication of the second device from the target image by: mapping the target image into an arc-shaped image according to a preset curvature, and cropping the image within the field of view from the arc-shaped image according to the field of view of the second device to determine the display screen; the edge region of the arc-shaped image is deflected outward relative to the center of the arc-shaped image along the horizontal direction according to the preset curvature; the preset curvature is determined according to target arc adaptation parameters configured in the second device, the target arc adaptation parameters including the target distance between the human eye and the arc-shaped image and / or the target size of the arc-shaped image.

[0118] Furthermore, exemplary embodiments of this disclosure also provide another display device applied to a second device. (See reference...) Figure 11As shown, the display device includes the following program modules: a receiving module 1110, configured to receive a target image sent by a first device, the target image being obtained by stitching together images displayed by at least one virtual display in the first device; a display image determination module 1120, configured to determine the field of view of the second device based on the posture information of the second device, so as to determine the display image indicated by the field of view from the target image; and a display module 1130, configured to display the display image.

[0119] In one exemplary embodiment, determining the display screen indicating the field of view from the target screen includes: mapping the target screen into an arc-shaped screen according to a preset curvature; cropping the screen within the field of view of the second device from the arc-shaped screen to determine the display screen; the edge region of the arc-shaped screen is deflected outward relative to the center of the arc-shaped screen along the horizontal direction according to the preset curvature; the preset curvature is determined according to target arc adaptation parameters configured in the second device, the target arc adaptation parameters including the target distance between the human eye and the arc-shaped screen and / or the target size of the arc-shaped screen.

[0120] In one exemplary embodiment, the method for determining the preset curvature includes: responding to a display adjustment operation on the second device, determining the target arc adaptation parameter indicated by the display adjustment operation; determining the target curvature corresponding to the target arc adaptation parameter according to a preset mapping relationship pre-configured in the second device, so as to obtain the preset curvature; wherein, the preset mapping relationship is used to indicate the mapping relationship between different arc adaptation parameters and different curvatures.

[0121] In one exemplary embodiment, the target screen has a fixed spatial anchor point; determining the field of view of the second device based on the posture information of the second device includes: determining the field of view of the second device based on the posture information of the second device and the preset viewing angle information of the second device; determining the display screen indicating the field of view from the target screen includes: determining the visible area of ​​the second device in the target screen based on the fixed spatial anchor point and the field of view; determining the display screen based on the screen in the target screen that is within the visible area.

[0122] The specific details of each part of the above-mentioned device have been described in detail in the method section of the implementation plan. For any undisclosed details, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0123] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0124] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0125] Exemplary embodiments of this disclosure also provide a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the display method described above.

[0126] In one implementation, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing a computer program, such as read-only memory, NAND flash memory, etc.

[0127] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.

[0128] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, C++, and Python. Program code can execute entirely on the user's computing device, partially on the user's computing device, or as a standalone software package. It can also execute partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0129] Computer programs can be carried or transmitted via signals such as electricity, magnetism, light, electromagnetic fields, and infrared radiation. Electronic devices can convert the signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, to execute by the processor of the electronic device) the method steps of various exemplary embodiments of this disclosure, such as the display method described above.

[0130] Exemplary embodiments of this disclosure also provide a terminal device, which can be understood as the first device described above, and may include a laptop computer, a desktop computer, a smartphone, a smart wearable device, such as a smartwatch, a smart neckband, etc.

[0131] For example, a terminal device may include a processor and a memory. The memory stores executable instructions for the processor, such as a computer program. The processor executes these executable instructions to perform the steps of the methods disclosed in various exemplary embodiments applied to the first device. Furthermore, the terminal device may also include a physical display for displaying a graphical user interface. Of course, the terminal device may also not include a physical display, and this exemplary embodiment does not specifically limit this.

[0132] The following is for reference. Figure 12 The terminal device is illustrated by way of a general-purpose computing device. It should be understood that... Figure 12 The electronic device 1200 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0133] like Figure 12 As shown, the electronic device 1200 may include: a processor 1210, a memory 1220, a bus 1230, an I / O (input / output) interface 1240, a network adapter 1250, and a display 1260.

[0134] Memory 1220 may include volatile memory, such as RAM 1221 and cache unit 1222, and may also include non-volatile memory, such as ROM 1223. Memory 1220 may also include one or more program modules 1224, such program modules 1224 including, but not limited to: operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. For example, program module 1224 may include the modules in the above-described apparatus.

[0135] The processor 1210 may include one or more processing units, such as an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor, and / or an NPU (Neural-Network Processing Unit).

[0136] The processor 1210 can be used to execute executable instructions stored in the memory 1220, such as the display method described above.

[0137] Bus 1230 is used to connect different components of electronic device 1200, and may include data bus, address bus and control bus.

[0138] Electronic device 1200 can communicate with one or more external devices 1300 (such as keyboard, mouse, external controller, etc.) through I / O interface 1240.

[0139] Electronic device 1200 can communicate with one or more networks via network adapter 1250. For example, network adapter 1250 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication. Network adapter 1250 can communicate with other modules of electronic device 1200 via bus 1230.

[0140] Electronic device 1200 can display a graphical user interface, such as a virtual display, through monitor 1260.

[0141] although Figure 12Other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, may also be configured in the electronic device 1200.

[0142] Exemplary embodiments of this disclosure also provide an extended reality device, which can be understood as the second device described above. This device may include wearable display devices, such as AR glasses, VR glasses, MR glasses, and other smart glasses. The display device may include a processor, a memory, and a display device. The memory stores executable instructions for the processor, such as computer programs. The processor executes these executable instructions to perform the method steps of the various exemplary embodiments of this disclosure applied to the second device. When the processor executes the display method applied to the second device, it drives the display device by outputting display control signals, and the display device displays the obtained display screen under the control of the processor.

[0143] Those skilled in the art will understand that various aspects of this disclosure can be implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be referred to as "circuit", "module" or "system" respectively.

[0144] It should be understood that this disclosure is not limited to the specific methods, steps, or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. Those skilled in the art will readily conceive of other embodiments based on the specific implementations provided in this disclosure. Therefore, the specific implementations provided in this disclosure are merely exemplary, and the scope and spirit of this disclosure are indicated by the claims, and should cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary technical means in the art not disclosed in this disclosure.

Claims

1. A display method, characterized in that, Applied to a first device, the method includes: The images displayed on at least one virtual display in the first device are stitched together to obtain the target image; The target image is sent to the second device, so that the second device can determine the display image indicated by the field of view of the second device from the target image and display the display image.

2. The display method according to claim 1, characterized in that, The step of stitching together the images displayed on at least one virtual display in the first device to obtain the target image includes: The pixel data corresponding to the images displayed on at least one virtual display in the first device are stitched together to obtain the target image.

3. The display method according to claim 2, characterized in that, The step of stitching together the pixel data corresponding to the images displayed on at least one virtual display in the first device to obtain the target image includes: Based on the preset positional relationship between the virtual displays in the first device, the pixel data corresponding to the images displayed by at least one virtual display in the first device are stitched together to obtain the target image; The method for determining the preset positional relationship includes at least the following: A virtual canvas is pre-generated to hold the target image; The virtual canvas is divided into regions to obtain at least one sub-region, wherein each sub-region is used to carry the screen displayed by the virtual display associated with the sub-region; Based on the positional relationship between each sub-region, the preset positional relationship between each virtual display is determined.

4. The display method according to claim 1, characterized in that, The step of stitching together the images displayed on at least one virtual display in the first device to obtain the target image includes: The target image is obtained by stitching together the images displayed on at least one virtual display in the first device and the images displayed on the physical display in the first device.

5. The display method according to claim 1, characterized in that, The method by which the second device determines the display screen indicating the field of view of the second device from the target screen includes: The second device acquires the pose information of the second device based on the pose sensor in the second device, determines the field of view of the second device based on the pose information, and crops the image within the field of view from the target image to determine the display image.

6. The display method according to claim 1, characterized in that, The second device includes a head-mounted display device, and the second device, based on a display screen indicating the field of view of the second device determined from the target screen, includes: The second device maps the target image into an arc-shaped image according to a preset curvature, and crops the image within the field of view from the arc-shaped image according to the field of view of the second device, so as to determine the display image; The edge region of the arc-shaped image is deflected outward along the horizontal direction relative to the center of the arc-shaped image according to the preset curvature; The preset curvature is determined based on the target arc adaptation parameters configured in the second device, which include the target distance between the human eye and the arc image and / or the target size of the arc image.

7. A display method, characterized in that, Applied to a second device, the method includes: The first device receives a target image sent by the first device, the target image being obtained by stitching together images displayed on at least one virtual display in the first device; The field of view of the second device is determined based on the posture information of the second device, so as to determine the display screen indicating the field of view from the target screen; The aforementioned display screen is then shown.

8. The display method according to claim 7, characterized in that, The display screen from which the field of view indicator is determined from the target image includes: The target image is mapped into an arc-shaped image according to a preset curvature; The display screen is determined by cropping the image within the field of view of the second device from the arc-shaped image. The edge region of the arc-shaped image is deflected outward along the horizontal direction relative to the center of the arc-shaped image according to the preset curvature; The preset curvature is determined based on the target arc adaptation parameters configured in the second device, which include the target distance between the human eye and the arc image and / or the target size of the arc image.

9. The display method according to claim 8, characterized in that, The method for determining the preset curvature includes: In response to a display adjustment operation on the second device, the target arc adaptation parameters indicated by the display adjustment operation are determined; Based on the preset mapping relationship pre-configured in the second device, the target curvature corresponding to the target arc adaptation parameter is determined to obtain the preset curvature; The preset mapping relationship is used to indicate the mapping relationship between different arc adaptation parameters and different curvatures.

10. The display method according to claim 7, characterized in that, The target image has fixed spatial anchor points; Determining the field of view of the second device based on its attitude information includes: The field of view of the second device is determined based on the attitude information and preset viewing angle information of the second device. The display screen from which the field of view indicator is determined from the target image includes: The visible area of ​​the second device in the target image is determined based on the fixed spatial anchor point and the field of view. The display screen is determined based on the screen within the visible area of ​​the target screen.

11. A display system, characterized in that, include: The first device is used to create at least one virtual display and stitch together the images displayed on the at least one virtual display to obtain the target image; The second device is used to determine the display screen indicating the field of view of the second device from the target screen, and to display the display screen.

12. The display system according to claim 11, characterized in that, The step of determining the display screen indicating the field of view of the second device from the target screen and displaying the display screen includes: The target image is mapped into an arc-shaped image according to a preset curvature; The display screen is determined by cropping the image within the field of view of the curved image from the second device's field of view. The edge region of the arc-shaped image is deflected outward along the horizontal direction relative to the center of the arc-shaped image according to the preset curvature; The preset curvature is determined based on the target arc adaptation parameters configured in the second device, which include the target distance between the human eye and the arc image and / or the target size of the arc image.

13. A display device, characterized in that, Applied to a first device, the device includes: The splicing module is configured to splice the images displayed on at least one virtual display in the first device to obtain the target image; The sending module is configured to send the target screen to a second device, so that the second device can determine the display screen indicated by the field of view of the second device from the target screen and display the display screen.

14. A display device, characterized in that, Applied to a second device, the device includes: The receiving module is configured to receive a target image sent by the first device, the target image being obtained by stitching together images displayed by at least one virtual display in the first device; The display screen determination module is configured to determine the field of view of the second device based on the posture information of the second device, so as to determine the display screen indicated by the field of view from the target screen; The display module is configured to display the display screen.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 10.

16. A terminal device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 6.

17. An extended reality device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 7 to 10; The display device is used to display the screen under the control of the processor.

Images