Display method, readable storage medium, and electronic device

Abstract

The present application relates to the technical field of image processing, and discloses a display method, a readable storage medium, and an electronic device. In the method, a first electronic device can acquire a field of gaze of a user of the first electronic device, and send to a second electronic device user field of view information corresponding to the field of gaze. According to the field of gaze indicated by the user field of view information, the second electronic device can generate a local field of view image corresponding to the field of gaze, and send to the first electronic device the local field of view image. Then, the first electronic device can, on the basis of the local field of view image and a size of a field of display of the first electronic device, render and display an image corresponding to at least part of a field of view in the local field of view image. As such, the second electronic device does not need to send to the first electronic device a panoramic image of the whole field of view, thus facilitating reduction in resources required for transmitting a panoramic image between the first electronic device and the second electronic device.

Description

Display method, readable storage medium and electronic device

[0001] The present application claims priority to the Chinese patent application No. 202410468692.6, filed on April 17, 2024, and entitled "Display method, readable storage medium and electronic device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of image processing, and in particular, to a display method, a readable storage medium and an electronic device. BACKGROUND

[0003] With the development of panorama technology, more and more electronic devices can realize the collection or display of panorama images (such as panorama pictures, panorama videos, etc.). For example, in a live broadcast scenario, an electronic device (hereinafter referred to as a source device) providing a panorama image can transmit the panorama image collected by itself or an image collection device to a display device for display by the display device.

[0004] A panorama image usually has a maximum field of view of 360° in the horizontal direction and a maximum field of view of 180° in the vertical direction. The source device can transmit a panorama image with a full field of view (such as a panorama image with a maximum field of view range that the source device can provide, hereinafter referred to as a full-view image) collected or stored or received to the display device. The display device can select a field of view range corresponding to a field of view area gazed by a user from the full-view image according to the size of the field of view range of the display device for display. That is, although the display device only displays part of the panorama image at one time, the source device still needs to transmit the full-view image to the display device, increasing the network resources occupied in the process of transmitting the panorama image. SUMMARY

[0005] Therefore, the present application provides a display method, a readable storage medium and an electronic device.

[0006] In a first aspect, a display method is provided, applied to a first electronic device, and the method includes: sending first field of view information to a second electronic device, wherein the first field of view information indicates a first field of view gazed by a user of the first electronic device; receiving a local field of view image sent by the second electronic device, wherein a field of view range corresponding to the local field of view image includes the first field of view, and the local field of view image is a local image of a corresponding first panorama image; and displaying at least part of an image area in the local field of view image, wherein the at least part of the image area includes an image area corresponding to the first field of view.

[0007] In the method, the first electronic device can send first field of view information indicating a first field of view of the user's gaze (e.g., the user's gaze field of view hereinafter) to the second electronic device, so that the second electronic device sends only a partial field of view image corresponding to the first field of view to the first electronic device instead of a full field of view image. In this way, the network resources occupied in transmission of the panoramic image between the first electronic device and the second electronic device can be reduced.

[0008] Optionally, the first panoramic image can be a panoramic image data collected by the second electronic device, or stored by the second electronic device, or received by the second electronic device from a third electronic device. The panoramic image data can be raw image data collected by a panoramic camera, or image data processed from the raw image data collected by the panoramic camera.

[0009] In a possible implementation of the first aspect, the first field of view information includes at least one of the following: a range of the first field of view; a field of view center of the first field of view; a head pose of the user; a line of sight direction of the user.

[0010] Optionally, the range of the first field of view can be indicated in any of the following ways: in a case where the first field of view is a polygon, using vertices of the polygon; in a case where the first field of view is a circle, using a radius of the circle; in a case where the first field of view is an ellipse, using lengths of a major axis and a minor axis of the ellipse.

[0011] In a possible implementation of the first aspect, a field of view range of the partial field of view image in a horizontal direction is greater than or equal to 124° and less than 360°.

[0012] Since the field of view range that can be clearly perceived by the human eye in the horizontal direction is about 124°, the partial field of view image with a field of view range greater than or equal to 124° and less than 360° in the horizontal direction can ensure that the image seen by the user is clear, and can also reduce the network resources occupied in transmission of the panoramic image.

[0013] In a possible implementation of the first aspect, the partial field of view image includes a first partial field of view image and at least one second partial field of view image, where the first partial field of view image corresponds to a first field of view region at the center of the first field of view, the second partial field of view image corresponds to an annular field of view region surrounding the first field of view region, the angular resolution of the first partial field of view image is greater than that of the second partial field of view image, and the angular resolution of the second partial field of view image decreases with an increase in the distance between the environment field of view region corresponding to the second partial field of view image and the center of the first field of view; and the displaying of at least part of the image region in the partial field of view image includes: rendering display content of the first field of view region based on the first partial field of view image, and rendering display content of the annular field of view region corresponding to the second partial field of view image based on the second partial field of view image, to obtain a third partial field of view image; and displaying the third partial field of view image.

[0014] In the method, the first electronic device can receive multiple local field-of-view images corresponding to one frame of image, and the multiple local field-of-view images have different angular resolutions (decreasing with the increase of the distance from the center of the first field of view), and the first electronic device can render the image content of the corresponding field of view based on the multiple local field-of-view images. In this way, it can be ensured that in the third local field-of-view image displayed by the first electronic device, the closer to the center, the more texture information retained by the image content, the clearer.

[0015] In a second aspect, a display method is provided, applied to a second electronic device, and the method comprises: receiving first field-of-view information sent by a first electronic device, wherein the first field-of-view information indicates a first field of view at which a user of the first electronic device gazes; and sending a local field-of-view image to the first electronic device, wherein the local field-of-view image corresponds to a field-of-view range including the first field of view, and the local field-of-view image is a partial image of a first panoramic image.

[0016] In the method, after receiving the first field-of-view information sent by the first electronic device, the second electronic device can send only a local field-of-view image corresponding to the first field of view to the first electronic device, instead of a full field-of-view image. In this way, the network resources occupied by the first electronic device and the second electronic device when transmitting the panoramic image can be reduced.

[0017] In a possible implementation of the second aspect, the first field-of-view information comprises at least one of the following information: a range of the first field of view; a field-of-view center of the first field of view; a head pose of the user; a line-of-sight direction of the user.

[0018] In a possible implementation of the second aspect, the local field-of-view image has a horizontal field-of-view range greater than or equal to 124° and less than 360°.

[0019] Since the human eye can clearly perceive a field of view range of about 124° in the horizontal direction, the local field-of-view image having a horizontal field-of-view range greater than or equal to 124° and less than 360° can ensure that the image seen by the user is clear, and can also reduce the network resources occupied by the transmission of the panoramic image.

[0020] In a possible implementation of the second aspect, the partial field of view image includes a first partial field of view image and at least one second partial field of view image, the first partial field of view image corresponds to a first field of view region at the center of the first field of view, the second partial field of view image corresponds to a ring-shaped field of view region surrounding the first field of view region, the first partial field of view image has a higher angular resolution than the second partial field of view image, and the angular resolution of the second partial field of view image decreases as the distance between the field of view region corresponding to the second partial field of view image and the center of the first field of view increases.

[0021] In the method, the first electronic device can send multiple partial field of view images corresponding to one frame of image, and the multiple partial field of view images have different angular resolutions (decrease as the distance from the center of the first field of view increases), so that the first electronic device can render the image content of the corresponding field of view based on the multiple partial field of view images. In this way, it can be ensured that the closer to the center in the third partial field of view image displayed by the first electronic device, the more texture information retained by the image content, the clearer the image, and the bandwidth occupied by the transmission of the panoramic image can be reduced.

[0022] In a possible implementation of the second aspect, the angular resolution of the first partial field of view image is less than or equal to the angular resolution of the first panoramic image.

[0023] In a possible implementation of the second aspect, the partial field of view image is obtained by sampling the first panoramic image by the second electronic device.

[0024] In a possible implementation of the second aspect, the partial field of view image includes a second field of view region at the center of the partial field of view image, and at least one ring-shaped field of view region surrounding the second field of view region, the content of the second field of view region is generated based on image data with a first sampling density, and the image data used to generate the content of the at least one ring-shaped field of view region has a sampling density less than the first sampling density.

[0025] In the implementation, the image content of different fields of view of the partial field of view image is generated based on original image data with different sampling densities (for example, angular resolutions), and the farther from the center of the first field of view, the lower the sampling density of the original image data used, so that the image content of the region at the center of the partial field of view image (equivalent to the center of the first field of view) can retain more texture, and the clarity of the image of the center region can be improved.

[0026] In a possible implementation of the second aspect, the sampling density of the image data used to generate the content of the at least one ring-shaped field of view region decreases as the distance between the ring-shaped field of view region and the second field of view region increases.

[0027] In a third aspect, a display method is provided. The method comprises: a first electronic device sending first field of view information to a second electronic device, wherein the first field of view information indicates a first field of view at which a user of the first electronic device gazes; the second electronic device sending a partial field of view image to the first electronic device in response to the first field of view information, wherein a field of view range corresponding to the partial field of view image includes the first field of view, and the partial field of view image is a partial image of a corresponding first panoramic image; and the first electronic device displaying at least part of an image region in the partial field of view image, wherein the at least part of the image region includes an image region corresponding to the first field of view.

[0028] In a fourth aspect, an electronic device is provided. The electronic device comprises: a memory configured to store one or more programs; and a processor configured to execute the one or more programs to cause the electronic device to implement the display method of the first aspect and any possible implementation of the first aspect, or the display method of the second aspect and any possible implementation of the second aspect, or the display method of the third aspect and any possible implementation of the third aspect.

[0029] In a fifth aspect, a readable storage medium is provided. The readable storage medium comprises one or more programs. When the one or more programs are executed on an electronic device, the one or more programs cause the electronic device to implement the display method of the first aspect and any possible implementation of the first aspect, or the display method of the second aspect and any possible implementation of the second aspect, or the display method of the third aspect and any possible implementation of the third aspect.

[0030] In a sixth aspect, a program product is provided. When the program product is run on an electronic device, the program product causes the electronic device to implement the display method of the first aspect and any possible implementation of the first aspect, or the display method of the second aspect and any possible implementation of the second aspect, or the display method of the third aspect and any possible implementation of the third aspect.

[0031] It should be understood that the beneficial effects of the third aspect to the sixth aspect described above can refer to the beneficial effects described in the first aspect and the second aspect, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0032] FIG. 1 shows a schematic diagram of the principle of equirectangular projection, according to some embodiments of the present application.

[0033] FIG. 2 shows a schematic diagram of a panoramic image transmission and display scenario, according to some embodiments of the present application.

[0034] FIG. 3A shows a schematic diagram of a user gazing at different fields of view at different times, according to some embodiments of the present application.

[0035] FIG. 3B shows a schematic diagram of a source device transmitting a partial field of view image to a display device, according to some embodiments of the present disclosure.

[0036] FIG. 3C shows another schematic diagram of a source device transmitting a partial field of view image to a display device, according to some embodiments of the present disclosure.

[0037] FIG. 4A shows a schematic diagram of a horizontal viewing angle of a human eye, according to some embodiments of the present disclosure.

[0038] FIG. 4B shows a schematic diagram of a vertical viewing angle of a human eye, according to some embodiments of the present disclosure.

[0039] FIG. 5A shows a schematic diagram of a sampling density, according to some embodiments of the present disclosure.

[0040] FIG. 5B shows a schematic diagram of a source device obtaining a partial field of view image based on a full field of view image, according to some embodiments of the present disclosure.

[0041] FIG. 5C shows a schematic diagram of a display device obtaining display content based on multiple partial field of view images, according to some embodiments of the present disclosure.

[0042] FIG. 6 shows a flowchart of a display method, according to some embodiments of the present disclosure.

[0043] FIG. 7A shows a schematic diagram of a partial field of view image unfolding, according to some embodiments of the present disclosure.

[0044] FIG. 7B shows another schematic diagram of a partial field of view image unfolding, according to some embodiments of the present disclosure.

[0045] FIG. 7C shows another schematic diagram of a partial field of view image unfolding, according to some embodiments of the present disclosure.

[0046] FIG. 8 shows a process diagram of a display method, according to some embodiments of the present disclosure.

[0047] FIG. 9 shows a structural diagram of a VR glasses 10, according to some embodiments of the present disclosure.

[0048] FIG. 10 shows a structural diagram of a server 30, according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0049] Illustrative embodiments of the present disclosure include, but are not limited to, display methods, readable storage media, and electronic devices.

[0050] To facilitate understanding, first introduce the terms related to the embodiments of the present disclosure.

[0051] equirectangular projection (ERP)

[0052] ERP, also known as equirectangular projection, equi-rectangular cylindrical projection, square projection, etc., is a projection method of projecting a spherical surface to a cylindrical surface (or projecting a cylindrical surface to a spherical surface). As shown in FIG. 1, the spherical surface is tangent to the cylindrical surface at the equator, the longitude and latitude on the spherical surface are equal, the longitude is projected as a set of parallel straight lines, the latitude is projected as another set of parallel straight lines, and the straight lines corresponding to the longitude and the straight lines corresponding to the latitude are perpendicular to each other, that is, the longitude and latitude grid forms a square grid on the corresponding unfolded plane of the cylindrical surface. Assuming that the radius of the spherical surface is R, the width of the unfolded rectangular surface of the cylindrical surface is w (for example, 2πR, R being the radius of the spherical surface or the radius of the cylindrical surface), and the height of the unfolded rectangular surface of the cylindrical surface is h (for example, πR), a rectangular coordinate system is established with the left side of the rectangular surface as the Y-axis and the line passing through the midpoint of the left side of the rectangular surface and perpendicular to the Y-axis as the X-axis, and a spherical coordinate system is established with the center of the spherical surface as the origin. Continuing to refer to FIG. 1, the longitude on the spherical surface corresponds to the Y-axis, and the straight line corresponding to the point with θ = 0° on the spherical surface corresponds to the X-axis, so the coordinates of any point (x, y) in the unfolded rectangular surface of the cylindrical surface in the spherical coordinate system are The panoramic image based on the ERP has the same amount of data sampled on each latitude line. Since the latitude closer to the equator (θ = 0°) has a longer circumference, the amount of data per unit length on the latitude closer to the equator is smaller, that is, the sampling density is smaller.

[0053] Based on the above description, any point on the spherical surface of the panoramic image can be represented by , wherein is used to indicate the angle in the horizontal direction, and θ is used to indicate the angle in the vertical direction, (which can also be other intervals indicating a 360° field of view, such as 0° to 360°), (which can also be other intervals indicating a 180° field of view, such as 0° to 180°). Generally, the ERP panoramic image is unfolded according to a fixed coordinate, and the center of the rectangular image unfolded from the cylindrical surface is usually a fixed position. For example, in The center of the rectangular image unfolded from the cylindrical surface can be (180°, 0°) (which can also be other values), and in The center of the rectangular image unfolded from the cylindrical surface can be (0°, 0°) (which can also be other values).

[0054] Based on the above description, the field of view size can be represented by A° x B°, which indicates that the field of view size in the horizontal direction is A° and the field of view size in the vertical direction is B°.

[0055] ​It should be noted that the display method provided in the embodiments of the present application can be applied to any display device and source device. Exemplarily, the display device can be any device or apparatus that can display a panoramic image, including but not limited to a virtual reality (VR) device, an augmented reality (AR) device, a mix reality (MR) device, a head up display (HUD) device, a system or apparatus that combines multiple devices to display a panoramic image, and the like. Exemplarily, the source device can be any device that can process and / or collect a panoramic image, including but not limited to a user equipment (UE), a mobile station (MS), a mobile terminal (MT), a tablet computer (Pad), a laptop computer, an industrial control device, a car machine, a device in a smart grid, a terminal device in transportation safety, a device in a smart city, a camera, a panoramic camera, a server, or a system or apparatus that combines multiple devices to collect or process a panoramic image, and the like. For ease of description, the following describes a display device as a VR glasses 10, a source device as a server 30, and a panoramic image as a panoramic image collected by a panoramic camera 20 and transmitted to the server 30.

[0056] It should be noted that the full-view image is a panoramic image with a maximum field of view range that can be provided by the source device. For example, if the panoramic image stored or collected by the source device has a field of view size of 360°x180°, the full-view image corresponds to a field of view with a size of 360°x180°; if the panoramic image stored or collected by the source device has a field of view range of 360°x120°, the full-view image corresponds to a field of view with a size of 360°x120°.

[0057] It should be noted that the method provided in the embodiments of the present application can be used in devices or systems that project in any way, including but not limited to an ERP, an equi-angular cubemap (EAC) format, and the like. The following describes an example of the ERP format.

[0058] The technical solutions of the present application are described below with reference to the accompanying drawings.

[0059] In some embodiments, although the display device will only display a panoramic image of part of the field of view at one time, the source device still needs to transmit the full-view image to the display device, increasing the network resources occupied in the process of transmitting the panoramic image.

[0060] Exemplarily, FIG. 2 shows a schematic diagram of panoramic image transmission and display according to some embodiments of the present application.

[0061] As shown in FIG. 2, the panoramic camera 20 (as an image acquisition device) can acquire a full-view image and transmit the acquired full-view image to the server 30 (as a source device), and the server 30 can send the full-view raw image to the VR glasses 10 (as a display device) after pre-processing (such as encoding, compression, etc.) of the full-view raw image. After receiving the full-view image, the VR glasses 10 can select a partial-view image including the gaze view of the user from the full-view image based on the viewable range of the VR glasses 10 and the gaze view of the user, and render and display the partial-view image.

[0062] Based on the above method, although the VR glasses 10 displays only the panoramic image of the partial view at the same time, the server 30 will also transmit the full-view image to the VR glasses 10, thus occupying a large amount of network resources. For example, assuming that the panoramic camera 20 samples at an 8K resolution (for example, a resolution of 7680x4320, that is, 7680 pixels are collected in the horizontal direction for each circle (corresponding to a 360° view), and 4320 pixels are collected in the vertical direction for each half circle (corresponding to a 180° view), and the data amount corresponding to 7680x4320 pixels is the data amount of one frame of full-view image. If one pixel includes 3 color channels, each color channel is represented by 8 bits, then the size of one frame of full-view image is 7680x4320x3x8 = 796262400 bits = 94.9 megabytes (MB), which occupies a large amount of network resources.

[0063] Therefore, embodiments of the present application provide a display method. The display device can send user view information (such as the center of the gaze view or the range of the gaze view, etc.) corresponding to the gaze view of the user (hereinafter referred to as the gaze view) to the source device. The source device can generate a partial-view image (such as a panoramic image with a horizontal view smaller than 360° and a view range including the gaze view) based on the gaze view indicated by the user view information and based on the full-view image, and send the partial-view image to the display device. After receiving the partial-view image, the display device can select an image of at least part of the view from the partial-view image to display according to the gaze view of the user (such as selecting a view including the gaze view of the user and having the same size of the viewable range of the display device to display). In this way, the display device can display the content within the gaze view range of the user, and the amount of data transmitted between the display device and the source device can be reduced, which is beneficial to reduce the network resources required for panoramic image transmission.

[0064] In some embodiments, the partial field of view image can be an image centered at the gaze field of view. For example, assuming the display device displays a field of view with a size of 130° horizontally and 120° vertically, and the center of the field of view of the gaze field of view is then the field of view range corresponding to the partial field of view image can be (-65°, 65°) horizontally and (-50°, 70°) vertically, denoted as

[0065] In some embodiments, the size of the field of view corresponding to the partial field of view image can be larger than the size of the field of view that the display device can display. In this way, the change of the gaze field of view of the user can be avoided from causing the display device to fail to timely display the panoramic image corresponding to the changed gaze field of view. For example, assuming the display device displays a field of view with a size of 130° horizontally and 120° vertically, and the center of the field of view of the gaze field of view is then the field of view range corresponding to the partial field of view image can be (-65°, 65°) horizontally and (-50°, 70°) vertically, denoted as (-50°-Δθ1, 70°+Δθ2), where Δθ1 and Δθ2 are angles greater than or equal to 0° and not equal to 0° at the same time.

[0066] Exemplarily, in some embodiments, the size of the field of view corresponding to the partial field of view image can be a fixed size, for example, 180° horizontally (other values are also possible) and 180° vertically (other values are also possible), which is equivalent to a partial field of view image corresponding to a hemisphere. Referring to FIG. 3A, the field of view A corresponding to the user at T1 and the field of view B corresponding to the user at T2 correspond to hemispheres. Corresponding to different time instants, the display device can send the center of the field of view of the gaze field of view to the source device, and the source device takes the center position of the gaze field of view as the center of the partial field of view image to obtain the partial field of view image corresponding to different hemispheres. In this way, assuming that the panoramic camera is sampled at a resolution of 8K (for example, a resolution of 7680x4320, that is, 7680 pixels are collected per week in the horizontal direction (360° field of view), and 4320 pixels are collected per half week in the vertical direction (180° field of view)), the data amount corresponding to each frame of the partial field of view image is 7680x180° / 360°x4320=16588800 pixels. If one pixel includes 3 color channels, each color channel is represented by 8 bits, then the size of one frame of the partial field of view image is 16588800x3x8=47.5 megabytes (MB), which is half of the transmission rate compared with the transmission of the full field of view image. ​

[0067] For example, referring to FIG. 3B, when the VR glasses 10 determine that the center of the field of view of the user's gaze field of view is , the center of the field of view may be sent to the server 30. After the server 30 receives the center of the field of view , the server 30 can obtain a partial field of view image with a horizontal field of view range of -110°-70° and a vertical field of view range of -90°-90° (denoted as ) from the full field of view image sent by the panoramic camera 20, and send the partial field of view image with a field of view range of to the VR glasses 10. The VR glasses 10 can select a part of the field of view from the partial field of view image with a field of view range of to display. For example, assuming that the VR glasses 10 display a field of view with a horizontal field of view of 130° and a vertical field of view of 120°, the VR glasses 10 can select an image with a field of view range of to display from the partial field of view image.

[0068] For example, referring to FIG. 3C, when the VR glasses 10 determine that the center of the field of view of the user's gaze field of view is , the center of the field of view may be sent to the server 30. After the server 30 receives the center of the field of view , the server 30 can obtain a partial field of view image with a horizontal field of view range of -180°-0° and a vertical field of view range of -90°-90° (denoted as ) from the full field of view image sent by the panoramic camera 20, and send the partial field of view image with a field of view range of to the VR glasses 10. The VR glasses 10 can select a part of the field of view from the partial field of view image with a field of view range of to display. For example, assuming that the VR glasses 10 display a field of view with a horizontal field of view of 130° and a vertical field of view of 120°, the VR glasses 10 can select an image with a field of view range of to display from the partial field of view image.

[0069] Based on the above method, the network resources occupied by the display device and the source device for transmitting the panoramic image can be effectively reduced.

[0070] In some embodiments, the field of view size of the partial field of view image in the horizontal direction can be any value less than 360°, for example, any value greater than 124° and less than 360°. In some embodiments, referring to FIG. 4A, the field of view size of a human eye in the horizontal direction is about 156° (different people can be other different values), the field of view size of a human eye in the horizontal direction is about 180°+4°x2=188°, and the overlapping field of view of a human eye in the horizontal direction is about 156°x2-188°=124°. That is, the best field of view size of a human eye in the horizontal direction is about 124°, and setting the field of view size of the partial field of view image in the horizontal direction to a value greater than 124° can ensure that the image seen by the user is clear and the comfort of the user's eyes.

[0071] In some embodiments, the field of view size of the partial field of view image in the vertical direction can be any value less than or equal to 180°, for example, any value greater than 60° and less than or equal to 180°. In some embodiments, referring to FIG. 4B, the field of view size of a human eye in the vertical direction is about 120° (for example, a field of view range of -70° to 50° with the horizontal plane as 0°, different people can be other different values), and the field of view size that a human eye can clearly distinguish in the vertical direction is about 60° (for example, a field of view range of -30° to 30° with the horizontal plane as 0°). Setting the field of view size of the partial field of view image in the vertical direction to a value greater than 60° can ensure that the image seen by the user is clear and the comfort of the user's eyes.

[0072] In some embodiments, the size of the gaze field of view of the user can be any size less than the field of view size of the partial field of view image, and the size of the gaze field of view of the user can be an empirical value, an experimental value, etc. For example, the size of the gaze field of view of the user can be 60° in the horizontal direction and 60° in the vertical direction. For example, assuming that the coordinates of the gaze point of the user are then the gaze field of view of the user can be

[0073] In some embodiments, when generating the partial field of view image, the source device can use original image data (for example, a full field of view image collected by a panoramic camera or a full field of view image stored by the source device) with different sampling densities (for example, angular resolution) to generate the partial field of view image corresponding to different field of view ranges, wherein the higher the sampling density of the original image data used, the closer to the center of the partial field of view image (equivalent to the closer to the center of the gaze field of view of the user). The higher the sampling density of the original image data used to generate the image, the more texture information that can be retained and the higher the clarity of the generated image. In this way, the clarity of the central region (for example, the region corresponding to the gaze field of view of the user) of the generated partial field of view image can be higher.

[0074] For example, when generating the partial field-of-view image, the source device can use the original image data with a higher sampling density (e.g., angular resolution) to generate the image content corresponding to the central field-of-view region (e.g., the central region of the user's gaze field-of-view region), and use the original image data with a lower sampling density to generate the image content corresponding to the N (N is an integer greater than 1) annular field-of-view regions surrounding the central field-of-view region. Optionally, the farther the N annular field-of-view regions are from the central field-of-view region, the lower the sampling density of the original image data used.

[0075] It should be noted that the annular field-of-view region can be a rectangular ring, a circular ring, an elliptical ring, a polygonal ring, an irregularly shaped ring, etc.

[0076] It should be noted that the N annular field-of-view regions can have overlapping field-of-view regions, or can have no overlapping field-of-view regions. The N annular field-of-view regions and the central field-of-view region can constitute the entire field-of-view region of the partial field-of-view image.

[0077] For example, referring to FIG. 5A, assuming that the center of the user's gaze field-of-view is (90°, 0°), compared to using the original image with the same sampling density, when generating the partial field-of-view image, the source device can use the original image data with a certain sampling density (set as a) to generate the image content of the annular field-of-view region outside the user's gaze field-of-view center ± 30° x ± 30°, use the original image data with a sampling density of 2a to generate the image content of the annular field-of-view region inside the user's gaze field-of-view center ± 30° x ± 30° and outside ± 15° x ± 15°, and use the original image data with a sampling density of 4a to generate the image content of the field-of-view (as the central field-of-view region) inside the user's gaze field-of-view center ± 15° x ± 15°. In this way, the image content of the central region of the user's gaze field-of-view can be ensured to be clearer and have a higher texture density.

[0078] It should be noted that the above ± 30° x ± 30° and ± 15° x ± 15° are only an example, and in other embodiments, other larger or smaller ranges can also be used, other different sampling densities can also be used, more or fewer sampling density levels can also be divided, and the embodiments of the present application are not limited thereto.

[0079] It can be understood that a better three-dimensional effect can be achieved when the angular resolution of the panoramic image is greater than 25 pixels per degree (PPD). Currently, the panoramic image transmitted by the source device to the display device usually has an 8K resolution, i.e., there are 7680 pixels on each parallel (e.g., each latitude line), and the corresponding angular resolution is 7680 / 360°=21.3 PPD, which is lower than 25 PPD, and thus the panoramic image cannot exhibit a better three-dimensional effect. If the angular resolution of the panoramic image is to be greater than 25 PPD, the horizontal resolution of the panoramic image based on the ERP format needs to be greater than 360°×25 PPD=9000 pixels, which will further increase the bandwidth required for the source device to transmit the panoramic image.

[0080] Based on this, in some embodiments, the local field of view image sent by the source device to the target device can include multiple field of view images, which can include a central local field of view image corresponding to the center region of the gaze field of view of the user, and K local field of view images corresponding to K annular field of view regions of the center region of the gaze field of view of the user (K is an integer greater than 1). Moreover, the angular resolution of the K local field of view images is less than the angular resolution of the central local field of view image.

[0081] Optionally, the angular resolution of the K local field of view images increases with the increase of the distance between the corresponding annular field of view region and the center of the gaze field of view of the user. In this way, the sampling density of the image within the gaze field of view of the user can be ensured to be higher without increasing or even reducing the total data amount of the local field of view images, and the content after rendering and display is clearer and has a higher texture density.

[0082] Exemplarily, referring to FIG. 5B, it is assumed that the panoramic camera 20 is sampled at a 16K resolution (e.g., a resolution of 15360×8640, i.e., 15360 pixels are collected on each parallel (horizontal direction) in the horizontal direction (i.e., a 360° field of view), and 8640 pixels are collected on each half parallel (vertical direction) in the vertical direction (180° field of view), and the full field of view image P10 (resolution: 15360×8640) is obtained.

[0083] The server 30 can select a partial field of view image Pll (15360 / 2=7680) x 8640) with a field of view size of 180° x 180° from the full field of view image P10, and downsample the partial field of view image Pll to obtain a plurality of partial field of view images. For example, the server 30 can downsample the partial field of view image Pll by 1 / 4 to obtain a partial field of view image P12 (1920 x 2160) with a maximum field of view size of 180° x 180° and an angular resolution of 42.7 PPD / 4=10.7 PDD horizontally and 48 PPD / 4=12 PDD vertically, corresponding to a ring-shaped field of view region of (-90°~ -30°, +30°~ +90°) x (-90°~ -30°, +30°~ +90°) centered on the center of the user's gaze field of view); downsample a region (a 60° x 60° region) corresponding to the gaze field of view in the partial field of view image Pll by 1 / 2 to obtain a partial field of view image P13 (1280 x 1440) with a maximum field of view size of 60° x 60° and an angular resolution of 42.7 PPD / 2=21.3 PDD horizontally and 48 PPD / 4=24 PDD vertically, corresponding to a ring-shaped field of view region of (-30°~ -15°, +15°~ +30°) x (-30°~ -15°, +15°~ +30°) centered on the center of the user's gaze field of view); and downsample a central region (e.g., a 30° x 30° region) centered on the center of the gaze field of view in the partial field of view image Pll without downsample to obtain a partial field of view image P14 (1280 x 1440) with a corresponding field of view size of 30° x 30° and an angular resolution of 42.7 PDD horizontally and 48 PDD vertically. Then, the server 30 can transmit the partial field of view image P12, the partial field of view image P13, and the partial field of view image P14 to the VR glasses 10.

[0084] Continuing to refer to FIG. 5B, after receiving the partial field of view image P12, the partial field of view image P13 and the partial field of view image P14, the VR glasses 10 can render and display the images according to the size of the field of view that the VR glasses 10 can display and the field of view that the user gazes at. In the rendering process, referring to FIG. 5C, the VR glasses 10 can render the display content of a field of view of (-15°~+15°)×(-15°~+15°) (a field of view of -15°~+15° in the horizontal direction and -15°~+15° in the vertical direction) centered on the center of the field of view that the user gazes at based on the partial field of view image P14, render the display content of a ring-shaped field of view area of (-30°~-15°, +15°~+30°)×(-30°~-15°, +15°~+30°) (a field of view of -30°~-15° in the horizontal direction, +15°~+30° in the vertical direction, and -30°~-15° in the vertical direction) centered on the center of the field of view that the user gazes at based on the partial field of view image P13, and render the display content of a ring-shaped field of view area of (-90°~-30°, +30°~+90°)×(-90°~-30°, +30°~+90°) (a field of view of -90°~-30° in the horizontal direction, +30°~+90° in the vertical direction, and -90°~-30° in the vertical direction) centered on the center of the field of view that the user gazes at based on the partial field of view image P12. Since the angular resolution of the partial field of view image P13 and the partial field of view image P14 is higher than the angular resolution of the partial field of view image P12, the display screen of the VR glasses 10 can be made to have a richer texture and higher definition in the display screen of the center of the field of view than in the display screen of the edge area.

[0085] In addition, if one pixel includes 3 color channels, each of which is represented by 8 bits, the sizes of the partial field of view image P12, the partial field of view image P13 and the partial field of view image P14 are 1920×2160×3×8 = 11.9 MB, 1280×1440×3×8 = 4.4 MB, 1280×1440×3×8 = 4.4 MB, respectively, for a total of 11.9 MB + 4.4 MB + 4.4 MB = 20.7 MB, which is 78% less than the 94.9 MB of the aforementioned transmission of the full field of view image.

[0086] It should be noted that the range of the field of view displayed by the VR glasses 10 can be part or all of the range of the field of view of the received partial field of view image. For example, after receiving the partial field of view image P12, the partial field of view image P13 and the partial field of view image P14, the VR glasses 10 can display a field of view of 90°×90° or a field of view of 150°×150° as shown in FIG. 5C.

[0087] It should be noted that the center of the picture displayed by the VR glasses 10 can be the same as or different from the center of the received partial field of view image, and the center of the field of view corresponding to the picture displayed by the VR glasses 10 can be determined according to the gaze field of view of the current user of the VR glasses 10.

[0088] It should be noted that the foregoing taking the field of view with different sampling densities as a rectangle is only an example, and in other embodiments, the division of the field of view can also be other shapes, such as a circle, an ellipse, a sector, a polygon, an irregular shape, etc., which are not limited herein.

[0089] The technical solutions of the present application will be described below in conjunction with FIG. 6.

[0090] FIG. 6 shows an interaction flow diagram of a display method according to some embodiments of the present application. As shown in FIG. 6, the interaction flow includes the following steps:

[0091] S601, the display device acquires the gaze field of view of the user.

[0092] Exemplarily, the display device can acquire the gaze field of view of the user. For example, the display device can determine the gaze field of view of the user according to the data collected by its own sensors (such as inertial sensors, gyroscopes, levels, cameras, etc.).

[0093] In some embodiments, the display device can determine the gaze field of view of the user based on the pose of the head of the user (such as the angle with the horizontal plane, the angle with the vertical plane, the rotation angle, etc.), and / or the eye movement information of the user (such as the line of sight direction of the eyeball of the user, etc.).

[0094] In some embodiments, the display device can also acquire the gaze field of view of the user through other devices.

[0095] In some embodiments, the display device can acquire the gaze field of view of the user periodically or aperiodically.

[0096] S602, the display device sends the user field of view information corresponding to the gaze field of view of the user to the source device.

[0097] After acquiring the gaze field of view of the user, the display device can send the user field of view information corresponding to the gaze field of view of the user to the source device.

[0098] In some embodiments, the user visual field information can include one or more of a visual field center of the user's gaze visual field, a range of the user's gaze visual field (e.g., if the gaze visual field is a polygon, the coordinates of the polygon vertices; if the gaze visual field is a circle, the radius; if the gaze visual field is an ellipse, the lengths of the short and long semi-axes of the ellipse), etc. For example, for the scenario shown in FIG. 3B, the VR glasses 10 (as the display device) can send the user's gaze visual field center (-20°, 0°) to the server 30 (as the source device); for the scenario shown in FIG. 3C, the VR glasses 10 can send the user's gaze visual field center (-90°, 0°) to the server 30.

[0099] In some other embodiments, the user visual field information can also include other information that can indicate the user's gaze visual field, which is not limited herein.

[0100] In some embodiments, the display device can also send display device information to the source device, which can include the size of the visual field that the display device can display, and / or the size of the local visual field image required by the display device, the sampling density of different visual field regions of the required local visual field image, the resolution, and / or the format of the required local visual field image of the display device, etc.

[0101] In some embodiments, the display device can periodically or non-periodically send the user visual field information corresponding to the user's gaze visual field to the source device.

[0102] In some embodiments, the display device can send the user visual field information corresponding to the user's current gaze visual field to the source device when the user's current gaze visual field is different from or significantly different from the last sent user visual field information, and not send the user visual field information corresponding to the user's current gaze visual field to the source device when the user's current gaze visual field is the same as or not significantly different from the last sent user visual field information. In this way, the frequency of the user visual field information sent by the display device to the source device can be reduced.

[0103] In some embodiments, the user visual field information can be sent by the display device directly to the source device, or sent by the display device to the source device via other devices, which is not limited herein.

[0104] In some embodiments, the user visual field information sent by the display device to the source device can also be the pose of the user's head (e.g., the angle with the horizontal plane, the angle with the vertical plane, the rotation angle, etc.), and / or the eye movement information of the user (e.g., the line-of-sight direction of the user's eyeballs, etc.), so that the source device can determine the user's gaze visual field based on the pose information of the user's head.

[0105] S603, the source device generates the partial view image based on the user view information and the full view image.

[0106] After receiving the user view information, the source device can generate the partial view image based on the user view information and the full view image.

[0107] For example, after receiving the user view information, the source device can determine the partial view image from the full view image, where the partial view image includes the gaze view of the user and is smaller than the full view image. For example, for the case shown in FIG. 3B, the server 30 (as the source device) can generate a partial view image with a view range of ; for the case shown in FIG. 3C, the server 30 can generate a partial view image with a view range of .

[0108] In some embodiments, the center of the partial view image can be the same as (or different from, which is not limited herein) the center of the gaze view indicated by the user view information.

[0109] In some embodiments, the view size of the partial view image can be a preset size, such as 180°x180°, 180°x150°, 160°x150° (or other sizes, which are not limited herein), or a size indicated by the display device information sent by the display device.

[0110] In some embodiments, the view size of the partial view image in the horizontal direction is greater than 124° and less than 360°.

[0111] In some embodiments, the view size of the partial view image in the vertical direction is greater than 120° and less than or equal to 180°.

[0112] In some embodiments, the sampling density of the original image corresponding to different regions in the partial view image generated by the source device can be different. For example, when generating the partial view image, the source device can use original image data with a higher sampling density (e.g., angular resolution) to generate image content corresponding to the center view region of the partial view image (e.g., the center region of the gaze view region of the user), and use original image data with a lower sampling density to generate image content corresponding to N (N is an integer greater than 1) annular view regions surrounding the center view region. Optionally, the greater the distance of the N annular view regions from the center view region, the smaller the sampling density of the original image data used.

[0113] For example, referring to FIG. 5A, compared with the original image with the same sampling density, when generating the partial view image, the source device can generate the image content of the annular view area outside the center of the user's gaze view field ± 30° x ± 30° based on the original image data of a certain sampling density (set as a), generate the image content of the annular view area inside the center of the user's gaze view field ± 30° x ± 30° and outside ± 15° x ± 15° based on the original image data of 2a sampling density, and generate the image content of the view field (as the central view area) inside the center of the user's gaze view field ± 15° x ± 15° based on the original image data of 4a sampling density. In this way, the content of the image of the central area of the user's gaze view field can be ensured to be clearer and have higher texture density.

[0114] In some embodiments, the source device can generate multiple view images, the multiple (for example, N, N is an integer greater than 1) partial view images correspond to different distances from the center of the user's gaze view field, and the distance from the center of the user's gaze view field to the center of the view field of the partial view image is greater, and the angular resolution of the partial view image is lower. In this way, without increasing or even reducing the total amount of data of the partial view image, the sampling density of the image within the user's gaze view field can be ensured to be higher, and the content after rendering and display can be clearer and have higher texture density.

[0115] For example, for the case shown in FIG. 5B, the size of the partial view image is 180° x 180°, and the resolution of the full view image P10 is 15360 x 8640. The source device (server 30) can select a partial view image P11 with a view field size of 180° x 180° from the full view image P10 ((15360 / 2 = 7680) x 8640), and downsample the partial view image P11 to obtain multiple partial view images. For example, the server 30 can downsample the partial view image P11 by 1 / 4 to obtain a partial view image P12 with a resolution of 1920 x 2160; downsample the region corresponding to the gaze view field (the central 60° x 60° region) of the partial view image P11 by 1 / 2 to obtain a partial view image P13 with a resolution of 1280 x 1440; and do not downsample the view field region (for example, a 30° x 30° region) of the center of the gaze view field of the partial view image P11 to obtain a partial view image P14 with a resolution of 1280 x 1440.

[0116] In some embodiments, the source device can also only collect the content of the field of view area corresponding to the partial field of view image when downsampling the full field of view image, so as to further reduce the size of the partial field of view image. For example, when generating the aforementioned partial field of view image P12, the area in the center filled by other partial field of view images (for example, the area filled by the partial field of view image P13 and the partial field of view image P14, that is, the area of the center 60°x60° area) can not be retained.

[0117] It should be noted that, corresponding to the image transmitted by the source device to the display device being a video (or said to include multiple continuous images), the source device can perform similar processing on each frame image in the video to obtain a partial field of view image.

[0118] It should be noted that the full field of view image can be an image stored or collected by the source device itself, or an image collected by other devices and transmitted to the source device, which is not limited here.

[0119] In some embodiments, the format of the partial field of view image can be the aforementioned ERP format or the format required by the display device.

[0120] It should be noted that, in some embodiments, compared with the center of the traditional ERP format image being fixed at a point, the center of the partial field of view image can change with the change of the user's gaze field of view, for example, the center of the partial field of view image can be the center of the user's gaze field of view. For example, referring to FIG. 7A, corresponding to the aforementioned situation shown in FIG. 3B, the center of the user's gaze field of view is Then the center of the obtained expanded image is The field of view range of the expanded image is For another example, referring to FIG. 7B, assuming that the center of the user's gaze field of view is The field of view size of the partial field of view image is 180°x150°, and the center of the obtained expanded image is The field of view range of the expanded image is

[0121] For example, referring to FIG. 7C, the source device can generate a partial field of view image based on the original image data corresponding to the full field of view image V0 (for example, the original data collected by the panoramic camera). Assuming that the center of the user's gaze field of view is on the TV 01, the source device can sample the original image data corresponding to the full field of view image V0 with the gaze center P1 on the TV 01 as the center to obtain a partial field of view image V1; assuming that the center of the user's gaze field of view is on the window 02, the source device can sample the original image data corresponding to the full field of view image V0 with the gaze center P2 on the window 02 as the center to obtain a partial field of view image V2.

[0122] In some embodiments, corresponding to the scenario shown in FIG. 7C, the closer to P1 or P2 the field of view area is, the greater the sampling density of the original image data corresponding to the full field of view image V0 is in generating the partial field of view image V1 or the partial field of view image V2.

[0123] S604, the source device sends the partial field of view image to the display device.

[0124] After determining the partial field of view image (one or more), the source device can send the partial field of view image to the display device. For example, for the scenario shown in FIG. 3B, the server 30 (as the source device) can send the partial field of view image with a field of view range of to the VR glasses 10 (as the display device); for the scenario shown in FIG. 3C, the server 30 can send the partial field of view image with a field of view range of to the VR glasses 10; for the scenario shown in FIG. 5B, the server 30 can send the partial field of view image P12, the partial field of view image P13 and the partial field of view image P14 to the VR glasses 10

[0125] In some embodiments, corresponding to the image transmitted by the source device to the display device being a video (or in other words, including multiple continuous images), different regions of the partial field of view image can be transmitted with different code rates.

[0126] In some embodiments, the source device can also send the origin point coordinates of the expansion of the partial field of view image (for example, the coordinates of the center of the partial field of view image, which can be the same as or associated with but different from the field of view center of the gaze field of view of the user, so that the display device can expand the partial field of view image based on the origin point coordinates.

[0127] S605, the display device renders and displays the image based on the partial field of view image.

[0128] After receiving the partial field of view image, the display device can decode, render and display the image.

[0129] Exemplarily, in the case where the partial field of view image corresponding to one frame of image is a single image, the display device can directly select the image with the displayable field of view size from the partial field of view image with the center of the field of view center of the gaze field of view of the user as the center for display. For example, assuming that the size of the partial field of view image is 180°x180°, the center of the gaze field of view of the user is the same as the center of the partial field of view image, and the displayable field of view size of the display device is 180°x150°, the display device can select the content of 180°x150° in the center of the partial field of view image for display.

[0130] For example, referring to FIG. 5B and FIG. 5C, assuming that the size of the local field of view image is 180°x180°, the center of the gaze field of view of the user is the same as the center of the local field of view image, and the size of the field of view that can be displayed by the display device is 180°x150° and the resolution is 7680x7200, for the local field of view image P12, the local field of view image P13, and the local field of view image P14 received, the display device can render the display content in a 30°x30° field of view at the center based on the local field of view image P14 (corresponding to an image with a center area of (7680x30° / 180°=1280)x(7200x30° / 150°=1440) pixels, and an angular resolution of 42.7 PDD in the horizontal direction and 48 PDD in the vertical direction), render the display content in a 60°x60° field of view at the center and outside the 30°x30° field of view at the center based on the local field of view image P13 (corresponding to an image with a center area of (7680x60° / 180°=2560)x(7200x60° / 150°=2880) and outside the center area of 1280x1440 pixels, and an angular resolution of 21.3 PDD in the horizontal direction and 24 PDD in the vertical direction), and render the display content outside the 60°x60° field of view at the center based on the local field of view image P12 (corresponding to an image outside the center area of (7680x60° / 180°=2560)x(7200x60° / 150°=2880), and an angular resolution of 10.7 PDD in the horizontal direction and 12 PDD in the vertical direction), and finally obtain an image with a resolution of 7680x7200. In this way, in the image displayed by the display device, the closer to the center of the gaze field of view of the user, the higher the sampling density (for example, the angular resolution), the clearer the content, and the higher the texture density.

[0131] Based on the above method, the source device does not need to transmit a full field of view image to the display device, but transmits a local field of view image corresponding to the gaze field of view of the user to the display device, thereby saving the bandwidth required for panoramic image transmission while ensuring display quality. In addition, in some scenarios, the local field of view image transmitted by the source device to the display device can be one or more, and the sampling density of the local field of view images or image regions corresponding to different fields of view can be different (for example, the closer to the center of the field of view of the gaze field of view of the user, the higher the sampling density), which not only improves the clarity and texture density of the center of the field of view in the image displayed by the display device, but also does not increase the bandwidth required for transmission.

[0132] In some embodiments, the present application also provides a display method. As shown in FIG. 8, the display method comprises the following steps:

[0133] S801, the display device acquires a real-time gaze area.

[0134] The display device can acquire the gaze area of the user in real time (equivalent to the aforementioned gaze field of view of the user), and send information corresponding to the gaze area (such as the aforementioned user field of view information) to the source device. For details, please refer to the aforementioned step S601, which will not be repeated here.

[0135] In some implementations, the gaze area can be determined according to the user's head pose or eye movement.

[0136] S802, the source device generates an ERP partial field of view image centered on the gaze area.

[0137] After receiving the information corresponding to the gaze area, the source device can generate an ERP format partial field of view image centered on the gaze area. For details of the process of generating the partial field of view image by the source device, please refer to the aforementioned step S603, which will not be repeated here.

[0138] In some embodiments, the code rate (equivalent to the aforementioned sampling density) of the central region of the gaze area can be set separately, for example, set to a higher code rate (or sampling density) than other regions.

[0139] S803, network transmission.

[0140] After generating the ERP format partial field of view image, the source device can send the generated partial field of view image to the display device.

[0141] S804, local decoding of the display device.

[0142] After receiving the ERP format partial field of view image, the display device can decode to obtain the data of the ERP texture and the coordinates of the central region.

[0143] S805, rendering of the display device.

[0144] After decoding the partial field of view image, the display device can select at least part of the field of view from the partial field of view image for display. For details of the process of rendering and displaying the image by the display device, please refer to the aforementioned step S605, which will not be repeated here.

[0145] Figure 9 shows a structural schematic diagram of a VR glasses 10 according to some embodiments of the present application.

[0146] As shown in Figure 9, the VR glasses 10 can include a processor 110, a memory 120, a sensor module 130, an audio module 140, a key 150, an input / output interface 160, a communication module 170, a power supply module 180, a display module 190, etc.

[0147] The processor 110 is generally configured to control overall operations of the VR glasses 10, and can include one or more processing units, for example: the processor 110 can include an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), a video processing unit (VPU) controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0148] In some embodiments, the processor 110 can be configured to determine a gaze field of the user based on data collected by the sensor module 130, and send user field of view information indicating the gaze field of the user to the source device, and after receiving the partial field of view image, select at least part of the content displayed from the partial field of view image based on the size of the field of view that can be displayed by the display module 12.

[0149] The memory 120 can be configured to store computer-executable program codes including instructions. The processor 110 executes various functional applications and data processing of the VR glasses 10 by running the instructions stored in the memory 120. The memory 120 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required by at least one function (such as an image display function, a video playing function), etc. The data storage area can store data created during the use of the VR glasses 10, such as user field of view information, partial field of view images, etc. In addition, the memory 120 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0150] The sensor module 130 can include sensors for detecting the working state and the use state of the VR glasses 10. Including but not limited to a proximity light sensor / contact sensor for detecting whether the user wears the VR glasses 10, an inertial sensor for detecting the motion state of the user's head, a camera for shooting the image of the user's eye, etc.

[0151] The audio module 140 can include a speaker, a microphone, etc., for realizing audio functions.

[0152] The key 150 can be one or more. The key 150 can be in the form of a button, a switch, a dial, and a touch or near touch sensing device (e.g., a touch sensor). The VR glasses 10 can trigger corresponding functions according to the user's operation on one or more of the keys 150.

[0153] The input / output interface 160 can connect other devices to the VR glasses 10 through suitable components. The components can include, for example, an audio / video jack, a data connector, and the like.

[0154] The communication module 170 can include a wireless communication module. The wireless communication module can provide a wireless communication solution including wireless local area networks (WLAN) (e.g., a wireless fidelity (Wi-Fi) network), Bluetooth (BT), a global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, and the like, which are applied to the VR glasses 10. The wireless communication module can be one or more devices that integrate at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module can also receive signals to be sent from the processor 110, frequency-modulate them, amplify them, and radiate them as electromagnetic waves via an antenna.

[0155] In some embodiments, the communication module 170 can be used to send user field of view information, device information, and receive partial field of view images from other devices, and the like.

[0156] The power supply module 180 can include a battery for supplying power to various modules of the VR glasses 10.

[0157] The display module 190 can include one or more display screens, optical lens assemblies, and the like, for displaying content, such as displaying panoramic images.

[0158] It can be understood that the structure of the VR glasses 10 shown in the embodiments of the present application does not constitute a specific limitation on the VR glasses 10. In other embodiments of the present application, the VR glasses 10 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0159] It can be understood that other forms of display devices can adopt the same or similar structure as the VR glasses 10, or different structure from the VR glasses 10, which is not limited herein.

[0160] FIG. 10 shows a structural schematic diagram of a server 30 according to some embodiments of the present application.

[0161] Referring to FIG. 10, the server 30 can include one or more processors 301, one or more memories 302, and the processor 301 and the memory 302 can be coupled through a bus.

[0162] The memory 302 can be used to store instructions corresponding to the display method provided by the embodiments of the present application, and the processor 301 can be used to execute the instructions stored in the memory 302 to realize the display method provided by the foregoing embodiments. For example, the processor 301 can execute instructions to receive user field of view information from a display device, determine a partial field of view image from a full field of view image according to the user field of view information, and transmit the partial field of view image to the display device. For details, reference can be made to the contents of the foregoing steps S602-S604, which are not described herein.

[0163] In some embodiments, the memory 302 can also be used to store the full field of view image (picture or video).

[0164] In some embodiments, the server 30 can further include a communication interface 303 for communication with other devices or modules or networks. For example, the server 30 can receive the full field of view image from an image acquisition device (such as the foregoing panoramic camera 20) based on the communication interface 303, receive the user field of view information from the display device based on the communication interface 303, and transmit the partial field of view image to the display device based on the communication interface 303.

[0165] It can be understood that the structure of the server 30 shown in the embodiments of the present application does not constitute a specific limitation on the server 30. In other embodiments of the present application, the server 30 can include more or fewer components than shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0166] It can be understood that other forms of source devices can adopt the same or similar structure as the server 30, or different structure from the server 30, which is not limited herein.

[0167] The embodiments of the present application also provide a computer program product, which includes computer program code, when the computer program code is run on a computer, so that the computer implements the method in the above embodiments of the present application.

[0168] The embodiment of the present application further provides a computer readable storage medium, which stores computer instructions, and when the computer instructions run on a computer, the computer implements the method in the above-mentioned embodiments of the present application.

[0169] In the drawings, some structural or methodological features can be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order can not be required. Instead, these features can be arranged in a different manner and / or order than shown in the illustrative drawings, in some embodiments. Additionally, the inclusion of a structural or methodological feature in a particular figure is not meant to imply that such feature is required in all embodiments, and in some embodiments, such feature can not be included or can be combined with other features.

[0170] It should be noted that the relational terms herein such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0171] In the embodiments provided by the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiment is merely exemplary. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0172] In various embodiments of the present application, the terms and / or descriptions between the various embodiments are consistent if there is no special description and logical conflict, and can be referred to each other, and the technical features in different embodiments can be combined to form a new embodiment according to their inherent logical relationship.

[0173] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment of the present application according to actual needs.

[0174] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0175] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A display method, applied to a first electronic device, characterized in that: The method comprises: sending first field of view information to a second electronic device, wherein the first field of view information indicates a first field of view of a gaze of a user of the first electronic device; receiving a partial field of view image sent by the second electronic device, wherein a field of view range corresponding to the partial field of view image comprises the first field of view, and the partial field of view image is a partial image of a corresponding first panoramic image; displaying at least part of an image region in the partial field of view image, wherein the at least part of the image region comprises an image region corresponding to the first field of view.

2. The method of claim 1, wherein, The first field of view information comprises at least one of the following information: a range of the first field of view; a field of view center of the first field of view; a head pose of the user; a line of sight direction of the user.

3. The method of claim 1, wherein a field of view range of the partial field of view image in a horizontal direction is greater than or equal to 124° and less than 360°.

4. The method of claim 1, wherein, The partial field of view image comprises a first partial field of view image and at least one second partial field of view image, wherein the first partial field of view image corresponds to a first field of view region at a center of the first field of view, the second partial field of view image corresponds to an annular field of view region surrounding the first field of view region, an angular resolution of the first partial field of view image is greater than an angular resolution of the second partial field of view image, and the angular resolution of the second partial field of view image decreases as a distance between an environmental field of view region corresponding to the second partial field of view image and the center of the first field of view increases; and The displaying at least part of the image region in the partial field of view image comprises: rendering display content of the first field of view region based on the first partial field of view image and display content of the annular field of view region based on the second partial field of view image to obtain a third partial field of view image; and displaying the third partial field of view image. 5.A display method applied to a second electronic device, the method comprising: The method comprises: receiving first field of view information sent by a first electronic device, wherein the first field of view information indicates a field of view of a gaze of a user of the first electronic device; sending a partial field of view image to the first electronic device, wherein a field of view range corresponding to the partial field of view image comprises the first field of view, and the partial field of view image is a partial image of a first panoramic image.

6. The method of claim 5, wherein, The first field of view information comprises at least one of the following information: a range of the first field of view; a field of view center of the first field of view; a head pose of the user; a line of sight direction of the user.

7. The method of claim 6, wherein a field of view range of the partial field of view image in a horizontal direction is greater than or equal to 124° and less than 360°.

8. The method of claim 1, wherein, The local field of view image includes a first local field of view image and at least one second local field of view image, wherein the first local field of view image corresponds to a first field of view region at the center of the first field of view, the second local field of view image corresponds to an annular field of view region surrounding the first field of view region, the angular resolution of the first local field of view image is greater than the angular resolution of the second local field of view image, and the angular resolution of the second local field of view image decreases as the distance between the environmental field of view region corresponding to the second local field of view image and the center of the first field of view increases.

9. The method of claim 8, wherein, The angular resolution of the first local field of view image is less than or equal to the angular resolution of the first panoramic image.

10. The method of claim 8, wherein, The local field of view image is obtained by sampling the first panoramic image by the second electronic device.

11. The method of claim 6, wherein, The local field of view image includes a second field of view region at the center of the local field of view image, and at least one annular field of view region surrounding the second field of view region; The content of the second field of view region is generated based on image data with a first sampling density, and the sampling density of the image data used to generate the content of the at least one annular field of view region is less than the first sampling density.

12. The method of claim 11, wherein, The sampling density of the image data used to generate the content of the at least one annular field of view region decreases as the distance between the annular field of view region and the second field of view region increases.

13. A display method characterized by comprising: Comprising: The first electronic device sends first field of view information to the second electronic device, wherein the first field of view information indicates a first field of view at which a user of the first electronic device gazes; The second electronic device sends a local field of view image to the first electronic device in response to the first field of view information, wherein the local field of view image corresponds to a field of view range including the first field of view, and the local field of view image is a partial image of a corresponding first panoramic image; The first electronic device displays at least part of the image region in the local field of view image, wherein the at least part of the image region includes an image region corresponding to the first field of view.

14. An electronic device, comprising: Comprising: A memory for storing one or more programs; A processor for executing the one or more programs to cause the electronic device to implement the display method of any one of claims 1 to 4, or the display method of any one of claims 5 to 12.

15. A readable storage medium, characterized by, The readable storage medium includes one or more programs, and the one or more programs, when executed on an electronic device, cause the electronic device to implement the display method of any one of claims 1 to 4, or the display method of any one of claims 5 to 12.

Images