Image display method and electronic equipment

Abstract

The embodiment of the invention discloses an image display method and electronic equipment. The method comprises the following steps: determining first position information of a center position of two eyes of an object watched by the electronic equipment relative to a screen of the electronic equipment; performing light field coding based on a target view cone corresponding to the first position information to obtain a target light field coding image; wherein the target view cone is any one of a plurality of view cones obtained by dividing a visual area corresponding to the viewing angle of the screen of the electronic equipment, and different view cones correspond to different light field coding modes; and performing three-dimensional display on the light field coding image through the screen of the electronic equipment.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional image display, and in particular to an image display method and electronic equipment. Background Art

[0002] Traditional single-person naked-eye 3D (Three Dimensional) solutions, such as Figure 1 As shown, one image is generated for the left eye and another for the right eye according to the position of the human eyes, which does not conform to the experience of the human eye viewing the real world. The content displayed on the screen changes in real time with the position of the human eyes, which will cause the problem of convergence accommodation conflict (Visual Accommodation Conflict, VAC), thereby causing dizziness and reducing the user's viewing experience. Summary of the invention

[0003] The technical solution of this application is implemented as follows:

[0004] The present application provides an image display method, the method comprising:

[0005] Determine first position information of the center position of both eyes of the electronic device viewing the object relative to the screen of the electronic device;

[0006] Performing light field encoding based on a target visual cone corresponding to the first position information to obtain a target light field encoded image; wherein the target visual cone is any one of a plurality of visual cones obtained by dividing a visible area corresponding to a viewing angle of the electronic device screen, and different visual cones correspond to different light field encoding methods;

[0007] The target light field coded image is displayed in three dimensions through the electronic device screen.

[0008] An embodiment of the present application provides an electronic device, including:

[0009] Multiple image acquisition devices,

[0010] A processing device is connected to the multiple image acquisition devices, the processing device determines first position information of the center position of both eyes of the electronic device viewing the object relative to the screen of the electronic device; performs light field encoding based on the target visual cone corresponding to the first position information to obtain a target light field encoded image acquired by the image acquisition device; wherein the target visual cone is any one of the multiple visual cones obtained by dividing the visible area corresponding to the viewing angle of the screen of the electronic device, and different visual cones correspond to different light field encoding methods;

[0011] A display device is connected to the processing device and performs three-dimensional display of the target light field coded image through the screen of the electronic device.

[0012] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. The computer-executable instructions are configured to execute the steps of the above-mentioned image display method. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A schematic diagram of a single-person naked-eye 3D display solution provided in the related art;

[0014] Figure 2 A schematic diagram of a naked-eye 3D display in a multi-person large-screen manner provided in the related art;

[0015] Figure 3 A schematic diagram of a flow chart of an image display method provided in an embodiment of the present application;

[0016] Figure 4 A schematic diagram of multiple viewing cones obtained by dividing a viewing area of ​​a screen of an electronic device provided in an embodiment of the present application;

[0017] Figure 5 A flowchart of an image display method based on human eye tracking and light field coding fusion provided in an embodiment of the present application;

[0018] Figure 6 A schematic diagram of the structure of a three-dimensional space coordinate system provided in an embodiment of the present application;

[0019] Figure 7 A schematic diagram of a viewing cone and a placement position of a virtual rendering camera corresponding to the viewing cone provided in an embodiment of the present application;

[0020] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0022] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0023] In the following description, reference is made to “some embodiments\other embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments\other embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0024] In the following description, the terms "first\second\" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0025] In the related art, there is a method of using a multi-person large screen to perform naked eye 3D display, such as Figure 2 As shown, the multi-person large-screen approach uses a multi-viewpoint solution to evenly distribute viewpoints in the entire viewing area, but it encounters a contradiction between angular resolution and spatial resolution.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0027] Based on the problems existing in the related art, the embodiment of the present application provides an image display method, such as Figure 3 FIG. 1 is a flow chart of an image display method provided in an embodiment of the present application, and the method comprises the following steps:

[0028] S101, determining first position information of the center positions of both eyes of an electronic device viewing an object relative to a screen of the electronic device.

[0029] In some embodiments, the electronic device may be an electronic device that can display three-dimensional images using naked-eye 3D display technology; the object viewed by the electronic device may be a user viewing the electronic device screen, a designer or developer of the electronic device, etc.; the center position of both eyes of the object viewed by the electronic device may be the midpoint position on the line connecting the centers of the two eyes of the object viewed by the electronic device.

[0030] In some embodiments, the first position information can be represented by position coordinates. For example, a three-dimensional space coordinate system can be established with the center of the electronic device screen as the origin, the horizontal direction of the electronic device screen as the x-axis, the vertical direction of the electronic device screen as the y-axis, and the direction perpendicular to the electronic device screen as the z-axis. The first position information can be the position coordinates of the center position of both eyes of the electronic device viewing the object in the three-dimensional space coordinate system.

[0031] S102 . Perform light field coding based on a target visual cone corresponding to the first position information to obtain a target light field coded image.

[0032] In some embodiments, the target visual cone may be any one of a plurality of visual cones obtained by dividing the visible area corresponding to the viewing angle of the electronic device screen, and the center position of both eyes of the electronic device viewing the object is within the range of the target visual cone.

[0033] In some embodiments, a viewing cone may correspond to one or more viewpoint images, and the viewpoint images may be multi-perspective images acquired by an image acquisition device through different shooting angles. Different viewing cones correspond to different viewpoint images, and light field encoding may be determined based on the viewpoint images. Therefore, different viewing cones correspond to different light field encoding methods, and the encoding method corresponding to the same viewing cone may be the same.

[0034] In some embodiments, in the process of performing light field encoding on the target visual cone corresponding to the first position information, light field encoding can be performed on the viewpoint image corresponding to the target visual cone based on the offset of the center position of both eyes relative to the center position of the electronic device screen, thereby obtaining the corresponding target light field encoded image.

[0035] S103: Display the target light field coded image in three dimensions through a screen of an electronic device.

[0036] In some embodiments, after the target light field coded image is obtained, the target light field coded image can be displayed in three dimensions on a screen of an electronic device through screen display technology.

[0037] In an embodiment of the present application, first position information of the center position of both eyes of an object being viewed by an electronic device relative to the screen of the electronic device is determined; light field encoding is performed based on the target visual cone corresponding to the first position information to obtain a target light field encoded image; the target visual cone is any one of a plurality of visual cones obtained by dividing the visible area corresponding to the viewing angle of the screen of the electronic device, and different visual cones correspond to different light field encoding methods; the target light field encoded image is three-dimensionally displayed through the screen of the electronic device. In this way, since the light field encoding methods corresponding to different visual cones are different, the light field encoding method in the target visual cone corresponding to the center position of both eyes of the object being viewed by the electronic device remains unchanged, and therefore, the screen of the electronic device will not frequently change the image content of the three-dimensional display, thereby avoiding the problem of dizziness of the object being viewed by the electronic device and improving the viewing experience of the object being viewed by the electronic device.

[0038] In some embodiments of the present application, light field encoding is performed based on the target visual cone corresponding to the first position information to obtain a target light field coded image, and the above step S102 can be implemented by the following steps S1021 to S1022, and each step is described separately below.

[0039] S1021. Determine, based on the first position information, offset information of the center positions of both eyes relative to the center of the electronic device screen.

[0040] In some embodiments, when the first position information is the position coordinates of the center positions of both eyes in a three-dimensional space coordinate system established with the electronic device screen as the center, the offset information of the center positions of both eyes relative to the center of the electronic device screen can be determined based on the position coordinates, and the offset information can be an offset.

[0041] S1022. Based on the offset information, perform light field encoding on the viewpoint image corresponding to the target viewing cone to obtain a target light field encoded image.

[0042] In some embodiments, the process of performing light field coding on the viewpoint images corresponding to the target view cone can be understood as the process of selecting the target viewpoint image from the viewpoint images corresponding to the target view cone, and the target viewpoint image can be the target light field coded image. The image content (or image pixels) in the target light field coded image can come from one or more viewpoint images. If the target light field coded image comes from one viewpoint image, the target light field coded image can be one of the viewpoint images corresponding to the target view cone; if the target light field coded image comes from multiple viewpoint images, the target light field coded image can be an image obtained by fusing the image contents at different positions in the multiple viewpoint images.

[0043] In some embodiments, the viewpoint images corresponding to the target view cone can be light field encoded based on the offset information, the number of viewpoint images corresponding to the target view cone, the inclination angle and the number of lines of a single cylindrical lens in the screen of the electronic device, etc., so as to determine the target light field encoded image from the viewpoint images corresponding to the target view cone.

[0044] It can be understood that, based on the offset information plane of the center position of both eyes relative to the center of the electronic device screen, the viewpoint image corresponding to the target visual cone is light field encoded to obtain a target light field encoded image, thereby realizing the determination of the content displayed on the electronic device screen according to the target visual cone corresponding to the center position of both eyes, thereby improving the angular resolution and spatial resolution of the displayed content.

[0045] In some embodiments of the present application, based on the first position information, the offset information of the center position of both eyes relative to the center of the electronic device screen is determined, that is, the above-mentioned step S1021 can be implemented by the following steps S201 to S202, and each step is described separately below.

[0046] S201. Determine a first distance from a center position of both eyes to a first straight line according to first position information.

[0047] In some embodiments, the first straight line is a straight line passing through the center point of the electronic device screen and perpendicular to the electronic device screen. When the first position information is a position coordinate, the first distance from the center position of the eyes to the first straight line can be determined according to the position coordinate.

[0048] Exemplarily, if the position coordinates of the center positions of both eyes in the three-dimensional space coordinate system established with the screen of the electronic device as the center are (a, b, c), where a, b, and c represent the x-coordinate, y-coordinate, and z-coordinate of the center positions of both eyes, respectively, then the first straight line can be a straight line coinciding with the z-axis, and the first distance from the center positions of both eyes to the first straight line can be a.

[0049] S202: Determine the first distance as offset information.

[0050] In some embodiments, the offset information may be understood as the distance in the horizontal direction of the electronic device screen that the center positions of the eyes of the electronic device viewing the object deviate from the center of the electronic device screen.

[0051] It can be understood that the first distance from the center position of the binoculars to the first straight line is determined based on the first position information, thereby determining the offset information of the center position of the binoculars relative to the center of the electronic device screen, so that the viewpoint image corresponding to the target visual cone can be accurately light field encoded according to the offset information.

[0052] In some embodiments of the present application, before performing light field encoding on the viewpoint image corresponding to the target viewing cone based on the offset information to obtain the target light field encoded image, that is, before step S1022, the following steps S301 to S303 may also be performed, and each step is described below.

[0053] S301: Obtain a viewing angle of a screen of an electronic device.

[0054] In some embodiments, the viewing angle of the electronic device can be the angle at which a user can observe the content displayed on the screen of the electronic device from different directions. The viewing angle of the electronic device screen can be determined by the material, structure, backlight technology, etc. of the electronic device screen. Therefore, after the electronic device screen is designed, the viewing angle is determined accordingly.

[0055] In some embodiments, the viewing angle of the electronic device screen may be an optimal viewing angle, within which the user can clearly observe the content displayed on the electronic device screen; the viewing angle of the electronic device screen may be a maximum viewable angle, within which the user can observe all displayed content on the electronic device screen; the viewing angle of the electronic device screen may be a horizontal viewing angle, which may be an angle at which the user can observe the content displayed on the electronic device screen from the left side to the right side of the electronic device screen; the viewing angle of the electronic device screen may be a vertical viewing angle, which may be an angle at which the user can observe the content displayed on the electronic device screen from the top to the bottom of the electronic device screen.

[0056] S302: Divide the visible area corresponding to the viewing angle into a plurality of viewing cones with the same angle.

[0057] In some embodiments, the size of the viewing cone is determined by the grating structure of the electronic device screen, and the viewing angle can be an integer multiple of the angle corresponding to the viewing cone, so the visible area corresponding to the viewing angle can be divided into multiple viewing cones with the same angle.

[0058] For example, if the viewing angle is 60 degrees and the angle corresponding to one viewing cone is 12 degrees, the visible area corresponding to the viewing angle can be divided into 5 viewing cones with angles of 12 degrees; if the viewing angle is 120 degrees and the angle corresponding to one viewing cone is 20 degrees, the visible area corresponding to the viewing angle can be divided into 6 viewing cones with angles of 20 degrees. The viewing angle and the angle of the viewing cone are merely exemplary descriptions and are not limited in this application.

[0059] S303: Determine the viewpoint image corresponding to each view cone according to the number of viewpoint images, the number of view cones, and the acquisition angle of the viewpoint images.

[0060] In some embodiments, the viewpoint images are two-dimensional images obtained by collecting the same object from multiple perspectives, and each viewpoint image can be evenly allocated to each view cone according to the data of the viewpoint image and the number of view cone images, so that each view cone has the same number of viewpoint images. In addition, in the process of allocating viewpoint images to each view cone, the collection angle or shooting angle of each viewpoint image also needs to be considered.

[0061] In some embodiments, the acquisition angles of viewpoint images corresponding to two adjacent view cones and the acquisition angles of multiple viewpoint images corresponding to the same view cone all change continuously. That is, the acquisition angles of each viewpoint image corresponding to the viewing angle all change continuously, and the acquisition angles of adjacent viewpoint images corresponding to two adjacent view cones change continuously.

[0062] For example, Figure 4 As shown, if the viewing angle of the electronic device screen is 60 degrees, it is divided into 5 12-degree cones, and 120 viewpoint images are collected every 3 degrees within the range of 360 degrees. The viewpoint images are sorted according to the collection angle, then the 1st to 24th images correspond to the first cone, the 25th to 48th images correspond to the second cone, the 49th to 72nd images correspond to the third cone, the 73rd to 96th images correspond to the fourth cone, and the 97th to 120th images correspond to the fifth cone.

[0063] It can be understood that the viewing angle of the electronic device screen is divided into multiple cones with the same angle, and then the viewpoint image corresponding to each cone is determined in combination with the number of cones, the number of viewpoint images and the acquisition angle of each viewpoint image, so as to facilitate the subsequent light field encoding according to the viewpoint image corresponding to the target cone where the center position of the eyes of the object viewed by the electronic device is located, thereby determining the target light field coded image that the viewing object can clearly observe on the screen of the electronic device.

[0064] In some embodiments of the present application, before performing light field encoding based on the target visual cone corresponding to the first position information to obtain the target light field encoded image, that is, before the above-mentioned step S102, the following steps S401 to S402 may also be performed, and each step is described separately below.

[0065] S401: Determine a first area where the centers of both eyes are located according to first position information.

[0066] In some embodiments, the first area is the same size as each visual cone, and after determining the first position information of the center position of the binoculars relative to the electronic device screen, the first area where the center position of the binoculars is located can be determined. For example, if the size of the visual cone is 12 degrees, the first area can be an area within 12 degrees to the left and right of the center position of the binoculars.

[0067] S402: Determine a visual cone intersecting with the first area as a target visual cone.

[0068] In some embodiments, the visual cone intersecting the first area can be considered as a visual cone intersecting with the first area. When the first area and the visual cone are the same size, the target visual cone intersecting the first area may include one or two.

[0069] It can be understood that by determining the first area where the centers of the eyes of the electronic device viewing the object are located and determining the target visual cone based on the first area, it is convenient to determine the viewpoint image that needs to be light field encoded based on the target visual cone to achieve light field encoding.

[0070] In some embodiments of the present application, after performing light field encoding based on the target visual cone corresponding to the first position information to obtain a target light field encoded image, that is, after the above step S102, the following steps S501 to S503 may also be performed, and each step is described below.

[0071] S501: If the position information of the center positions of both eyes relative to the screen of the electronic device is changed from the first position information to the second position information.

[0072] In some embodiments, the second position information is different from the first position information, and the second position information may also be represented by position coordinates. For example, the second position information may also be position coordinates in a three-dimensional space coordinate system with the center of the electronic device screen as the origin, the horizontal direction of the electronic device screen as the x-axis, the vertical direction as the y-axis, and the direction perpendicular to the electronic device screen as the z-axis.

[0073] S502 : Perform light field coding based on the first viewing cone corresponding to the second position information to obtain a reference light field coded image.

[0074] In some embodiments, the first visual cone corresponding to the second position information may be a visual cone corresponding to the second area determined based on the second position information, the second area may be an area with the same size as the visual cone determined when the center position of both eyes is at the position corresponding to the second position information, the first visual cone may be a visual cone intersecting with the second area, and the second visual cone may be one or two.

[0075] In some embodiments, in the process of performing light field encoding based on the first visual cone corresponding to the second position information, reference offset information of the center position of both eyes relative to the center of the electronic device screen can be first determined based on the second position information; based on the reference offset information, light field encoding is performed on the viewpoint image corresponding to the first visual cone to obtain a reference light field encoded image, and the reference light field encoded image is different from the target encoded image.

[0076] In some embodiments, in the process of performing light field encoding on the viewpoint images corresponding to the first view cone based on reference offset information, light field encoding can be performed on the viewpoint images corresponding to the first view cone according to the reference offset information, the number of viewpoint images corresponding to the first view cone, the inclination angle and the number of lines of a single cylindrical lens in the screen of the electronic device, etc., so as to determine the reference light field coded image from the viewpoint images corresponding to the first view cone.

[0077] S503: Convert the target light field coded image into a reference light field coded image for display.

[0078] In some embodiments, after the position of the object being viewed on the electronic device screen moves, causing the center position of both eyes to change from the first position information to the second position information, it is necessary to switch the content displayed on the electronic device screen, that is, to convert the target light field coded image displayed in three dimensions at the first position information into a reference light field coded image for three-dimensional display.

[0079] It can be understood that after the position information of the center of both eyes relative to the screen of the electronic device is converted from the first position information to the second position information, light field encoding is performed based on the first visual cone corresponding to the second position information to obtain a reference light field coded image, and the originally displayed target light field coded image is replaced with the reference light field coded image, thereby realizing the determination of the content to be displayed according to the visual cone corresponding to the position of the viewed object, which can improve the angular resolution and spatial resolution and enhance the user's viewing experience.

[0080] In some embodiments of the present application, light field encoding is performed based on the first view cone corresponding to the second position information to obtain a reference light field coded image, that is, the above step S503 can be implemented by the following steps S5031A to S5033A, and each step is described below.

[0081] S5031A: If the first visual cone includes multiple ones, determine the size of the area where each first visual cone intersects with the first area where the center position of the two eyes is located.

[0082] In some embodiments, the size of the region where the first visual cone intersects with the first region where the centers of both eyes are located may also be the area of ​​the intersecting region. The sum of the areas of the regions where each first visual cone intersects with the first region where the centers of both eyes are located is the area of ​​the first region (or first visual cone). For example, if the area of ​​the first region is 8 square meters, the area of ​​the region where the first first visual cone intersects with the first region may be 2 square meters, and the area of ​​the region where the second first visual cone intersects with the first region may be 6 square meters.

[0083] In some embodiments, the size of the area where the first visual cone intersects the first area where the centers of both eyes are located may be the ratio of the area where the first visual cone intersects the first area where the centers of both eyes are located to the entire first area (or the first visual cone), and the sum of the ratios of the areas where the first visual cones intersect the first area where the centers of both eyes are located to the entire first area is 1. For example, the ratio corresponding to the first first visual cone may be 1 / 3, and the ratio corresponding to the second first visual cone may be 2 / 3.

[0084] S5032A: Based on the sizes of the respective regions and the positional relationship between the center positions of the binoculars and the respective first visual cones, determine the first viewpoint image from the viewpoint images corresponding to the respective first visual cones.

[0085] In some embodiments, the number of first viewpoint images corresponding to each first view cone can be determined based on the ratio of the size of the area where each first view cone intersects with the first area where the center of the binocular position is located. For example, if the first view cone includes two, each first view cone corresponds to 5 viewpoint images, and the ratio of the size of the area where the two first view cones intersect with the first area where the center of the binocular position is located is 3:2, then the number of first viewpoint images determined from the viewpoint images corresponding to the two first view cones can be 3 and 2 respectively.

[0086] In some embodiments, in the process of determining the first viewpoint image from the viewpoint images corresponding to each first view cone based on the size of the area where each first view cone intersects with the first area where the center position of both eyes is located, the positional relationship between the center position of both eyes and each first view cone can also be combined. The positional relationship can be the position of each first view cone relative to the center of both eyes. For example, the first view cone may be on the left side of the center position of both eyes, or on the right side of the center position of both eyes. Based on this positional relationship, it can be determined which viewpoint images corresponding to the first view cones are used as the first viewpoint image.

[0087] Exemplarily, if there are two first visual cones, the first first visual cone is on the left side of the center position of the binocular eyes, and the second first visual cone is on the right side of the center position of the binocular eyes, each first visual cone corresponds to 5 viewpoint images, and the ratio of the size of the area where the two first visual cones intersect the first area where the center position of the binocular eyes is located is 3:2, then for the first first visual cone, three consecutive viewpoint images close to the center position of the binocular eyes can be determined as its corresponding first viewpoint images; for the second first visual cone, two consecutive viewpoint images close to the center position of the binocular eyes can be determined as its corresponding first viewpoint images.

[0088] S5033A. Perform light field encoding on the first viewpoint image according to the offset information of the center positions of the eyes corresponding to the second position information relative to the center of the electronic device screen to obtain a reference light field encoded image.

[0089] In some embodiments, the offset information of the binocular center position corresponding to the second position information relative to the center of the electronic device screen may be a second distance between the binocular center position corresponding to the second position information and a first straight line, and the first straight line may be a straight line passing through the center of the electronic device screen and perpendicular to the electronic device screen. When the second position information is a position coordinate, the offset information may be determined according to the position coordinate.

[0090] In some embodiments, if the position coordinates corresponding to the second position information in a three-dimensional space coordinate system established with the center of the electronic device screen as the origin, the horizontal direction of the electronic device screen as the x-axis, the vertical direction as the y-axis, and the direction perpendicular to the electronic device screen as the z-axis are (m, n, q), m, n, q are the x-coordinate, y-coordinate, and z-coordinate, respectively, then the offset information of the center position of both eyes corresponding to the second position information relative to the center of the electronic device screen can be m.

[0091] In some embodiments, after determining the first viewpoint image corresponding to each first view cone and the offset information of the center position of the eyes corresponding to the second position information relative to the center of the electronic device screen, the first viewpoint image can be light field encoded according to the offset information to obtain a reference light field encoded image. The image content of the reference light field encoded image may come from the first viewpoint images corresponding to different first view cones, or may come from the first viewpoint image corresponding to the same first view cone.

[0092] It can be understood that, in the case where the first visual cones include multiple ones, the first viewpoint image is determined from the viewpoint images corresponding to the first visual cones according to the size of the area where each first visual cone intersects with the first area where the center position of the binocular eyes is located, and the positional relationship between the center position of the binocular eyes and each first visual cone, and then the first viewpoint image is light field encoded according to the offset information of the center position of the binocular eyes corresponding to the second position information relative to the center of the electronic device screen. This can achieve light field encoding across visual cones and ensure the resolution of the electronic device screen for three-dimensional display.

[0093] In some embodiments of the present application, light field encoding is performed based on the first view cone corresponding to the second position information to obtain a reference light field coded image, that is, step S503 can be implemented by the following steps S5031B to S5032B, and each step is described below.

[0094] S5031B: If the first viewing cone includes one, determine the viewpoint image corresponding to the first viewing cone.

[0095] In some embodiments, if the first visual cone includes one, it means that the second area where the center position of the eyes corresponding to the second position information is located completely overlaps with the first visual cone. In this case, the image that needs to be light field encoded is the image of all viewpoints corresponding to the first visual cone.

[0096] S5032B. Perform light field encoding on the viewpoint image corresponding to the first visual cone according to the offset information of the center position of the eyes corresponding to the second position information relative to the center of the electronic device screen to obtain a reference light field coded image.

[0097] In some embodiments, if there is only one first visual cone, light field encoding can be performed on the viewpoint image corresponding to the first visual cone based on the offset information of the binocular center position type corresponding to the second position information relative to the center of the electronic device screen, and the image content of the reference light field coded image obtained after light field encoding comes from the viewpoint image corresponding to the first visual cone.

[0098] It can be understood that, in the case where the first visual cone includes one, by performing light field encoding on the viewpoint image corresponding to the first visual cone based on the offset information of the center position of the eyes corresponding to the second position information relative to the center of the electronic device screen, a reference light field coded image suitable for viewing by the viewing object at the current position can be determined.

[0099] In some embodiments of the present application, before or after determining the first position information of the center position of both eyes of an object viewed by an electronic device relative to the screen of the electronic device, that is, before or after step S101, the following steps S601 to S603 may also be executed, and each step is described below.

[0100] S601: Determine first position information of the center positions of eyes of multiple electronic devices viewing an object relative to the screen of the electronic device.

[0101] In some embodiments, when there are multiple viewing objects watching the electronic device screen, the first position information of the center position of both eyes of each viewing object relative to the electronic device screen can be determined respectively. The first position information can be represented by position coordinates. For example, the three-dimensional coordinates of the center position of both eyes of each viewing object in a three-dimensional space coordinate system with the center of the electronic device screen as the origin, the horizontal direction of the electronic device screen as the x-axis, the vertical direction as the y-axis, and the direction perpendicular to the electronic device screen as the z-axis can be determined respectively, thereby obtaining each first position information. The first position information corresponding to different viewing objects may be different.

[0102] S602 , performing light field coding respectively on target visual cones corresponding to respective pieces of first position information to obtain a plurality of target light field coded images.

[0103] In some embodiments, light field encoding can be performed separately according to the target view cones corresponding to the first position information corresponding to each viewing object. If each first position information corresponds to only one target view cone, then the target view cones may be different; if each first position information corresponds to multiple target view cones, then the target view cones corresponding to each first position information may be the same.

[0104] In some embodiments, light field encoding may be performed simultaneously according to target visual cones corresponding to respective first position information without affecting each other. The viewpoint images of target visual cones corresponding to different first position information are not completely the same, and the target light field coded images obtained after light field encoding may also be different.

[0105] Exemplarily, if there are two viewing objects, the first viewing object corresponds to the target vision cone A, and the second viewing object corresponds to the target vision cone A and the target vision cone B, then for the first viewing object, light field encoding can be performed according to the viewpoint image corresponding to the target vision cone A to obtain the target light field encoded image corresponding to the first viewing object; for the second viewing object, the first viewpoint image can be determined from the viewpoint image corresponding to the target vision cone A and the first viewpoint image can be determined from the viewpoint image corresponding to the target vision cone B according to the size of the area where the target vision cone A intersects with the first area where the binocular centers of the second viewing object are located, the size of the area where the target vision cone B intersects with the first area where the binocular centers of the second viewing object are located, and the positional relationship between the target vision cone A and the target vision cone B relative to the binocular centers of the second viewing object, and then, light field encoding is performed based on the first viewpoint images determined from the two target vision cones to obtain the target light field encoded image corresponding to the second viewing object.

[0106] S603: Display each target light field coded image separately using a screen of an electronic device.

[0107] In some embodiments, after determining the target light field coded images corresponding to the center of each viewing object's eye position, each target light field coded image can be displayed simultaneously on the electronic device screen, and the positions of each target light field coded image displayed on the electronic device screen may be different.

[0108] It can be understood that when multiple viewers are viewing the electronic device screen at the same time, light field encoding is performed based on the target visual cone corresponding to the first position information of the center position of each viewing object's eyes relative to the electronic device screen. After each target light field encoded image obtained after light field encoding is displayed in three dimensions on the electronic device screen, each viewing object can see clear image content, thereby satisfying the viewing experience of different viewing objects.

[0109] In an embodiment of the present application, first position information of the center position of both eyes of an object being viewed by an electronic device relative to the screen of the electronic device is determined; light field encoding is performed based on the target visual cone corresponding to the first position information to obtain a target light field encoded image; the target visual cone is any one of a plurality of visual cones obtained by dividing the visible area corresponding to the viewing angle of the screen of the electronic device, and different visual cones correspond to different light field encoding methods; the target light field encoded image is three-dimensionally displayed through the screen of the electronic device. In this way, since the light field encoding methods corresponding to different visual cones are different, the light field encoding method in the target visual cone corresponding to the center position of both eyes of the object being viewed by the electronic device remains unchanged, and therefore, the screen of the electronic device will not frequently change the image content of the three-dimensional display, thereby avoiding the problem of dizziness of the object being viewed by the electronic device and improving the viewing experience of the object being viewed by the electronic device.

[0110] Next, the implementation process of the application embodiment in the actual application scenario is introduced.

[0111] like Figure 5 FIG. 1 is a flow chart of an image display method based on human eye tracking and light field coding fusion provided by the present application, the method comprising:

[0112] S701, determine the spatial coordinate value of the person relative to the screen origin (equivalent to "determining the first position information of the center position of the eyes of the electronic device viewing the object relative to the electronic device screen" in other embodiments).

[0113] In some embodiments, the eye tracking module of the electronic device can output the spatial coordinate values ​​of the eye in the three-dimensional space coordinate system at a high frame rate, such as Figure 6 As shown, the three-dimensional space coordinate system takes the center point O of the screen as the origin, the x-axis direction is the horizontal direction of the screen, the y-axis direction is the vertical upward direction of the screen, and the z-axis direction is the direction perpendicular to the screen from the inside to the outside.

[0114] For example, Figure 7 As shown in the figure, using the repeatability of multi-viewpoint light fields, a small-angle grating structure is designed, with a 12-degree cone. The entire screen can support a viewing angle of 60 degrees, a total of 5 cone structures, and the best viewing angle is 2m. The virtual rendering camera of the entire system architecture is placed as follows Figure 7 As shown, the overall width of the camera placement can be expressed by formula (1):

[0115] width = 2*tan30°*2 (1);

[0116] The uniformly placed camera position baseline can be expressed by formula (2):

[0117] baseline=width / (120-1) (2);

[0118] Normalize the human eye position to the optimal viewing distance along the z-axis. (Ex, Ey, Ez) is the coordinate of the middle point of the human eye. Taking 2m as the optimal viewing distance as an example, the normalized x-coordinate value of the human eye is: x=Ex*2 / Ez.

[0119] S702 , selecting 24 cameras centered at the human eye position as target rendering points according to the human eye position.

[0120] For example, Figure 7 As shown in the figure, when there are 120 virtual rendering cameras, 24 virtual rendering cameras can be selected as target rendering cameras according to the normalized human eye position as the midpoint, and each virtual rendering camera corresponds to a viewpoint image. This solution supports importing existing 3D models, or using dense expressions reconstructed by sparse cameras, such as Mesh models, Gaussian or Neural Radiance Fields (NeRF) expressions, as long as it supports multi-view rendering or can generate multi-view images.

[0121] S703, calculating the offset according to the position of the human eyes (equivalent to "determining the offset information of the center positions of both eyes relative to the center of the electronic device screen according to the first position information" in other embodiments), and performing light field coding display.

[0122] The rendered images of the 24 target cameras are used as the input of the light field encoding, and the light field encoding is offset according to the position of the human eye. The light field encoding method can be implemented through the following process:

[0123]

[0124] Among them, Tg_lens is the angle of the microlens, lineNum is the number of lines of the lens, img_num is the number of viewpoint images (24), eye_bias is the offset calculated according to the position of the human eye, bias is the fitting error of the lens, i, j, k are the horizontal coordinate, vertical coordinate and RGB channel number of the pixel point on the screen respectively, viewp is the viewpoint image from which the current sub-pixel point is taken, and through viewp, the viewpoint image corresponding to the current position that needs to be rendered to the screen can be determined, that is, the light field coding image (equivalent to the "target light field coding image" in other embodiments) is obtained, and finally the light field coding image is rendered to the screen for three-dimensional display.

[0125] The image display method based on the fusion of human eye tracking and light field coding provided in the present application uses a human eye tracking plus light field solution to form a multi-viewpoint light field that always follows the human eye at the position of the human eye, and selects content according to the position of the human eye for light field coding display, which can provide high angular resolution and spatial resolution at the same time, and can reduce people's dizziness.

[0126] The present application also provides an electronic device, Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the electronic device 800 includes:

[0127] A plurality of image acquisition devices 801,

[0128] A processing device 802 is connected to the multiple image acquisition devices 801, and the processing device 802 determines first position information of the center position of the eyes of the electronic device viewing the object relative to the screen of the electronic device; performs light field encoding based on the target visual cone corresponding to the first position information to obtain a target light field encoded image acquired by the image acquisition device; wherein the target visual cone is any one of the multiple visual cones obtained by dividing the visible area corresponding to the viewing angle of the screen of the electronic device, and different visual cones correspond to different light field encoding methods;

[0129] The display device 803 is connected to the processing device 802 and displays the target light field coded image in three dimensions through the screen of the electronic device.

[0130] It should be noted that the description of the electronic device in the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in this embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0131] It should be noted that in the embodiment of the present application, if the above-mentioned image display method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant solution. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0132] Accordingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image display method provided in the above embodiment is implemented.

[0133] The description of the above storage medium embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the storage medium embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0134] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises at least one ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0135] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0136] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0137] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0138] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, disks or optical disks.

[0139] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a product to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0140] The above are only implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technical object familiar with the technical field can be easily thought of within the technical scope disclosed in the present application. Changes or substitutions should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An image display method, comprising: Determine first position information of the center position of both eyes of the electronic device viewing the object relative to the screen of the electronic device; Performing light field encoding based on a target visual cone corresponding to the first position information to obtain a target light field encoded image; wherein the target visual cone is any one of a plurality of visual cones obtained by dividing a visible area corresponding to a viewing angle of the electronic device screen, and different visual cones correspond to different light field encoding methods; The target light field coded image is displayed in three dimensions through the electronic device screen.

2. According to the method of claim 1, performing light field coding on the target visual cone corresponding to the first position information to obtain a target light field coded image comprises: Determine, according to the first position information, offset information of the center positions of the binoculars relative to the center of the screen of the electronic device; Based on the offset information, light field encoding is performed on the viewpoint image corresponding to the target viewing cone to obtain the target light field encoded image.

3. The method according to claim 2, further comprising: Acquiring a viewing angle of the electronic device screen; Dividing the visible area corresponding to the viewing angle into a plurality of viewing cones with the same angle; The size of the viewing cone is determined by the grating structure of the electronic device screen; Determine the viewpoint image corresponding to each view cone according to the number of viewpoint images, the number of view cones, and the acquisition angle of the viewpoint image; the viewpoint image is a two-dimensional image obtained by acquiring the same object from multiple perspectives; The number of viewpoint images corresponding to each viewing cone is the same, and the acquisition angles of the viewpoint images corresponding to two adjacent viewing cones and the acquisition angles of multiple viewpoint images corresponding to the same viewing cone all change continuously.

4. The method according to claim 3, further comprising: Determine a first area where the center positions of the two eyes are located according to the first position information; The first area is the same size as each visual cone; A viewing cone intersecting the first area is determined as the target viewing cone.

5. The method according to claim 2, wherein determining the offset information of the binocular center position relative to the center of the electronic device screen according to the first position information comprises: Determine a first distance from the center position of the binoculars to a first straight line according to the first position information; The first straight line is a straight line passing through the center point of the electronic device screen and perpendicular to the electronic device screen; The first distance is determined as the offset information.

6. The method according to claim 1, further comprising: If the position information of the center position of both eyes relative to the screen of the electronic device is changed from the first position information to the second position information; Performing light field coding based on the first visual cone corresponding to the second position information to obtain a reference light field coded image; the reference light field coded image is different from the target coded image; The target light field coded image is converted into the reference light field coded image for display.

7. The method according to claim 6, wherein the step of performing light field coding based on the first view cone corresponding to the second position information to obtain a reference light field coded image comprises: If the first visual cone includes multiple ones, determine the size of the area where each first visual cone intersects with the first area where the center position of the binoculars is located; Based on the sizes of the respective regions and the positional relationship between the binocular center positions and the respective first visual cones, determining the first viewpoint image from the viewpoint images corresponding to the respective first visual cones; According to the offset information of the center positions of both eyes corresponding to the second position information relative to the center of the screen of the electronic device, the first viewpoint image is light field encoded to obtain the reference light field encoded image.

8. The method according to claim 6, wherein the step of performing light field coding based on the first view cone corresponding to the second position information to obtain a reference light field coded image comprises: If the first viewing cone includes one, determining a viewpoint image corresponding to the first viewing cone; According to the offset information of the center positions of the eyes corresponding to the second position information relative to the center of the screen of the electronic device, light field encoding is performed on the viewpoint image corresponding to the first visual cone to obtain the reference light field encoded image.

9. The method according to claim 1, further comprising: Determine first position information of the center positions of eyes of multiple electronic devices viewing an object relative to the screen of the electronic device; Performing light field coding respectively on the target visual cones corresponding to the respective first position information to obtain a plurality of target light field coded images; The electronic device screen is used to display each target light field coded image respectively.

10. An electronic device comprising: Multiple image acquisition devices, A processing device is connected to the multiple image acquisition devices, the processing device determines first position information of the center position of both eyes of the electronic device viewing the object relative to the screen of the electronic device; performs light field encoding based on the target visual cone corresponding to the first position information to obtain a target light field encoded image acquired by the image acquisition device; wherein the target visual cone is any one of the multiple visual cones obtained by dividing the visible area corresponding to the viewing angle of the screen of the electronic device, and different visual cones correspond to different light field encoding methods; A display device is connected to the processing device and performs three-dimensional display of the target light field coded image through the screen of the electronic device.

Images