Image display method, device, electronic equipment and storage medium

Abstract

This disclosure relates to an image display method, apparatus, electronic device, and storage medium. The method includes: determining the field-of-view parameters of a human eye relative to the screen of a virtual display device; determining a second image of the real object under the same field-of-view parameters based on a first image obtained by the virtual display device for a real object under the same field-of-view parameters; rendering a preset virtual object according to the same field-of-view parameters to obtain a third image; and displaying a target image obtained by superimposing the third image and the second image on the screen of the virtual display device. In this solution, the second image of the real object under the same field-of-view parameters is determined, and the field-of-view parameters are used as rendering parameters to render the virtual object. This ensures that the target image obtained by superimposing the third image and the second image conforms to the observation habits of the human eye in a real scene, improving the realism of the augmented reality image.

Description

Technical Field

[0001] This disclosure relates to the field of augmented reality technology, and more particularly to image display methods, apparatus, electronic devices, and storage media. Background Technology

[0002] In augmented reality (AR) services, users gain an augmented reality experience by observing a screen. That is, virtual objects are superimposed on real-world images to synthesize augmented reality images and present them on the screen.

[0003] During their research into existing technologies, the inventors discovered that current industry solutions use the device's camera's native field of view (FOV) to convert virtual objects into virtual 3D scenes and overlay them onto real-world scenes. This method of observing the world through a camera causes a discrepancy between the user's visual perception of the virtual scene and their visual perception of the real scene, reducing the realism of augmented reality images. Summary of the Invention

[0004] This disclosure provides an image display method, apparatus, electronic device, and storage medium to at least solve the problem in the related art of inconsistency between the user's visual perception of a virtual scene and their visual perception of a real scene. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, an image display method is provided, the method comprising:

[0006] Determine the field of view parameters of the human eye relative to the screen of the virtual display device;

[0007] Based on the first image of the real object obtained by the virtual display device under the camera field of view parameters, a second image of the real object under the human eye field of view parameters is determined.

[0008] Based on the human eye field angle parameters, a preset virtual object is rendered to obtain a third image;

[0009] The target image, obtained by superimposing the third image and the second image, is displayed on the screen of the virtual display device.

[0010] Optionally, determining the human eye's field of view parameters relative to the screen of the virtual display device includes:

[0011] Obtain the screen width of the virtual display device, and obtain the distance from the human eye to the screen;

[0012] The field of view parameters of the human eye are determined based on the screen width and the distance from the human eye to the screen.

[0013] Optionally, obtaining the distance from the human eye to the screen includes:

[0014] The user's facial image is acquired using the camera of the virtual display device;

[0015] Determine the distance between the pupils of the two eyes in the facial image to obtain the interpupillary distance;

[0016] The distance from the human eye to the screen is obtained by determining the quotient of the pupil distance between the two eyes and the pre-acquired standard pupil distance; the standard pupil distance is the distance between the pupils of the two eyes in a facial image taken at a distance of 1 meter from the screen.

[0017] Optionally, determining the second image of the real object under the human eye's field of view parameters based on the first image obtained by the virtual display device for the real object under the camera's field of view parameters includes:

[0018] Using the human eye field of view parameters, the first position of each pixel in the first image is transformed in the screen coordinate system of the virtual display device to obtain the second position of each pixel in the world coordinate system.

[0019] The imaging position on the first image corresponding to the second position is determined to obtain the third position;

[0020] Obtain the pixel value at the third position from the first image, and use the pixel value as the target pixel value at the first position;

[0021] The image with the pixel value at the first position being the target pixel value is determined as the second image of the real object under the human eye field of view parameters.

[0022] Optionally, the step of using the human eye field angle parameter to transform the first position of each pixel in the first image in the screen coordinate system of the virtual display device to obtain the second position of each pixel in the world coordinate system includes:

[0023] Obtain the distance between the real object and the camera to get the object distance;

[0024] The product of the object distance and the human eye field angle parameter is determined to obtain the first product;

[0025] The quotient of the product of the first position of each pixel in the first image in the screen coordinate system of the virtual display device and the first position is determined to obtain the second position of each pixel in the world coordinate system.

[0026] Optionally, obtaining the distance between the real object and the camera includes:

[0027] The distance between the real object and the camera is obtained using a stereo color camera or a neural network algorithm.

[0028] Optionally, the human eye includes the left and right eyes, the screen of the virtual display device includes a left screen and a right screen, and the method further includes:

[0029] Determine the left eye field of view parameters of the left eye relative to the left screen of the virtual display device, and determine the right eye field of view parameters of the right eye relative to the right screen of the virtual display device;

[0030] Based on the first image of the real object captured by the virtual display device under the camera field of view parameters, it is determined that the real object corresponds to the second image of the left eye under the left eye field of view parameters, and it is determined that the real object corresponds to the second image of the right eye under the right eye field of view parameters.

[0031] Based on the left eye field of view parameters, a preset virtual object is rendered to obtain a third image for the left eye, and the right eye field of view parameters are used as rendering parameters to render the virtual object to obtain a third image for the right eye.

[0032] The left-eye target image, obtained by superimposing the left-eye third image and the left-eye second image, and the right-eye target image, obtained by superimposing the right-eye third image and the right-eye second image, are displayed on the left screen and the right screen, respectively.

[0033] According to a second aspect of the present disclosure, an image display apparatus is provided, the apparatus comprising:

[0034] The human eye field of view parameter determination module is configured to determine the human eye field of view parameters relative to the screen of the virtual display device;

[0035] The second image determination module is configured to execute a first image of the real object obtained by the virtual display device for the real object under the camera field of view parameters, and determine a second image of the real object under the human eye field of view parameters.

[0036] The rendering module is configured to render a preset virtual object based on the human eye field angle parameters to obtain a third image;

[0037] The display module is configured to display a target image obtained by superimposing the third image and the second image on the screen of the virtual display device.

[0038] Optionally, the human eye field angle parameter determination module is specifically configured to execute:

[0039] Obtain the screen width of the virtual display device, and obtain the distance from the human eye to the screen;

[0040] The field of view parameters of the human eye are determined based on the screen width and the distance from the human eye to the screen.

[0041] Optionally, the human eye field angle parameter determination module is further configured to perform:

[0042] The user's facial image is acquired using the camera of the virtual display device;

[0043] Determine the distance between the pupils of the two eyes in the facial image to obtain the interpupillary distance;

[0044] The distance from the human eye to the screen is obtained by determining the quotient of the pupil distance between the two eyes and the pre-acquired standard pupil distance; the standard pupil distance is the distance between the pupils of the two eyes in a facial image taken at a distance of 1 meter from the screen.

[0045] Optionally, the second image determination module is specifically configured to perform:

[0046] Using the human eye field of view parameters, the first position of each pixel in the first image is transformed in the screen coordinate system of the virtual display device to obtain the second position of each pixel in the world coordinate system.

[0047] The imaging position on the first image corresponding to the second position is determined to obtain the third position;

[0048] Obtain the pixel value at the third position from the first image, and use the pixel value as the target pixel value at the first position;

[0049] The image with the pixel value at the first position being the target pixel value is determined as the second image of the real object under the human eye field of view parameters.

[0050] Optionally, the second image determination module is further configured to perform:

[0051] Obtain the distance between the real object and the camera to get the object distance;

[0052] The product of the object distance and the human eye field angle parameter is determined to obtain the first product;

[0053] The quotient of the product of the first position of each pixel in the first image in the screen coordinate system of the virtual display device and the first position is determined to obtain the second position of each pixel in the world coordinate system.

[0054] Optionally, the second image determination module is further configured to perform:

[0055] The distance between the real object and the camera is obtained using a stereo color camera or a neural network algorithm.

[0056] Optionally, the device further includes:

[0057] The left and right eye field of view determination module is configured to determine the left eye field of view parameters of the left eye relative to the left screen of the virtual display device, and to determine the right eye field of view parameters of the right eye relative to the right screen of the virtual display device.

[0058] The left and right eye second image determination module is configured to perform the following: based on the first image captured by the virtual display device for the real object under the camera field of view parameters, determine that the real object corresponds to the left eye second image under the left eye field of view parameters, and determine that the real object corresponds to the right eye second image under the right eye field of view parameters.

[0059] The left and right eye image rendering modules are configured to render a preset virtual object according to the left eye field of view parameter to obtain a left eye third image, and to render the virtual object using the right eye field of view parameter as a rendering parameter to obtain a right eye third image.

[0060] The left and right eye display modules are configured to display a left-eye target image obtained by superimposing the left-eye third image and the left-eye second image, and a right-eye target image obtained by superimposing the right-eye third image and the right-eye second image, respectively, on the left screen and the right screen.

[0061] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0062] processor;

[0063] Memory used to store the processor's executable instructions;

[0064] The processor is configured to execute the instructions to implement the image display method as described in the first aspect.

[0065] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of a server, enables the server to perform the image display method as described in the first aspect.

[0066] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the image display method described in the first aspect.

[0067] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0068] In embodiments of this disclosure, the field-of-view parameters of the human eye relative to the screen of a virtual display device are determined; based on a first image of the real object obtained by the virtual display device under the field-of-view parameters of the camera, a second image of the real object under the same field-of-view parameters is determined; based on the field-of-view parameters, a preset virtual object is rendered to obtain a third image; and a target image obtained by superimposing the third image and the second image is displayed on the screen of the virtual display device. In this scheme, the second image of the real object under the same field-of-view parameters is determined, and the field-of-view parameters are used as rendering parameters to render the virtual object. This ensures that the target image obtained by superimposing the third image and the second image conforms to the observation habits of the human eye in a real scene, improving the realism of the augmented reality image.

[0069] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0070] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0071] Figure 1 This is a flowchart illustrating the steps of a first image display method according to an exemplary embodiment;

[0072] Figure 2 A schematic diagram illustrating the field of view of the human eye and the field of view of the camera provided in an embodiment of this disclosure;

[0073] Figure 3 A schematic diagram of the first image, second image, third image, and target image provided for embodiments of this disclosure;

[0074] Figure 4 This is a flowchart illustrating the steps of a second image display method according to an exemplary embodiment;

[0075] Figure 5 This is a flowchart illustrating the steps of a third image display method according to an exemplary embodiment;

[0076] Figure 6 This is a structural block diagram of an image display device according to an exemplary embodiment;

[0077] Figure 7 This is a block diagram illustrating an electronic device for image display according to an exemplary embodiment. Detailed Implementation

[0078] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0079] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0080] Figure 1 This is a flowchart illustrating the steps of a first image display method according to an exemplary embodiment, such as... Figure 1 As shown, the method includes the following steps.

[0081] In step S11, the field of view parameters of the human eye relative to the screen of the virtual display device are determined.

[0082] The virtual display device in this embodiment can be an augmented reality (AR) display device. AR display devices combine real-world and virtual scenes, specifically displaying and interacting with images captured by a camera by combining virtual visuals. In an AR system, the virtual world is overlaid on the real world, supplementing the real world as perceived by viewers with useful information. For a virtual display device to create a sense of immersion, a suitable field of view (FOV) is crucial. The size of the FOV determines the range of objects entering the eye, directly affecting the quality of the immersive experience.

[0083] The field of view of a camera is defined as the angle between the two edges of the lens through which the image of the target object can pass, with the camera of the virtual display device as the vertex.

[0084] The current industry solution is to use the device's camera's native field of view, i.e., the camera's field of view, to convert virtual objects into virtual 3D scenes and overlay them onto the real-world scene. The user experience of this method is that the user observes the world through the camera, which makes the user's visual perception of the virtual scene inconsistent with the visual perception of the real scene, reducing the user's sense of realism.

[0085] This solution determines the human eye's field of view parameters relative to the screen of a virtual display device, and uses this field of view to convert virtual objects into virtual 3D scenes. This method allows users to observe the world through their own eyes while using the virtual display device, ensuring that the user's visual perception of the virtual scene is consistent with their visual perception of the real scene, thereby enhancing the user's sense of realism.

[0086] Similar to the field of view of a camera, the field of view of a human eye refers to the angle formed by the two edges of the maximum range through which the image of the target object can pass through the two eyes, with the midpoint of the line connecting the two pupils of the human eye as the vertex.

[0087] The field of view parameter of the human eye for virtual display devices refers to the angle between the two edges of the screen of the virtual display device, with the midpoint of the line connecting the two pupils of the human eye as the vertex, which is the maximum range that can be communicated between the two eyes.

[0088] Figure 2 This is a schematic diagram of the human eye's field of view and the camera's field of view, provided in an embodiment of this disclosure.

[0089] like Figure 2 As shown, Fh represents the field of view of the human eye, and Fd represents the field of view of the camera.

[0090] In step S12, based on the first image of the real object obtained by the virtual display device for the real object under the camera field of view parameters, a second image of the real object under the human eye field of view parameters is determined.

[0091] Augmented reality (AR) display devices combine virtual images with images captured by a camera for presentation and interaction. The first image here is the image captured by the camera of the virtual display device, obtained for a real object under the camera's field of view parameters. To enhance the user's realistic experience, distortion correction can be used to convert the first image (based on the camera's field of view parameters) into a second image (based on the human eye's field of view parameters).

[0092] In step S13, the preset virtual object is rendered according to the human eye field angle parameters to obtain the third image.

[0093] Virtual objects are virtual scene objects used in augmented reality to be overlaid on real-world scenes. Virtual objects can be pre-defined according to requirements.

[0094] The current industry solution uses the camera's field of view parameter to render virtual objects. This solution uses the human eye's field of view as the rendering parameter to render virtual objects, and the resulting rendering image is a third image.

[0095] Specifically, most 3D engines in the industry currently support adjusting camera parameters when rendering objects, and the human eye's field of view can be used as the camera parameter when rendering virtual objects.

[0096] In step S14, the target image obtained by superimposing the third image and the second image is displayed on the screen of the virtual display device.

[0097] The overlay area of ​​the third image on the second image can be preset, and the pixel values ​​of the pixels in the overlay area of ​​the second image can be replaced with the pixel values ​​of the corresponding points in the third image to obtain the target image.

[0098] In the target image, both the third and second images are obtained based on the human eye's field of view parameters. The target image is displayed on the screen of the virtual display device, so that the virtual scene displayed in the virtual display device conforms to the user's visual perception habits in the real scene, thereby improving the user's sense of realism.

[0099] Figure 3 A schematic diagram of a first image, a second image, a third image, and a target image provided for embodiments of this disclosure.

[0100] Reference Figure 3 I1 is the first image, which is the image of the real object obtained by the virtual display device under the camera's field of view parameters. I2 is the second image, which is the image of the real object under the human eye's field of view parameters. I3 is the third image, which is the image obtained by rendering the virtual object. I4 is the target image, which is the image obtained by superimposing the second image and the third image.

[0101] Depend on Figure 3 As can be seen, the perspective relationship at the dashed box in I2 has changed compared to I1, and the size of the objects has also changed. This is due to the change in the field of view parameters. I2 is rendered using human eye field of view parameters, which better matches the observation habits of the human eye in real-world scenes. Furthermore, I3 is also rendered using human eye field of view parameters, and I3 also matches the observation habits of the human eye in real-world scenes. Thus, the augmented reality image I4, obtained by superimposing I2 and I3, is equivalent to the user observing the world through their own eyes, making the user's visual perception of the virtual scene consistent with their visual perception of the real scene, thereby enhancing the user's sense of realism.

[0102] In summary, in the embodiments of this disclosure, the human eye's field of view parameters relative to the screen of the virtual display device are determined; based on the first image of the real object obtained by the virtual display device under the camera's field of view parameters, a second image of the real object under the human eye's field of view parameters is determined; based on the human eye's field of view parameters, a preset virtual object is rendered to obtain a third image; and the target image obtained by superimposing the third image and the second image is displayed on the screen of the virtual display device. In this solution, the second image of the real object under the human eye's field of view parameters is determined, and the human eye's field of view parameters are used as rendering parameters to render the virtual object, so that the target image obtained by superimposing the third image and the second image conforms to the human eye's observation habits in a real scene, improving the realism of the augmented reality image.

[0103] Figure 4 This is a flowchart illustrating the steps of a second image display method according to an exemplary embodiment, such as... Figure 4 As shown, the method includes the following steps.

[0104] In step S21, the screen width of the virtual display device is obtained, and the distance from the human eye to the screen is obtained.

[0105] The screen width of a virtual display device can be obtained from the device's factory specifications or through prior measurement.

[0106] Because different users have different facial contours and eye sizes, the distance between their eyes and the screen will vary when wearing a virtual display device. Furthermore, facial movements while wearing the device can also alter this distance. Therefore, it is impossible to measure the exact distance between the user's eyes and the screen beforehand.

[0107] Here, the camera can capture an image of the user's face to obtain the distance between the pupils of both eyes. This distance is then compared and calculated based on a pre-obtained standard pupil distance to determine the distance from the user's eyes to the screen.

[0108] In one possible implementation, obtaining the distance from the human eye to the screen includes:

[0109] Step S211: Use the camera of the virtual display device to acquire the user's facial image;

[0110] Step S212: Determine the distance between the pupils of the two eyes in the face image to obtain the pupil distance between the two eyes;

[0111] Step S213: Determine the quotient of the pupil distance between the two eyes and the pre-acquired standard pupil distance to obtain the distance from the human eye to the screen; the standard pupil distance is the distance between the pupils of the two eyes in a facial image taken at a distance of 1 meter from the screen for a standard human face.

[0112] In steps S211-S213, the length of the line connecting the pupils of both eyes in a standard face image at a distance of 1 meter from the screen is pre-obtained to obtain the standard pupil distance. A standard face can be a face model with an average size that is generally accepted in the industry. Since the standard face is of average size, using it as a reference for obtaining the user's pupil distance will keep the error within a controllable range.

[0113] Specifically, after the user wears the virtual display device, the device's camera is used to capture the user's facial image. Through image recognition, the positions of the eyes in the facial image are determined, and the distance between the pupils is calculated to obtain the interpupillary distance.

[0114] Then, using the standard pupillary distance as a reference, the distance from the user's eye to the screen is calculated. Specifically, the distance De from the user's eye to the screen is calculated using the following method:

[0115] De = de / ds(1)

[0116] Where De represents the distance from the user's eye to the screen, de represents the distance between the user's pupils, and ds represents the standard pupillary distance.

[0117] In steps S211-S213, by using the method described above to acquire the user's facial image and using the standard pupillary distance as a reference, the distance from the user's eyes to the screen can be obtained in real time, quickly and accurately. Since the standard face is of average size, using the standard face as a reference for acquiring the user's pupillary distance makes the calculated distance from the user's eyes to the screen more accurate and has controllable error.

[0118] In step S22, the field of view parameters of the human eye are determined based on the screen width and the distance from the human eye to the screen.

[0119] The human eye's field of view includes the horizontal field of view, the vertical field of view, and the diagonal field of view. When using virtual display devices, we generally use the most important parameter in the human eye's viewing characteristics—the horizontal field of view. Therefore, in this solution, the camera's field of view and the human eye's field of view refer to the horizontal field of view of the camera and the horizontal field of view of the human eye, respectively.

[0120] It is understood that in other embodiments, the camera field of view and the human eye field of view can also be the vertical field of view of the camera and the vertical field of view of the human eye, or the diagonal field of view of the camera and the diagonal field of view of the human eye. All of these embodiments are within the protection scope of this disclosure.

[0121] Specifically, the calculation method for the human eye field angle parameter Fh in this scheme is as follows:

[0122] Fh=2*arctan(W*0.5 / De)(2)

[0123] Where W represents the screen width of the virtual display device, and De represents the distance from the human eye to the screen. arctan() is the arctangent function.

[0124] In step S23, the first image of the real object is captured by the virtual display device under the camera's field of view parameters.

[0125] When capturing a real object using a virtual display device, the field of view parameter is set to the camera's field of view to capture the first image.

[0126] In step S24, the first position of each pixel in the first image in the screen coordinate system of the virtual display device is transformed using the human eye field angle parameter to obtain the second position of each pixel in the world coordinate system.

[0127] The pixels on the screen of a virtual display device are called device pixels, which refer to the smallest physical unit that the device can control for display; they are individual points on the screen. The positions of the pixels on the screen are fixed.

[0128] The first position is the location of a screen pixel in the camera coordinate system. Using the principle of triangle similarity in the imaging process, this first position can be converted into a second position on the corresponding real object in the world coordinate system. During this conversion, the included angle of the imaging triangle is based on the human eye's field of view, ensuring that the correspondence between the first and second positions is set under the premise of the human eye's field of view parameters.

[0129] In one possible implementation, step S24 includes:

[0130] Step S241: Obtain the distance between the real object and the camera to get the object distance;

[0131] Step S242: Determine the product of the object distance and the human eye's field of view parameter to obtain the first product;

[0132] Step S243: Determine the quotient of the first position of each pixel in the first image in the screen coordinate system of the virtual display device and the first product, and obtain the second position of each pixel in the world coordinate system.

[0133] In steps S241-S243, the object distance refers to the distance from the object to the optical center of the lens. In this scheme, the object distance is the distance between the real object and the optical center of the device camera.

[0134] Based on the principle of triangle similarity in the imaging process, the transformation relationship between the first and second positions, including the object distance and the human eye's field of view parameters, is obtained as follows:

[0135] P'=Pi / Fh*Dobj(3)

[0136] Where P' represents the second position, pi represents the first position, Fh represents the human eye field angle parameter, and Dobj represents the object distance.

[0137] Using the above conversion formula, we can obtain the first position pi of each pixel on the screen of the virtual display device, and the second position P' of the real object in the world coordinate system.

[0138] In one possible implementation, obtaining the distance between the real object and the camera includes:

[0139] Use a stereo color camera or neural network algorithm to obtain the distance between the real object and the camera.

[0140] Specifically, a stereo color camera (RGBD) refers to a camera that measures distance using physical methods, producing an RGB image plus a depth map. The depth map represents the distance between various parts of the subject and the camera. Thus, a stereo color camera can obtain the distance between the real image and the camera.

[0141] The distance between a real object and a camera can be obtained using a neural network algorithm. The specific steps include: acquiring close-up and distant images of the real object along the camera's optical axis; using a neural network for target recognition to identify the same target in both the close-up and distant images, generating target bounding boxes in both images; based on the neural network, determining the position and size of the target bounding boxes; and then substituting this information into an optical equation to obtain the distance between the real object and the camera.

[0142] In step S25, the second position is determined to correspond to the imaging position on the first image, and the third position is obtained.

[0143] The second position of the real object in the real world is calculated from the first position of the pixel on the screen. Then, the imaging position of the second position on the first image is determined to obtain the third position.

[0144] The second position is the position in the world coordinate system. The transformation relationship between the world coordinate system and the image coordinate system can be used to convert the second position into the third position in the image coordinate system.

[0145] The conversion between the world coordinate system and the image coordinate system is a current technique and will not be described in detail here.

[0146] In step S26, the pixel value at the third position is obtained from the first image, and the pixel value is used as the target pixel value at the first position.

[0147] Obtain the pixel value at the third position on the first image, and use this pixel value as the pixel value at the first position on the screen of the virtual display device. In this way, each pixel on the screen of the virtual display device is assigned the target pixel value.

[0148] In step S27, the image with the pixel value at the first position being the target pixel value is determined as the second image corresponding to the real object under the human eye field of view parameters.

[0149] Each pixel on the screen of the virtual display device is assigned a target pixel value to form a second image. Because the transition from the first position to the second position uses the human eye's field of view parameter, the target pixel value is the pixel value under the human eye's field of view parameter. Thus, the second image is the image under the human eye's field of view parameter.

[0150] Reference Figure 2 , Figure 2 The “position of the object on the screen” shown refers to the position of each pixel on the screen of the virtual display device, i.e., the first position; the “object position corresponding to the pixel” refers to the position of the first position on the real object in the world coordinate system, i.e., the second position; and the “corrected position” refers to the imaging position of the second position on the first image, i.e., the third position.

[0151] Steps S23-S27 described above are the process steps for converting the first image obtained by the virtual display device under the camera's field of view parameters into a second image under the human eye's field of view parameters. In this process, the first position is first converted into a second position corresponding to the real object in the world coordinate system. During this conversion, the included angle of the imaging triangle is based on the human eye's field of view, ensuring that the correspondence between the first and second positions is set under the premise of the human eye's field of view parameters.

[0152] Then, the imaging position on the first image corresponding to the second position is determined to obtain the third position. The pixel value at the third position is then used as the target pixel value at the first position. In this way, each pixel on the screen of the virtual display device is assigned the target pixel value, which is under the human eye's field of view parameters, making the second image an image under the human eye's field of view parameters. Thus, by using only a simple image transformation relationship, the image of a real object under the camera's field of view is converted into an image under the human eye's field of view parameters. The computational and time complexity are relatively low, and the processing speed is fast. This allows users to observe the world under the human eye's field of view parameters in real time through the virtual display device, improving the user's sense of realism.

[0153] In step S28, the preset virtual object is rendered according to the human eye field angle parameters to obtain the third image.

[0154] In this embodiment of the disclosure, step S28 can refer to step S13, and will not be repeated here.

[0155] In step S29, the target image obtained by superimposing the third image and the second image is displayed on the screen of the virtual display device.

[0156] In this embodiment of the disclosure, step S29 can refer to step S14, and will not be repeated here.

[0157] In conclusion, Figure 4 The image display method in the text, in addition to having Figure 1 In addition to the beneficial effects of the image display method in the image display method, it also uses only simple image transformation relationships to convert the first image of the real object under the camera's field of view into the second image under the human eye's field of view parameters. The computational and time complexity are relatively low, and the processing speed is fast, so that users can observe the world under the human eye's field of view parameters in real time through virtual display devices, which improves the user's sense of realism.

[0158] Figure 5 This is a flowchart illustrating the steps of a third image display method according to an exemplary embodiment, such as... Figure 5 As shown, the method includes the following steps.

[0159] In step S31, the left eye field of view parameter for the left eye relative to the left screen of the virtual display device is determined, and the right eye field of view parameter for the right eye relative to the right screen of the virtual display device is determined.

[0160] In this embodiment of the disclosure, due to the distance between the two eyes, there are slight differences in the field of view parameters of the left and right eyes. The human eye field of view parameters are obtained by converting the second image and rendering the third image using the midpoint of the line connecting the two pupils of the human eye as the vertex. Compared with the actual images seen by the left and right eyes, there are still deviations.

[0161] Therefore, in order to further enhance the user's realistic experience, the screen of the virtual display device can be divided into a left screen and a right screen, which are used to display the image under the left eye field of view parameter and the image under the right eye field of view parameter, respectively.

[0162] The left eye field of view parameter refers to the angle formed by the two edges of the maximum range through which the image of the target object can be seen from the left eye, with the center of the left pupil as the vertex. The right eye field of view parameter refers to the angle formed by the two edges of the maximum range through which the image of the target object can be seen from the right eye, with the center of the right pupil as the vertex.

[0163] The methods for determining the left and right eye field of view are similar to those for determining the human eye's field of view. For example, the method for determining the left eye field of view is as follows:

[0164] Get the width of the left screen and the distance from the left eye to the left screen.

[0165] The left eye field of view parameters are determined based on the width of the left screen and the distance from the left eye to the left screen.

[0166] In step S32, based on the first image of the real object captured by the virtual display device under the camera field of view parameters, the second image of the real object corresponding to the left eye under the left eye field of view parameters is determined, and the second image of the real object corresponding to the right eye under the right eye field of view parameters is determined.

[0167] The methods for determining the second image for the left and right eyes are similar to those for determining the second image under the human eye's field of view parameters. For example, the method for determining the second image for the left eye is as follows:

[0168] Using the left eye field of view parameter, determine the first position of each pixel on the left screen of the virtual display device, and the corresponding second position on the real object in the world coordinate system;

[0169] The second position is determined to correspond to the imaging position on the first image, and the third position is obtained.

[0170] Obtain the pixel value at the third position from the first image, and use the pixel value as the target pixel value at the first position;

[0171] The image with the pixel value at the first position as the target pixel value is determined as the second left-eye image corresponding to the real object under the left-eye field of view parameter.

[0172] In step S33, the preset virtual object is rendered according to the left eye field angle parameter to obtain the left eye third image, and the right eye field angle parameter is used as the rendering parameter to render the virtual object to obtain the right eye third image.

[0173] The rendering process can be referred to in step S13, and will not be repeated here.

[0174] In step S34, the left-eye target image obtained by superimposing the left-eye third image and the left-eye second image, and the right-eye target image obtained by superimposing the right-eye third image and the right-eye second image, are displayed on the left screen and the right screen, respectively.

[0175] Specifically, the target image for the left eye is displayed on the left screen, and the target image for the right eye is displayed on the right screen.

[0176] In steps S31-S34, the left-eye field of view parameters for the left eye relative to the left screen of the virtual display device are determined, and the right-eye field of view parameters for the right eye relative to the right screen of the virtual display device are determined. The left-eye target image under the left-eye field of view parameters and the right-eye field of view image under the right-eye field of view parameters are calculated and displayed on the left and right screens of the virtual display device, respectively. This eliminates display errors caused by human eye field of view parameters, ensuring that the image seen by the user's left eye conforms to left-eye observation habits, and the image seen by the user's right eye conforms to right-eye observation habits, further enhancing the user's realistic experience when using the virtual display device.

[0177] Figure 6 This is a structural block diagram of an image display device according to an exemplary embodiment. For example... Figure 6 As shown, the image display device 40 includes:

[0178] The human eye field of view parameter determination module 41 is configured to determine the human eye field of view parameters relative to the screen of the virtual display device.

[0179] The second image determination module 42 is configured to execute a first image of the real object obtained by the virtual display device for the real object under the camera field of view parameters, and determine a second image of the real object under the human eye field of view parameters.

[0180] The rendering module 43 is configured to render a preset virtual object according to the human eye field angle parameters to obtain a third image;

[0181] Display module 44 is configured to display a target image obtained by superimposing the third image and the second image on the screen of the virtual display device.

[0182] In one possible implementation, the human eye field angle parameter determination module is specifically configured to perform:

[0183] Obtain the screen width of the virtual display device, and obtain the distance from the human eye to the screen;

[0184] The field of view parameters of the human eye are determined based on the screen width and the distance from the human eye to the screen.

[0185] In one possible implementation, the human eye field angle parameter determination module is further configured to perform:

[0186] The user's facial image is acquired using the camera of the virtual display device;

[0187] Determine the distance between the pupils of the two eyes in the facial image to obtain the interpupillary distance;

[0188] The distance from the human eye to the screen is obtained by determining the quotient of the pupil distance between the two eyes and the pre-acquired standard pupil distance; the standard pupil distance is the distance between the pupils of the two eyes in a facial image taken at a distance of 1 meter from the screen.

[0189] In one possible implementation, the second image determination module is specifically configured to perform:

[0190] Using the human eye field of view parameters, the first position of each pixel in the first image is transformed in the screen coordinate system of the virtual display device to obtain the second position of each pixel in the world coordinate system.

[0191] The imaging position on the first image corresponding to the second position is determined to obtain the third position;

[0192] Obtain the pixel value at the third position from the first image, and use the pixel value as the target pixel value at the first position;

[0193] The image with the pixel value at the first position being the target pixel value is determined as the second image of the real object under the human eye field of view parameters.

[0194] In one possible implementation, the second image determination module is further configured to perform:

[0195] Obtain the distance between the real object and the camera to get the object distance;

[0196] The product of the object distance and the human eye field angle parameter is determined to obtain the first product;

[0197] The quotient of the product of the first position of each pixel in the first image in the screen coordinate system of the virtual display device and the first position is determined to obtain the second position of each pixel in the world coordinate system.

[0198] Optionally, the second image determination module is further configured to perform:

[0199] The distance between the real object and the camera is obtained using a stereo color camera or a neural network algorithm.

[0200] In one possible implementation, the device further includes:

[0201] The left and right eye field of view determination module is configured to determine the left eye field of view parameters of the left eye relative to the left screen of the virtual display device, and to determine the right eye field of view parameters of the right eye relative to the right screen of the virtual display device.

[0202] The left and right eye second image determination module is configured to perform the following: based on the first image captured by the virtual display device for the real object under the camera field of view parameters, determine that the real object corresponds to the left eye second image under the left eye field of view parameters, and determine that the real object corresponds to the right eye second image under the right eye field of view parameters.

[0203] The left and right eye image rendering modules are configured to render a preset virtual object according to the left eye field of view parameter to obtain a left eye third image, and to render the virtual object using the right eye field of view parameter as a rendering parameter to obtain a right eye third image.

[0204] The left and right eye display modules are configured to display a left-eye target image obtained by superimposing the left-eye third image and the left-eye second image, and a right-eye target image obtained by superimposing the right-eye third image and the right-eye second image, respectively, on the left screen and the right screen.

[0205] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0206] Figure 7 This is a block diagram illustrating an electronic device for image display according to an exemplary embodiment. Its internal structure diagram can be as follows: Figure 7 As shown, the server or electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the server or electronic device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an image display method.

[0207] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the server or electronic device to which the present disclosure is applied. A specific server or electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0208] In an exemplary embodiment, a server or electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the image display method as described in the embodiments of this disclosure.

[0209] In an exemplary embodiment, a computer-readable storage medium is also provided, which, when executed by a processor of a server or electronic device, enables the server or electronic device to perform the image display method of the present disclosure. The computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0210] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the image display method of the present disclosure embodiments.

[0211] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0212] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0213] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image display method, characterized in that, The method includes: Determine the human eye's field of view parameters relative to the screen of the virtual display device; the human eye's field of view parameters are determined based on the screen width and the distance from the human eye to the screen; Based on the first image of the real object obtained by the virtual display device under the camera field of view parameters, a second image of the real object under the human eye field of view parameters is determined. Based on the human eye field angle parameters, a preset virtual object is rendered to obtain a third image; The target image, obtained by superimposing the third image and the second image, is displayed on the screen of the virtual display device.

2. The method according to claim 1, characterized in that, Obtaining the distance from the human eye to the screen includes: The user's facial image is acquired using the camera of the virtual display device; Determine the distance between the pupils of the two eyes in the facial image to obtain the interpupillary distance; The distance from the human eye to the screen is obtained by determining the quotient of the pupil distance between the two eyes and the pre-acquired standard pupil distance; the standard pupil distance is the distance between the pupils of the two eyes in a facial image taken at a distance of 1 meter from the screen.

3. The method according to claim 1, characterized in that, The step of determining a second image of the real object under the human eye's field of view parameters based on a first image obtained by the virtual display device for the real object under the camera's field of view parameters includes: Using the human eye field of view parameters, the first position of each pixel in the first image is transformed in the screen coordinate system of the virtual display device to obtain the second position of each pixel in the world coordinate system. The imaging position on the first image corresponding to the second position is determined to obtain the third position; Obtain the pixel value at the third position from the first image, and use the pixel value as the target pixel value at the first position; The image with the pixel value at the first position being the target pixel value is determined as the second image of the real object under the human eye field of view parameters.

4. The method according to claim 3, characterized in that, The step of transforming the first position of each pixel in the first image in the screen coordinate system of the virtual display device to obtain the second position of each pixel in the world coordinate system includes: Obtain the distance between the real object and the camera to get the object distance; The product of the object distance and the human eye field angle parameter is determined to obtain the first product; The quotient of the product of the first position of each pixel in the first image in the screen coordinate system of the virtual display device and the first position is determined to obtain the second position of each pixel in the world coordinate system.

5. The method according to claim 4, characterized in that, The step of obtaining the distance between the real object and the camera includes: The distance between the real object and the camera is obtained using a stereo color camera or a neural network algorithm.

6. The method according to claim 1, characterized in that, The human eye includes the left and right eyes, the screen of the virtual display device includes a left screen and a right screen, and the method further includes: Determine the left eye field of view parameters of the left eye relative to the left screen of the virtual display device, and determine the right eye field of view parameters of the right eye relative to the right screen of the virtual display device; Based on the first image of the real object captured by the virtual display device under the camera field of view parameters, it is determined that the real object corresponds to the second image of the left eye under the left eye field of view parameters, and it is determined that the real object corresponds to the second image of the right eye under the right eye field of view parameters. Based on the left eye field of view parameters, a preset virtual object is rendered to obtain a third image for the left eye; and based on the right eye field of view parameters, the virtual object is rendered to obtain a third image for the right eye. The left-eye target image, obtained by superimposing the left-eye third image and the left-eye second image, and the right-eye target image, obtained by superimposing the right-eye third image and the right-eye second image, are displayed on the left screen and the right screen, respectively.

7. An image display device, characterized in that, The device includes: The human eye field of view parameter determination module is configured to determine the human eye field of view parameters relative to the screen of the virtual display device; the human eye field of view parameters are determined based on the screen width and the distance from the human eye to the screen; The second image determination module is configured to execute a first image of the real object obtained by the virtual display device for the real object under the camera field of view parameters, and determine a second image of the real object under the human eye field of view parameters. The rendering module is configured to render a preset virtual object based on the human eye field angle parameters to obtain a third image; The display module is configured to display a target image obtained by superimposing the third image and the second image on the screen of the virtual display device.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the image display method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the server, the server is able to perform the image display method as described in any one of claims 1 to 6.

Images