Image display method and device

Abstract

The invention discloses an image display method and device, and the method comprises the steps: obtaining a first image feature of a first image and a second image feature of a second image in a target image pair, and enabling the first image and the second image to be obtained through the photographing of two photographing devices; fusing the first image feature and the second image feature to obtain a first fused image feature and a second fused image feature; determining depth information according to the first fusion image feature and the second fusion image feature; processing the depth information, images contained in the target image pair, Gaussian parameters of the target image pair and equipment parameters of the shooting equipment to obtain Gaussian point cloud data; generating a viewpoint image according to the Gaussian point cloud data; and displaying the viewpoint image through the naked eye holographic display screen.

Description

Technical Field

[0001] The present application relates to the field of image display technology, and in particular to an image display method and device. Background Art

[0002] The naked-eye holographic display is a new type of display that allows users to watch three-dimensional content from multiple viewpoints with their naked eyes without the help of other auxiliary devices. In order to meet the display requirements, it is necessary to obtain the viewpoint images for display on the naked-eye holographic display based on the plane image and the depth information of the plane image.

[0003] In the related art, a camera module with a depth sensor is usually used to obtain a planar image and its depth information. However, in daily use scenarios, most camera modules generally do not have a depth sensor. Therefore, how to use a shooting device without a depth sensor to obtain a viewpoint image for display on a naked-eye holographic display screen has become an urgent problem to be solved. Summary of the invention

[0004] To this end, this application discloses the following technical solutions:

[0005] The first aspect of the present application provides an image display method, comprising:

[0006] Obtaining a first image feature of a first image and a second image feature of a second image in a target image pair, wherein the first image and the second image are respectively captured by two capturing devices;

[0007] fusing the first image feature and the second image feature to obtain a first fused image feature and a second fused image feature;

[0008] Determine depth information according to the first fused image feature and the second fused image feature;

[0009] Processing the depth information, the images contained in the target image pair, Gaussian parameters of the target image pair, and device parameters of the shooting device to obtain Gaussian point cloud data;

[0010] Generate a viewpoint image according to the Gaussian point cloud data;

[0011] The viewpoint image is displayed through a naked-eye holographic display screen.

[0012] Optionally, the first image feature and the second image feature both include a plurality of row features;

[0013] The fusing the first image feature and the second image feature to obtain a first fused image feature and a second fused image feature comprises:

[0014] The corresponding row features in the first image feature and the second image feature are fused to obtain a first fused image feature and a second fused image feature.

[0015] Optionally, determining the depth information according to the first fused image feature and the second fused image feature includes:

[0016] Determine disparity information of the target image pair according to the first fused image feature and the second fused image feature;

[0017] Depth information is determined based on the disparity information.

[0018] Optionally, also include:

[0019] The first image and the second image are subjected to stereoscopic correction to obtain a corrected first image and a corrected second image.

[0020] Optionally, the target image pair corresponds to multiple viewpoints;

[0021] The generating a viewpoint image according to the Gaussian point cloud data comprises:

[0022] A viewpoint image corresponding to each viewpoint is generated according to the Gaussian point cloud data.

[0023] Optionally, displaying the viewpoint image through a naked-eye holographic display screen includes:

[0024] Combining multiple frames of viewpoint images to obtain a frame of holographic image with a target resolution;

[0025] The holographic image is displayed through a naked-eye holographic display screen.

[0026] Optionally, there are multiple target image pairs, and the multiple target image pairs are obtained by photographing multiple horizontally arranged shooting devices, wherein images photographed by every two adjacent shooting devices constitute one target image pair.

[0027] Optionally, the Gaussian parameters of the target image pair include:

[0028] A full-size Gaussian point rotation parameter map, a full-size Gaussian point size parameter map, a full-size Gaussian depth residual map, a full-size Gaussian point transparency parameter map and a full-size Gaussian point color parameter map are obtained by processing the contained image according to the depth information and the target image.

[0029] Optionally, the processing of the depth information, the images included in the target image pair, Gaussian parameters of the target image pair, and device parameters of the shooting device to obtain Gaussian point cloud data includes:

[0030] According to the device parameters of the shooting device, the depth information, the images contained in the target image pair and the Gaussian parameters of the target image pair are back-projected to obtain Gaussian point cloud data.

[0031] A second aspect of the present application provides an image display device, including a memory, a processor, and a naked-eye holographic display screen;

[0032] The processor is used to:

[0033] Obtaining a first image feature of a first image and a second image feature of a second image in a target image pair, wherein the first image and the second image are respectively captured by two capturing devices;

[0034] fusing the first image feature and the second image feature to obtain a first fused image feature and a second fused image feature;

[0035] Determine depth information according to the first fused image feature and the second fused image feature;

[0036] Processing the depth information, the images contained in the target image pair, Gaussian parameters of the target image pair, and device parameters of the shooting device to obtain Gaussian point cloud data;

[0037] Generate a viewpoint image according to the Gaussian point cloud data;

[0038] The viewpoint image is displayed through the naked-eye holographic display screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0040] Figure 1 is a flow chart of an image display method provided by an embodiment of the present application;

[0041] Figure 2 is a schematic diagram of the position of a photographing device provided in an embodiment of the present application;

[0042] Figure 3 is a schematic diagram of a stereo correction provided by an embodiment of the present application;

[0043] Figure 4 is a schematic diagram of fusing a first image feature and a second image feature provided by an embodiment of the present application;

[0044] Figure 5It is a structural schematic diagram of an image display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] The present application embodiment provides an image display method, see Figure 1 , is a flow chart of the method, and the method may include the following steps.

[0047] S101, obtaining a first image feature of a first image and a second image feature of a second image in a target image pair, wherein the first image and the second image are respectively captured by two capturing devices.

[0048] In step S101, two shooting devices may be used to shoot the same target object at the same time, and the obtained two frames of images are used as a target image pair.

[0049] Two cameras can be fixed at the same level.

[0050] by Figure 2 For example, you can use Figure 2 The shooting device 1 and the shooting device 4 located at the same horizontal height shoot the target object at the same time to obtain a target image pair, the image shot by the shooting device 1 is used as the first image, and the image shot by the shooting device 4 is used as the second image.

[0051] After obtaining the target image pair, the first image can be processed with a pre-built encoding module for extracting image features to obtain the corresponding first image features, and the second image can be processed with the encoding module to obtain the second image features. The encoding module may include a plurality of residual blocks and downsampling layers. The structure and working principle of the encoding module can be referred to in the related art and will not be described in detail.

[0052] S102: Fusing the first image feature and the second image feature to obtain a first fused image feature and a second fused image feature.

[0053] In step S102, the correlation between the first image feature and the second image feature may be analyzed, and the analysis result may be added to the first image feature to obtain a first fused image feature, and the analysis result may be added to the second image feature to obtain a second fused image feature.

[0054] As an example, the first image feature f1 and the second image feature f2 may be fused as follows to obtain a first fused image feature f1 ′ and a second fused image feature f2 ′.

[0055] f1'=f1+Att(Q1, K2, V2), f2'=f2+Att(Q2, K1, V1).

[0056] Among them, Att represents a pre-built extraction model for extracting the correlation between image features. This module can be a neural network model built based on the attention mechanism. Its working principle and structure can be referred to in the relevant technology.

[0057] Q1, K1 and V1 represent the query weight W in turn. Q , key weight W K Sum weight W V The processing result obtained after processing the first image feature, that is, Q1 = f1*W Q , K1=f1*W K , V1=f1*W V Q2, K2 and V2 respectively represent the processing results obtained after processing the second image feature with the above weights.

[0058] S103: Determine depth information according to the first fused image feature and the second fused image feature.

[0059] The depth information determined in this step may include first depth information corresponding to the first image, may include second depth information corresponding to the second image, or may include first depth information and second depth information.

[0060] The depth information of an image reflects the distance between the object being photographed and the photographing device when the image is photographed. In this embodiment, the depth information of any image can be represented by a depth map having the same size as the image.

[0061] Taking the first image as an example, the first depth information may be a first depth map having the same size as the first image, the pixels of the first depth map correspond one-to-one to the pixels at the same position in the first image, and the grayscale value of each pixel in the first depth map represents the depth value of the corresponding pixel in the first image, and the depth value of the pixel may represent the distance from the surface of the object and the point corresponding to the pixel to the camera when the first image is taken.

[0062] S104, processing the depth information, the images contained in the target image pair, the Gaussian parameters of the target image pair, and the device parameters of the shooting device to obtain Gaussian point cloud data.

[0063] The Gaussian parameters of the target image pair can be obtained according to the depth information and the image processing contained in the target image pair.

[0064] The Gaussian parameters of the target image pair may include first Gaussian parameters corresponding to the first image, may include second Gaussian parameters corresponding to the second image, or may include first Gaussian parameters and second Gaussian parameters.

[0065] The first Gaussian parameters may be obtained by encoding and decoding the first depth information and the first image using a pre-constructed convolutional neural network to obtain the first Gaussian parameters, and the second Gaussian parameters may be obtained by encoding and decoding the second depth information and the second image using a pre-constructed convolutional neural network to obtain the second Gaussian parameters.

[0066] Optionally, the Gaussian parameters corresponding to the image may be represented by a Gaussian parameter map, in which each pixel corresponds to one or more pixels in the image, and the grayscale value of each pixel represents the Gaussian parameter value of the corresponding pixel in the image.

[0067] Optionally, in order to improve the image quality of the obtained viewpoint image, the Gaussian parameter map obtained in step S104 can be a full-size Gaussian parameter map, the size of the full-size Gaussian parameter map is consistent with the size of the corresponding image, the pixels of the full-size Gaussian parameter map correspond one-to-one to the pixels at the same position of the corresponding image, and the full-size Gaussian parameter map includes both the Gaussian parameter values ​​of the target object in the corresponding image and the Gaussian parameter values ​​of the image background.

[0068] That is to say, the above-mentioned first Gaussian parameters may include a first full-size Gaussian point rotation parameter map r1, a first full-size Gaussian point size parameter map s1, a first full-size Gaussian depth residual map t1, a first full-size Gaussian point transparency parameter map o1 and a first full-size Gaussian point color parameter map c1 corresponding to the first image; the second Gaussian parameters may include a second full-size Gaussian point rotation parameter map r2, a second full-size Gaussian point size parameter map s2, a second full-size Gaussian depth residual map t2, a second full-size Gaussian point transparency parameter map o2 and a second full-size Gaussian point color parameter map c2 corresponding to the second image.

[0069] The grayscale value of each pixel in r1 can represent the Gaussian point rotation parameter value of the corresponding pixel in the first image, the grayscale value of each pixel in s1 can represent the Gaussian point size parameter value of the corresponding pixel in the first image, the grayscale value of each pixel in t1 can represent the Gaussian depth residual parameter value of the corresponding pixel in the first image, the grayscale value of each pixel in o1 can represent the Gaussian point transparency parameter value of the corresponding pixel in the first image, and the grayscale value of each pixel in c1 can represent the Gaussian point color parameter value of the corresponding pixel in the first image.

[0070] The Gaussian parameter map can be obtained as follows: r1, s1, o1, t1, c1 = F(I1, D1). I1 represents the first image, and D1 represents the first depth map.

[0071] Gaussian point cloud data may include: first Gaussian point cloud data obtained by processing first depth information, a first image, first Gaussian parameters and device parameters of a shooting device; second Gaussian point cloud data obtained by processing second depth information, a second image, second Gaussian parameters and device parameters of a shooting device; or first Gaussian point cloud data and second Gaussian point cloud data.

[0072] S105, generating a viewpoint image according to the Gaussian point cloud data.

[0073] In step S105, for any viewpoint of the naked-eye holographic display screen, the viewpoint can be rendered based on the Gaussian point cloud data obtained in S104 to obtain a viewpoint image corresponding to the viewpoint. When rendering, the rendering can be performed based on the first Gaussian point cloud data, the rendering can be performed based on the second Gaussian point cloud data, or the rendering can be performed based on the first Gaussian point cloud data and the second Gaussian point cloud data. The method of rendering a specific viewpoint based on Gaussian point cloud data can be referred to the relevant technology and will not be described in detail.

[0074] S106, displaying the viewpoint image through the naked-eye holographic display screen.

[0075] In step S106, the obtained viewpoint image may be provided to a naked-eye holographic display screen, so that the viewpoint image is displayed on the naked-eye holographic display screen.

[0076] The beneficial effects of this embodiment are:

[0077] On the one hand, this embodiment can obtain the depth information of the image according to the image features of the two frames of the image, and then obtain the viewpoint image for display on the naked-eye holographic display screen according to the depth information and the image, so as to obtain the image required by the naked-eye holographic display screen without relying on the shooting equipment with a depth sensor, so that the naked-eye holographic display screen can have a wider range of applications;

[0078] On the other hand, this embodiment fuses the image features of the two frames when obtaining depth information, and determines the depth information based on the obtained fused image features. This allows reference to the correlation between the two frames when obtaining the depth information, which is conducive to obtaining more accurate depth information.

[0079] In some optional embodiments, after obtaining the target image pair and before extracting the first image feature and the second image feature, the following steps may be further performed:

[0080] The first image and the second image are subjected to stereoscopic correction to obtain a corrected first image and a corrected second image.

[0081] In the corrected first image and the corrected second image obtained after stereo correction, every two mutually matching pixels may be located on the same horizontal line. Mutually matching pixels are defined as, for a pixel in the first image and a pixel in the second image, if both are pixel points obtained by projecting the same position of the surface of the object on the two frames of images, then the two pixel points match each other.

[0082] by Figure 3 For example, the pixel point P1 of the first image and the pixel point P2 of the second image are both projections of the point P0 of the photographed object, and P1 and P2 match each other. Figure 3 As can be seen from (1), before correction, P1 of the first image and P2 of the second image are not on the same horizontal line, and there is a certain parallax between the first image and the second image in the vertical direction. After stereo correction, Figure 3 As shown in (2), P1 of the corrected first image and P2 of the corrected second image are located on the same horizontal line, and there is only horizontal parallax between the corrected first image and the corrected second image.

[0083] The method of stereo correction can be found in the relevant technology and will not be described in detail.

[0084] In the case of performing stereoscopic correction, the first image and the second image used in each step of the display method of this embodiment may be the above-mentioned corrected first image and corrected second image.

[0085] In some embodiments, see Figure 4 , the first image feature and the second image feature may each include a plurality of row features.

[0086] Accordingly, the method of fusing the first image feature and the second image feature may be:

[0087] The corresponding row features in the first image feature and the second image feature are fused to obtain a first fused image feature and a second fused image feature.

[0088] That is to say, when fusing the first image feature and the second image feature, for each row feature of the first image feature, another row feature corresponding to it can be determined in the second image feature, and then the two are fused to obtain the row features of the first fused image feature and the second fused image feature, and so on, until each row feature constituting the first fused image feature and the second fused image feature is obtained.

[0089] by Figure 4 For example, when merging, Figure 4The second row feature of the first image feature shown in the shaded part (in order from top to bottom), determines the corresponding row feature in the second image feature, assumes it to be the second row feature of the second image feature, and then fuses the second row feature of the first image feature and the second row feature of the second image feature to obtain the second row feature of the first fused image feature and the second row feature of the second fused image feature.

[0090] The fusion process can be expressed as f1' i =f1 i +Att(Q 1i , K 2j , V 2j ), f2' j =f2 j +Att(Q 2j , K 1i , V 1i ), where f1' i represents the i-th row feature of the first fused image feature, f1 i represents the i-th row feature of the first image feature, f2' j Represents the j-th row feature of the second fused image feature, f2 j represents the j-th row feature of the second image feature, Q 1i , K 1i and V 1i In turn, it means using the preset query weight W Q , key weight W K Sum weight W V The processing result obtained after processing the i-th row feature of the first image feature, Q 2j , K 2j and V 2j Similarly, the i-th row feature of the first image feature and the j-th row feature of the second image feature correspond to each other.

[0091] In this embodiment, for two row features corresponding to the first image and the second image respectively, if the pixel points corresponding to the two row features match each other, it can be considered that the two row features correspond to each other. For example, the a-th row of pixels in the first image and the b-th row of pixels in the second image match each other, then the row features obtained based on the a-th row of pixels and the row features obtained based on the b-th row of pixels correspond to each other.

[0092] In the above fusion method, if the first image and the second image are stereoscopically corrected in advance, it can be considered that the pixels on the same row in the corrected first image and the corrected second image match each other. Therefore, it can be determined that in the first image feature and the second image feature, the two row features of the same row correspond to each other, that is, the first row feature of the first image feature and the first row feature of the second image feature correspond to each other, the second row feature of the first image feature and the second row feature of the second image feature correspond to each other, and so on.

[0093] In the above fusion method, if the first image and the second image are not stereoscopically corrected in advance, the correspondence between the row features of the first image features and the row features of the second image features can also be determined by using a pre-constructed matching model or based on other matching methods. The specific method can be found in the relevant technology and will not be described in detail.

[0094] In some embodiments, the depth information may be determined by first determining the disparity information based on the fused image features, and then determining the depth information based on the disparity information. That is, step S103 may include:

[0095] Determine disparity information of the target image pair according to the first fused image feature and the second fused image feature;

[0096] Depth information is determined based on the disparity information.

[0097] The implementation process is described below by taking the example that the depth information includes the first depth information and the second depth information. In this embodiment, the first fused image feature and the second fused image feature can be input into a pre-constructed neural network (such as a recurrent neural network module), and the matching relationship between the first fused image feature and the second fused image feature can be calculated by the neural network, and then the first disparity map and the second disparity map can be estimated based on the matching relationship, wherein the first disparity map represents the disparity of each pixel of the second image relative to each pixel of the first image with the first image as a reference, and the second disparity map represents the disparity of each pixel of the first image relative to each pixel of the second image with the second image as a reference, and the matching relationship on the left and right feature maps is calculated.

[0098] After obtaining the first disparity map d1, the first disparity map can be processed according to the baseline length B between the first shooting device and the second shooting device, the focal length F1 of the first shooting device, and the corrected offset T of the principal points of the two shooting devices to obtain the corresponding first depth map D1. The processing process can be expressed as D1=BF1 / (d1+T); after obtaining the second disparity map d2, the second disparity map can be processed according to the baseline length B between the first shooting device and the second shooting device, the focal length F2 of the second shooting device, and the corrected offset T of the principal points of the two shooting devices to obtain the corresponding second depth map D2. The processing process can be expressed as D2=BF2 / (d2+T). Among them, the first shooting device refers to the shooting device that shoots the first image, and the second shooting device refers to the shooting device that shoots the second image. The method for obtaining the corrected offset of the principal points of the two shooting devices can refer to the relevant technology.

[0099] Optionally, if only the first depth information or the second depth information needs to be obtained, only the corresponding steps in the above embodiments may be performed, and no further details are given.

[0100] Optionally, the method of obtaining Gaussian point cloud data may be:

[0101] According to the device parameters of the shooting device, the depth information, the images contained in the target image pair and the Gaussian parameters of the target image pair are back-projected to obtain Gaussian point cloud data.

[0102] The device parameters of the photographing device may be calibrated device parameters.

[0103] If the first Gaussian point cloud data needs to be obtained, the first depth information D1, the first image I1 and the first Gaussian parameter can be back-projected according to the corrected device parameter π1 of the first shooting device to obtain the corresponding first Gaussian point cloud data G1, that is, G1 = π1 -1 (I1, D1+t1, r1, s1, o1, c1); if the second Gaussian point cloud data needs to be obtained, the second depth information D2, the second image I2 and the second Gaussian parameter can be back-projected according to the device parameter π2 of the calibrated second shooting device to obtain the corresponding second Gaussian point cloud data G2. The back-projection processing method can refer to the relevant technology and will not be described in detail.

[0104] In some optional embodiments, a target image pair may correspond to multiple viewpoints of a naked-eye holographic display screen. In this case, generating viewpoint images according to Gaussian point cloud data may include:

[0105] According to the Gaussian point cloud data, a viewpoint image corresponding to each viewpoint is generated.

[0106] That is to say, after obtaining the Gaussian point cloud data of the target image pair, each viewpoint corresponding to the target image pair can be rendered according to the Gaussian point cloud data to obtain the viewpoint image corresponding to the viewpoint, thereby obtaining multi-frame viewpoint images of multiple viewpoints corresponding to the target image pair.

[0107] Exemplarily, a target image pair can correspond to 16 viewpoints of the naked eye permission display screen, namely viewpoint 1 to viewpoint 16. At this time, each of the 16 viewpoints can be rendered according to the Gaussian point cloud data of the target image pair to obtain 16 frames of viewpoint images corresponding to the 16 viewpoints, that is, viewpoint image 1 corresponding to viewpoint 1, viewpoint image 2 corresponding to viewpoint 2, and so on.

[0108] Optionally, the corresponding viewpoints can be determined according to the distances from each viewpoint of the naked-eye holographic display screen to a shooting device used to shoot the target image pair. For example, 16 viewpoints can be selected from each viewpoint of the naked-eye holographic display screen in order of distance from the shooting device from near to far as the viewpoints corresponding to the target image pair.

[0109] Optionally, the number of target image pairs may be multiple, and the multiple target image pairs are obtained by photographing multiple shooting devices arranged horizontally, wherein images photographed by every two adjacent shooting devices constitute a target image pair. The multiple target image pairs may be composed of multiple frames of images obtained by photographing the same target object by multiple shooting devices at the same time.

[0110] For example, there may be three target image pairs, which are captured by four capturing devices respectively. Figure 2 The shooting devices 1, 4, 7 and 10 shown in the figure capture 4 frames of images, wherein the images captured by the shooting devices 1 and 4 constitute a target image pair, the images captured by the shooting devices 4 and 7 constitute a target image pair, and the images captured by the shooting devices 7 and 10 constitute a target image pair.

[0111] In the case where there are multiple target image pairs, each target image pair may correspond to a different viewpoint of the naked-eye holographic display screen. For example, the naked-eye holographic display screen may have 50 viewpoints, and among the above three target image pairs, the first two target image pairs may correspond to 17 of the viewpoints respectively, and the last target image pair may correspond to the remaining 16 viewpoints.

[0112] When there are multiple target image pairs, you can press Figure 1 The method shown processes each target image pair, thereby obtaining viewpoint images of a plurality of viewpoints corresponding to each target image pair based on Gaussian point cloud data of each target image pair.

[0113] Optionally, when multiple frames of viewpoint images are obtained, the method of displaying the viewpoint images through the naked-eye holographic display screen may be:

[0114] Combining multiple frames of viewpoint images to obtain a frame of holographic image with target resolution;

[0115] The holographic image is displayed through a naked-eye holographic display.

[0116] For example, assuming that three target image pairs are processed separately, a total of 50 frames of viewpoint images corresponding to 50 viewpoints of the naked-eye holographic display screen are obtained, wherein the resolution of each frame of the viewpoint image can be 810*1440, and these 50 frames of viewpoint images can be combined in the form of 10 rows and 5 columns to obtain a frame of holographic image with a resolution of 8100*7200, and then the frame of holographic image is provided to the naked-eye holographic display screen for display.

[0117] Optionally, the processing model used in the above embodiments may be trained in the following manner, and the processing model may include the encoder, convolutional neural network, recurrent neural network, etc. of the above embodiments.

[0118] Obtain multiple reference images corresponding to the target image pair.

[0119] The images contained in the reference image and the target image pair can be obtained by photographing the same target object at the same time.

[0120] by Figure 2 For example, photographing devices 1 to 10 photograph the target object at the same time, wherein the images photographed by photographing devices 1, 4, 7 and 10 are used as three target image pairs, and the images photographed by photographing devices 2, 3, 5, 6, 8 and 9 are used as reference images.

[0121] The viewpoint corresponding to the reference image is rendered based on the Gaussian point cloud data to obtain a viewpoint image.

[0122] The viewpoint corresponding to the reference image can be understood as the location of the camera that captured the reference image, for example Figure 2 The position of the shooting device 2 can be used as the viewpoint corresponding to the reference image shot by the shooting device 2.

[0123] The reference image and the viewpoint image are compared, the model loss of the processing model is determined according to the difference between the two, and the processing model is trained according to the model loss.

[0124] During the comparison, the reference image can be compared with the viewpoint image of its corresponding viewpoint to obtain the difference between the two. If there are multiple reference images, the difference between each reference image and the viewpoint image can be fused by taking the average or summing to obtain the model loss of the processing model. If there is only one reference image, the difference between the reference image and the viewpoint image of the corresponding viewpoint can be used as the model loss.

[0125] For the method of processing the model according to the model loss training, please refer to the relevant technology and will not go into details.

[0126] Optionally, you can further introduce a geometric consistency regularization term into the model loss in the following way:

[0127] For each target image pair, the first depth information of the first image and the second depth information of the second image in the target image pair are obtained, the difference between the first depth information and the second depth information is compared, and the geometric consistency regularization term of the target image pair is determined based on the size of the difference. The larger the difference, the larger the geometric consistency regularization term. Then, the geometric consistency regularization terms of several target image pairs and the difference between the aforementioned reference image and the viewpoint image are fused to obtain the model loss. In other words, the model loss can be determined based on the geometric consistency regularization terms of each target image pair and the difference between the reference image and the viewpoint image.

[0128] The benefit of introducing a geometric consistency regularization term into the model loss is that it can improve the three-dimensional consistency of each target image pair, thereby improving the accuracy of the rendered viewpoint image.

[0129] In order to ensure the three-dimensional consistency of the left and right viewpoints of each camera group, a left and right viewpoint geometric consistency regularization term is introduced to further ensure the accuracy of the rendering results.

[0130] Optionally, during the training phase, in order to improve the generalization ability of the processing model, target image pairs and reference images can be captured successively at multiple different moments (for example, each shooting device captures 300 frames of images in each scene) in multiple different scenes (for example, 3 scenes), and the above-mentioned processing model can be trained based on these target image pairs and reference images.

[0131] The present application also provides an image display device. Figure 5 , the device may include a memory 501, a processor 502 and a naked-eye holographic display screen 503.

[0132] The memory 501 may store a computer program, and the processor 502 may execute the computer program to perform the following steps:

[0133] Obtaining a first image feature of a first image and a second image feature of a second image in a target image pair, wherein the first image and the second image are respectively captured by two capturing devices;

[0134] Fusing the first image feature and the second image feature to obtain a first fused image feature and a second fused image feature;

[0135] Determining depth information according to the first fused image feature and the second fused image feature;

[0136] Processing the depth information, the images contained in the target image pair, the Gaussian parameters of the target image pair, and the device parameters of the shooting device to obtain Gaussian point cloud data;

[0137] Generate a viewpoint image based on Gaussian point cloud data;

[0138] The viewpoint images are displayed through the naked-eye holographic display screen 503 .

[0139] The working principle of the image display device of this embodiment can refer to the relevant steps of the image display method of the above-mentioned embodiment, which will not be described in detail.

[0140] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0141] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0142] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0143] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0144] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image display method, comprising: Obtaining a first image feature of a first image and a second image feature of a second image in a target image pair, wherein the first image and the second image are respectively captured by two capturing devices; fusing the first image feature and the second image feature to obtain a first fused image feature and a second fused image feature; Determine depth information according to the first fused image feature and the second fused image feature; Processing the depth information, the images contained in the target image pair, Gaussian parameters of the target image pair, and device parameters of the shooting device to obtain Gaussian point cloud data; Generate a viewpoint image according to the Gaussian point cloud data; The viewpoint image is displayed through a naked-eye holographic display screen.

2. The method according to claim 1, wherein the first image feature and the second image feature each comprise a plurality of row features; The fusing the first image feature and the second image feature to obtain a first fused image feature and a second fused image feature comprises: The corresponding row features in the first image feature and the second image feature are fused to obtain a first fused image feature and a second fused image feature.

3. The method according to claim 1, wherein determining the depth information according to the first fused image feature and the second fused image feature comprises: Determine disparity information of the target image pair according to the first fused image feature and the second fused image feature; Depth information is determined based on the disparity information.

4. The method according to claim 1, further comprising: The first image and the second image are subjected to stereoscopic correction to obtain a corrected first image and a corrected second image.

5. The method according to claim 1, wherein the target image pair corresponds to a plurality of viewpoints; The generating a viewpoint image according to the Gaussian point cloud data comprises: A viewpoint image corresponding to each viewpoint is generated according to the Gaussian point cloud data.

6. The method according to claim 1, wherein displaying the viewpoint image through a naked-eye holographic display screen comprises: Combining multiple frames of viewpoint images to obtain a frame of holographic image with a target resolution; The holographic image is displayed through a naked-eye holographic display screen.

7. According to the method of claim 1, there are multiple target image pairs, and the multiple target image pairs are captured by multiple horizontally arranged shooting devices, wherein images captured by every two adjacent shooting devices constitute one target image pair.

8. The method according to claim 1, wherein the Gaussian parameters of the target image pair include: A full-size Gaussian point rotation parameter map, a full-size Gaussian point size parameter map, a full-size Gaussian depth residual map, a full-size Gaussian point transparency parameter map and a full-size Gaussian point color parameter map are obtained by processing the contained image according to the depth information and the target image.

9. The Gaussian point cloud estimation method according to claim 1, wherein the processing of the depth information, the images contained in the target image pair, the Gaussian parameters of the target image pair, and the device parameters of the shooting device to obtain Gaussian point cloud data comprises: According to the device parameters of the shooting device, the depth information, the images contained in the target image pair and the Gaussian parameters of the target image pair are back-projected to obtain Gaussian point cloud data.

10. An image display device comprising a memory, a processor and a naked-eye holographic display screen; The processor is used to: Obtaining a first image feature of a first image and a second image feature of a second image in a target image pair, wherein the first image and the second image are respectively captured by two capturing devices; fusing the first image feature and the second image feature to obtain a first fused image feature and a second fused image feature; Determine depth information according to the first fused image feature and the second fused image feature; Processing the depth information, the images contained in the target image pair, Gaussian parameters of the target image pair, and device parameters of the shooting device to obtain Gaussian point cloud data; Generate a viewpoint image according to the Gaussian point cloud data; The viewpoint image is displayed through the naked-eye holographic display screen.

Images