Three-dimensional light field display optimization method, system, electronic device and storage medium

Abstract

The present invention provides a three-dimensional light field display optimization method, system, electronic device and storage medium, the method includes obtaining display information in a three-dimensional light field based on light field information; constructing a viewpoint model of a three-dimensional light field according to a quantitative relationship; establishing a dense viewpoint fitting algorithm based on the viewpoint model, and obtaining three-dimensional reconstructed light field image information according to an optimal light beam divergence angle; performing matrix multiplication calculation based on a processed initial unit image array and a processed high-resolution unit image array, and loading the unit image array into a display panel; inputting the unit image array and the weighted matrix array into a convolutional neural network to output an image after a deconvolution operation; folding each side information in a preset order, and updating the topological information of each side information after folding to simplify the image after the deconvolution operation. The present invention can avoid the situation of parallax imbalance, and can simplify large and complex three-dimensional scenes, and avoid the problem of display difficulties.

Description

Technical Field

[0001] The present invention relates to the field of image display technology, and in particular to a three-dimensional light field display optimization method, system, electronic equipment and storage medium. Background Art

[0002] With the advancement of display technology, people's demand for display experience is also increasing. A new display technology that can truly express three-dimensional depth information, three-dimensional display technology, has emerged. Thanks to the rapid development of computer vision technology, computer vision technology can obtain three-dimensional information of objects from multiple angles, and integrate the three-dimensional information into three-dimensional images that can be presented on the display through algorithms, so that observers can perceive the true shape and spatial position of objects from multiple angles.

[0003] Among many display technologies, three-dimensional light field display technology can reproduce the real light field distribution of three-dimensional scenes and achieve dynamic naked-eye three-dimensional display effects. It has been widely used in various fields such as medical care and education.

[0004] Among the existing technologies, the current three-dimensional light field display technology has the problem of parallax imbalance, which will cause visual fatigue and dizziness to the observer, and when displaying large and complex three-dimensional scenes, it will be difficult to display model data, and it will increase the difficulty and complexity of processing three-dimensional objects and scene data. Summary of the invention

[0005] Based on this, an object of the present invention is to provide a three-dimensional light field display optimization method, system, electronic device and storage medium to at least solve the above-mentioned deficiencies in the prior art.

[0006] In a first aspect, the present invention provides a three-dimensional light field display optimization method, the method comprising:

[0007] Traversing different observation positions in the space, recording the parallax light field captured at different viewpoint angles of each observation position to collect light field information, and obtaining display information in the three-dimensional light field based on the light field information;

[0008] Acquire the width of the viewpoint area of ​​the different viewpoints, acquire the light beam divergence angle based on the display information, and acquire the quantitative relationship between the width of the viewpoint area and the light beam divergence angle, and construct the viewpoint model of the three-dimensional light field according to the quantitative relationship;

[0009] Establishing a dense viewpoint fitting algorithm based on the viewpoint model, fitting information of overlapping areas of adjacent light beams with an ideal light field with dense viewpoints according to the dense viewpoint fitting algorithm, solving the dense viewpoint fitting algorithm to obtain optimal light beam divergence angles under the different viewpoint angles, and obtaining three-dimensional reconstructed light field image information according to the optimal light beam divergence angle;

[0010] Capturing image data of the three-dimensional reconstructed light field image information, acquiring an initial unit image array and a high-resolution unit image array of the image data by using synthetic image coding, processing the initial unit image array and the high-resolution unit image array, performing matrix multiplication calculation based on the processed initial unit image array and the processed high-resolution unit image array to obtain a unit image array, and loading the unit image array into a display panel;

[0011] Generate a plurality of weighting matrices based on the viewpoint model, form a weighting matrix array based on the plurality of weighting matrices, and input the unit image array and the weighting matrix array into a convolutional neural network to output an image after a deconvolution operation;

[0012] Acquire each side information in the image after the deconvolution operation, fold the each side information in a preset order, and update the topological information of the folded each side information to simplify the image after the deconvolution operation.

[0013] Compared with the prior art, the beneficial effects of the present invention are: a dense viewpoint fitting algorithm is established through a viewpoint model, and by fitting and solving the dense viewpoint fitting algorithm, the optimal light beam divergence angle can be obtained to obtain three-dimensional reconstructed light field image information, thereby effectively avoiding the parallax imbalance, and the unit image array obtained by matrix multiplication calculation can avoid dizziness and visual fatigue of the observer when observing the image, and by inputting the weighted matrix array into a convolutional neural network and updating the topological information of the folded edge information, the image can be simplified, thereby simplifying large and complex three-dimensional scenes and avoiding the problem of display difficulties.

[0014] Furthermore, after the step of collecting light field information, the method further includes:

[0015] Using a light control element to modulate the light in the light field information to obtain modulated light;

[0016] The parallax light field captured at the different viewpoint angles is decoded, and a position through which the light passes is adjusted according to the observation position and the modulated light.

[0017] Furthermore, the expression of the viewpoint model of the three-dimensional light field is:

[0018]

[0019] In the formula, represents a three-dimensional light field, represents the number of viewpoints, Represents viewpoint information, Indicates the number of pixels, Indicates the width of the viewpoint area, Represents two-dimensional image information.

[0020] Furthermore, the step of establishing a dense viewpoint fitting algorithm based on the viewpoint model includes:

[0021] A dense viewpoint fitting algorithm is established based on the viewpoint model and dense viewpoints with the beam divergence angle as an unknown parameter. The expression of the dense viewpoint fitting algorithm is:

[0022]

[0023] In the formula, represents the dense fitting algorithm, The disparity sequence diagram representing the ideal light field, represents the unknown parameter matrix with the beam divergence angle, represents the number of viewpoints, Represents two-dimensional image information.

[0024] Furthermore, the step of obtaining three-dimensional reconstructed light field image information according to the optimal light beam divergence angle includes:

[0025] Acquire the overlapping part of the optimal light beam divergence angles of the different viewpoints, so that the overlapping part serves as a light field information carrier;

[0026] Based on the light field information carrier, an accurate viewpoint image is provided for the three-dimensionally reconstructed light field to obtain three-dimensionally reconstructed light field image information.

[0027] Furthermore, after the step of outputting the image after the deconvolution operation, the method includes:

[0028] Evaluating the image after the deconvolution operation based on peak signal-to-noise ratio and structural similarity to obtain image evaluation parameters;

[0029] Reconstructing the difference between the image after the deconvolution operation and the ideal image according to the evaluation parameters to obtain a reconstructed image;

[0030] Wherein, the expression for evaluating the image after the deconvolution operation based on the peak signal-to-noise ratio is:

[0031] ;

[0032] The expression for evaluating the image after the deconvolution operation based on the structural similarity is:

[0033] ;

[0034] In the formula, represents the peak signal-to-noise ratio, represents the mean square error between the image after the deconvolution operation and the ideal image, Indicates the number of bits of data corresponding to each image sampling pixel, Indicates the structural similarity, , Respectively represent the pixel indexes of the two images after deconvolution operations, Represents the mean of the pixel indices of one of the images after the deconvolution operation, Represents the mean of the pixel index of another image after deconvolution operation, The variance of the corresponding index, express and The covariance of represents the first constant, Represents the second constant.

[0035] Furthermore, the step of obtaining each side information in the image after the deconvolution operation, folding each side information in a preset order, and updating the topological information of each side information after folding to simplify the image after the deconvolution operation includes:

[0036] Reading the model data in the image after the deconvolution operation, and obtaining the mesh topology information in the model data;

[0037] Calculating the folding costs of the vertices in the mesh topology information, and constructing a priority folding queue according to the folding costs;

[0038] performing edge folding operations on the mesh topology information in sequence based on the priority folding queue to obtain folded mesh topology information;

[0039] The surrounding topological information of the folded mesh topological information is updated, and it is determined whether the folded mesh topological information meets the preset simplification requirement, and if so, a simplified image is output.

[0040] In a second aspect, the present invention further provides a three-dimensional light field display optimization system, the system comprising:

[0041] A traversal recording module, used to traverse different observation positions in the space, record the parallax light field captured at different viewpoint angles of each observation position, so as to collect light field information, and obtain display information in the three-dimensional light field based on the light field information;

[0042] An acquisition and construction module is used to acquire the width of the viewpoint area of ​​the different viewpoints, acquire the light beam divergence angle based on the display information, and acquire the quantitative relationship between the width of the viewpoint area and the light beam divergence angle, and construct the viewpoint model of the three-dimensional light field according to the quantitative relationship;

[0043] Establishing a fitting module, which is used to establish a dense viewpoint fitting algorithm based on the viewpoint model, fitting the information of the overlapping area of ​​adjacent light beams with the ideal light field with dense viewpoints according to the dense viewpoint fitting algorithm, and solving the dense viewpoint fitting algorithm to obtain the optimal light beam divergence angle under the different viewpoint angles, and obtaining three-dimensional reconstructed light field image information according to the optimal light beam divergence angle;

[0044] a capture processing module, used to capture image data of the three-dimensional reconstructed light field image information, obtain an initial unit image array and a high-resolution unit image array of the image data by using synthetic image coding, process the initial unit image array and the high-resolution unit image array, perform matrix multiplication calculation based on the processed initial unit image array and the processed high-resolution unit image array to obtain a unit image array, and load the unit image array into a display panel;

[0045] A generating output module is used to generate a plurality of weighting matrices based on the viewpoint model, form a weighting matrix array based on the plurality of weighting matrices, and input the unit image array and the weighting matrix array into a convolutional neural network to output an image after a deconvolution operation;

[0046] The simplification module is used to obtain each side information in the image after the deconvolution operation, fold the each side information in a preset order, and update the topological information of the folded each side information to simplify the image after the deconvolution operation.

[0047] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned three-dimensional light field display optimization method when executing the computer program.

[0048] In a fourth aspect, the present invention further provides a storage medium having a computer program stored thereon, which implements the above-mentioned three-dimensional light field display optimization method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a three-dimensional light field display optimization method in a first embodiment of the present invention;

[0050] Figure 2 is a structural block diagram of a three-dimensional light field display optimization system in a second embodiment of the present invention;

[0051] Figure 3 FIG. 4 is a schematic diagram of the hardware structure of an electronic device in a third embodiment of the present invention.

[0052] Description of main component symbols:

[0053] 10. Traverse the record module;

[0054] 20. Get the building blocks;

[0055] 30. Establish a fitting module;

[0056] 40. Capture processing module;

[0057] 50. Generate output module;

[0058] 60. Get the simplified module;

[0059] 70. Bus; 71. Processor; 72. Memory; 73. Communication interface.

[0060] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0061] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0062] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0064] Embodiment 1

[0065] See also Figure 1 , which shows a three-dimensional light field display optimization method in a first embodiment of the present invention, the method comprises steps S1 to S6:

[0066] S1, traversing different observation positions in the space, recording the parallax light field captured at different viewpoint angles of each observation position to collect light field information, and obtaining display information in the three-dimensional light field based on the light field information;

[0067] It can be understood that the light propagating from an object in space can be represented by multiple parameters. In the present embodiment, it can be represented by four parameters and therefore can be recorded as four-dimensional light field data. By traversing different observation positions in space and recording the parallax light field captured at different viewpoint angles at each observation position, information collection of a three-dimensional light field with full parallax can be achieved.

[0068] Specifically, the step S1 includes steps S11 to S12:

[0069] S11, using a light control element to modulate the light in the light field information to obtain modulated light;

[0070] S12, decoding the parallax light field captured at different viewpoint angles, and adjusting the position where the light passes according to the observation position and the modulated light;

[0071] It can be understood that according to the reversibility principle of the optical path, the light is modulated by using the light control element, the parallax light field captured from different viewpoints is decoded, and according to the observation position and the modulated light, the light is allowed to pass through the corresponding position, thereby bringing the viewer a real and three-dimensional experience with parallax.

[0072] S2, obtaining widths of viewpoint areas of different viewpoints, obtaining light beam divergence angles based on the display information, and obtaining a quantitative relationship between the widths of the viewpoint areas and the light beam divergence angles, and constructing a viewpoint model of the three-dimensional light field according to the quantitative relationship;

[0073] It can be understood that, due to the existence of a certain divergence angle of the light beam, the viewpoint area mapped by the light beam on the viewing plane is widened. In this embodiment, the width of the viewpoint area is first obtained, and the light beam divergence angle is obtained. The width of the viewpoint area is associated with the light beam divergence angle. Therefore, the viewpoint model of the three-dimensional light field can be established through the quantitative relationship between the width of the viewpoint area and the light beam divergence angle. In this embodiment, the expression of the viewpoint model of the three-dimensional light field is:

[0074]

[0075] In the formula, represents a three-dimensional light field, represents the number of viewpoints, Represents viewpoint information, Indicates the number of pixels, Indicates the width of the viewpoint area, It can be seen that the viewpoint model takes into account the influence of the beam divergence angle on the viewpoint distribution, reflects the phenomenon of beam overlap and information fusion, and can more accurately and completely describe the information of the actual three-dimensional light field.

[0076] S3, establishing a dense viewpoint fitting algorithm based on the viewpoint model, fitting the information of the overlapping area of ​​adjacent light beams with the ideal light field with dense viewpoints according to the dense viewpoint fitting algorithm, solving the dense viewpoint fitting algorithm to obtain the optimal light beam divergence angle under the different viewpoint angles, and obtaining three-dimensional reconstructed light field image information according to the optimal light beam divergence angle;

[0077] It can be understood that due to the existence of the beam divergence angle, the beams of adjacent viewpoint information will overlap after passing through the cylindrical lens, and the overlapping areas can present new viewpoint image information to the human eye respectively;

[0078] Specifically, the step S3 includes steps S31 to S33:

[0079] S31, establishing a dense viewpoint fitting algorithm based on the viewpoint model and dense viewpoints with beam divergence angles as unknown parameters, wherein the expression of the dense viewpoint fitting algorithm is:

[0080]

[0081] In the formula, represents the dense fitting algorithm, The disparity sequence diagram representing the ideal light field, represents the unknown parameter matrix with the beam divergence angle, represents the number of viewpoints, Represents two-dimensional image information.

[0082] S32, obtaining an overlapping portion of the optimal light beam divergence angles of the different viewpoints, so that the overlapping portion serves as a light field information carrier;

[0083] S33, providing an accurate viewpoint image for a three-dimensionally reconstructed light field based on the light field information carrier to obtain three-dimensionally reconstructed light field image information;

[0084] It should be noted that a dense viewpoint fitting algorithm with the beam divergence angle as an unknown parameter is established. The algorithm fits the information of the overlapping area of ​​adjacent light beams with the ideal light field with dense viewpoints, so that the overlapping part serves as an additional light field information carrier, thereby providing accurate viewpoint images to the human eye at more positions. The three-dimensional reconstructed light field image information obtained by the optimal beam divergence angle under different viewpoint angles makes the three-dimensional light field close to the display effect of the ideal light field, avoiding the problem of parallax imbalance.

[0085] S4, capturing image data of the three-dimensional reconstructed light field image information, acquiring an initial unit image array and a high-resolution unit image array of the image data by using synthetic image coding, processing the initial unit image array and the high-resolution unit image array, performing matrix multiplication calculation based on the processed initial unit image array and the processed high-resolution unit image array to obtain a unit image array, and loading the unit image array into a display panel;

[0086] It can be understood that a camera array or a virtual camera array is used to shoot a target three-dimensional scene and capture a set of parallax image sets. In this embodiment, a virtual camera array is used for acquisition to obtain image data, and then synthetic image encoding is used to obtain an initial unit image array and a high-resolution unit image array of the image data. The initial unit image array and the high-resolution unit image array are input into a convolutional neural network for processing, and then matrix multiplication is performed on the processed initial unit image array and the processed high-resolution unit image array. This can be used to simulate the process in which the element images in a real light field display overlap after passing through a lens, so that a unit image array for loading can be obtained, and the unit image array is loaded in a real panel, so that the information of the overlapping area of ​​the light beam divergence angle can be fitted with the ideal light field of dense viewpoints, thereby achieving smooth motion parallax and avoiding parallax imbalance.

[0087] S5, generating a plurality of weighting matrices based on the viewpoint model, forming a weighting matrix array based on the plurality of weighting matrices, and inputting the unit image array and the weighting matrix array into a convolutional neural network to output an image after the deconvolution operation;

[0088] It can be understood that several weighted matrices can be generated through the viewpoint model, which can reflect the influence of the beam divergence angle of the lens unit at different positions on the viewpoint distribution. Therefore, a weighted matrix array composed of multiple weighted matrices can be further extracted, and the unit image array and the weighted matrix array are matrix multiplied. The multiplication result is packaged as the input of the convolutional neural network, which can improve the correlation between the unit image array and the weighted matrix array. In this embodiment, the convolutional neural network uses five convolutional layers, and the five convolutional layers process each unit image array in turn. Then, through the deconvolution operation, the resolution of the image can be gradually improved, and the processed unit image array is made to be the same size as the input initial unit image.

[0089] In addition, the step S5 further includes steps S51 to S52:

[0090] S51, evaluating the image after the deconvolution operation based on peak signal-to-noise ratio and structural similarity to obtain image evaluation parameters;

[0091] S52, reconstructing the difference between the image after the deconvolution operation and the ideal image according to the evaluation parameter to obtain a reconstructed image;

[0092] Wherein, the expression for evaluating the image after the deconvolution operation based on the peak signal-to-noise ratio is:

[0093] ;

[0094] The expression for evaluating the image after the deconvolution operation based on the structural similarity is:

[0095] ;

[0096] In the formula, represents the peak signal-to-noise ratio, represents the mean square error between the image after the deconvolution operation and the ideal image, Indicates the number of bits of data corresponding to each image sampling pixel, Indicates the structural similarity, , Respectively represent the pixel indexes of the two images after deconvolution operations, Represents the mean of the pixel indices of one of the images after the deconvolution operation, Represents the mean of the pixel index of another image after deconvolution operation, The variance of the corresponding index, express and The covariance of represents the first constant, Represents the second constant.

[0097] It can be understood that in order to evaluate the image after the deconvolution operation, the image after the deconvolution operation is evaluated by the peak signal-to-noise ratio and the structural similarity. The closer the value is to 1, the smaller the structural difference between the image and the ideal image is, the higher the image quality is, and the image after the deconvolution operation can be reconstructed according to the difference between the image after the deconvolution operation and the ideal image. That is, by adjusting the image after the deconvolution operation, the reconstructed image quality can be made higher.

[0098] S6, obtaining each side information in the image after the deconvolution operation, folding each side information in a preset order, and updating the topological information of each side information after folding, so as to simplify the image after the deconvolution operation;

[0099] Specifically, the step S6 includes steps S61 to S64:

[0100] S61, reading the model data in the image after the deconvolution operation, and obtaining the mesh topology information in the model data;

[0101] It is understandable that the 3D model mesh in the image will have more feature information. By reading the model data in the image after the deconvolution operation, the mesh topology information in the model data can be obtained. For subsequent folding, it is necessary to calculate the vertex normal vector and constraint factor in the mesh topology information. The vertex normal vector and triangle normal vector are used in the process of calculating the constraint factor and folding cost. The calculation expression of the vertex normal vector is:

[0102] ;

[0103] In the formula, Represents a vertex The normal vector of represents the number of adjacent triangles, Indicates The face normal vector of the triangle, Indicates The area of ​​a triangle;

[0104] In addition, it is worth noting that, in this embodiment, the constraint factor is the local undulation of the triangle vertex, and the constraint factor is expressed as:

[0105] ;

[0106] In the formula, Represents the scale factor.

[0107] S62, calculating the folding costs of the vertices in the mesh topology information, and constructing a priority folding queue according to the folding costs;

[0108] Specifically, in this embodiment, the calculation expression of the folding cost is:

[0109] ;

[0110] In the formula, represents the folding cost, , Represents vertices ,vertex The local fluctuation of the vertex, , Represents vertices ,vertex The local average area of ​​the vertices;

[0111] It can be understood that a queue for folding each edge can be established according to the folding cost.

[0112] S63, performing edge folding operations on the mesh topology information in sequence based on the priority folding queue to obtain folded mesh topology information;

[0113] S64, updating the surrounding topology information of the folded mesh topology information, and determining whether the folded mesh topology information meets the preset simplification requirement, and if so, outputting a simplified image;

[0114] It can be understood that the edges of the mesh topology information are folded according to the queue for folding each edge, so that the folded mesh topology information can be obtained, and the topology information around it is updated in real time, and then a judgment is made. In this embodiment, the preset simplification requirement is to retain the degree of features, and finally a simplified image is output to achieve the purpose of simplifying the complex three-dimensional scene model. In addition, the degree of feature retention in the simplification process can be improved through the relevant vertex undulation, the local area of ​​the top and the constraint factor, so that the three-dimensional scene model can be more realistic after simplification.

[0115] In summary, the three-dimensional light field display optimization method in the above-mentioned embodiment of the present invention establishes a dense viewpoint fitting algorithm through a viewpoint model, and by fitting and solving the dense viewpoint fitting algorithm, the optimal light beam divergence angle can be obtained to obtain three-dimensional reconstructed light field image information, thereby effectively avoiding the parallax imbalance, and the unit image array obtained by matrix multiplication calculation can avoid dizziness and visual fatigue of the observer when observing the image, and by inputting the weighted matrix array into the convolutional neural network and updating the topological information of the folded edge information, the image can be simplified, thereby simplifying large and complex three-dimensional scenes and avoiding the problem of display difficulties.

[0116] Embodiment 2

[0117] See also Figure 2 , which shows a three-dimensional light field display optimization system in a second embodiment of the present invention, the system comprises:

[0118] A traversal recording module 10 is used to traverse different observation positions in the space, record the parallax light field captured at different viewpoint angles of each observation position, so as to collect light field information, and obtain display information in the three-dimensional light field based on the light field information;

[0119] The acquisition construction module 20 is used to obtain the width of the viewpoint area of ​​the different viewpoints, obtain the light beam divergence angle based on the display information, and obtain the quantitative relationship between the width of the viewpoint area and the light beam divergence angle, and construct the viewpoint model of the three-dimensional light field according to the quantitative relationship. The expression of the viewpoint model of the three-dimensional light field is:

[0120]

[0121] In the formula, represents a three-dimensional light field, represents the number of viewpoints, Represents viewpoint information, Indicates the number of pixels, Indicates the width of the viewpoint area, Represents two-dimensional image information;

[0122] Establishing a fitting module 30, which is used to establish a dense viewpoint fitting algorithm based on the viewpoint model, fitting the information of the overlapping area of ​​adjacent light beams with the ideal light field with dense viewpoints according to the dense viewpoint fitting algorithm, and solving the dense viewpoint fitting algorithm to obtain the optimal light beam divergence angle under the different viewpoint angles, and obtaining three-dimensional reconstructed light field image information according to the optimal light beam divergence angle;

[0123] A capture processing module 40, used for capturing image data of the three-dimensional reconstructed light field image information, acquiring an initial unit image array and a high-resolution unit image array of the image data by using synthetic image coding, processing the initial unit image array and the high-resolution unit image array, performing matrix multiplication calculation based on the processed initial unit image array and the processed high-resolution unit image array to obtain a unit image array, and loading the unit image array into a display panel;

[0124] A generating output module 50 is used to generate a plurality of weighting matrices based on the viewpoint model, form a weighting matrix array based on the plurality of weighting matrices, and input the unit image array and the weighting matrix array into a convolutional neural network to output an image after a deconvolution operation;

[0125] The acquisition simplification module 60 is used to obtain each side information in the image after the deconvolution operation, fold the each side information in a preset order, and update the topological information of the folded each side information to simplify the image after the deconvolution operation.

[0126] In some optional embodiments, the traversal record module 10 includes:

[0127] A modulation unit, used to modulate the light in the light field information by using a light control element to obtain modulated light;

[0128] A decoding unit is used to decode the parallax light field captured at different viewpoint angles, and adjust the position where the light passes according to the observation position and the modulated light.

[0129] In some optional embodiments, the establishing the fitting module 30 includes:

[0130] An establishing unit is used to establish a dense viewpoint fitting algorithm based on the viewpoint model and dense viewpoints with beam divergence angles as unknown parameters, wherein the expression of the dense viewpoint fitting algorithm is:

[0131]

[0132] In the formula, represents the dense fitting algorithm, The disparity sequence diagram representing the ideal light field, represents the unknown parameter matrix with the beam divergence angle, represents the number of viewpoints, Represents two-dimensional image information;

[0133] An acquisition unit, configured to acquire an overlapping portion of the optimal light beam divergence angles of the different viewpoints, so that the overlapping portion serves as a light field information carrier;

[0134] A providing unit is used to provide accurate viewpoint images for a three-dimensionally reconstructed light field based on the light field information carrier to obtain three-dimensionally reconstructed light field image information.

[0135] In some optional embodiments, the generating output module 50 includes:

[0136] An evaluation unit, used for evaluating the image after the deconvolution operation based on a peak signal-to-noise ratio and a structural similarity to obtain an image evaluation parameter;

[0137] a reconstruction unit, configured to reconstruct the difference between the image after the deconvolution operation and the ideal image according to the evaluation parameters to obtain a reconstructed image;

[0138] Wherein, the expression for evaluating the image after the deconvolution operation based on the peak signal-to-noise ratio is:

[0139] ;

[0140] The expression for evaluating the image after the deconvolution operation based on the structural similarity is:

[0141] ;

[0142] In the formula, represents the peak signal-to-noise ratio, represents the mean square error between the image after the deconvolution operation and the ideal image, Indicates the number of bits of data corresponding to each image sampling pixel, Indicates the structural similarity, , Respectively represent the pixel indexes of the two images after deconvolution operations, Represents the mean of the pixel indices of one of the images after the deconvolution operation, Represents the mean of the pixel index of another image after deconvolution operation, The variance of the corresponding index, express and The covariance of represents the first constant, Represents the second constant.

[0143] In some optional embodiments, the acquisition simplification module 60 includes:

[0144] A reading unit, used to read the model data in the image after the deconvolution operation, and obtain the grid topology information in the model data;

[0145] A calculation and construction unit, used for calculating the folding costs of the vertices in the mesh topology information, and constructing a priority folding queue according to the folding costs;

[0146] A folding unit, configured to sequentially perform edge folding operations on the mesh topology information based on the priority folding queue to obtain folded mesh topology information;

[0147] The updating unit is used to update the surrounding topology information of the folded mesh topology information and determine whether the folded mesh topology information meets the preset simplification requirement, and if so, output a simplified image.

[0148] The functions or operation steps implemented when the above modules and units are executed are generally the same as those in the above method embodiments, and will not be repeated here.

[0149] The three-dimensional light field display optimization system provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0150] Embodiment 3

[0151] See also Figure 3 , which is a schematic diagram of the hardware structure of the electronic device in the third embodiment of the present invention.

[0152] The electronic device may include a processor 71 and a memory 72 storing computer program instructions.

[0153] Specifically, the processor 71 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the present application.

[0154] Among them, the memory 72 may include a large-capacity memory for data or instructions. For example, but not limitation, the memory 72 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 72 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 72 may be inside or outside the data processing device. In a specific embodiment, the memory 72 is a non-volatile memory. In a specific embodiment, the memory 72 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0155] The memory 72 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 71 .

[0156] The processor 71 implements the three-dimensional light field display optimization method of the first embodiment by reading and executing the computer program instructions stored in the memory 72 .

[0157] In some of the embodiments, the electronic device may further include a communication interface 73 and a bus 70. Figure 3 As shown, the processor 71, the memory 72, and the communication interface 73 are connected via a bus 70 and communicate with each other.

[0158] The communication interface 73 is used to realize the communication between the modules, devices, units and / or equipment in the present application. The communication interface 73 can also realize data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage and image / data processing workstations.

[0159] The bus 70 includes hardware, software or both, and couples the components of the device to each other. The bus 70 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 70 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 70 may include one or more buses. Although this application describes and illustrates a particular bus, this application contemplates any suitable bus or interconnect.

[0160] The electronic device can obtain the three-dimensional light field display optimization system and execute the three-dimensional light field display optimization method of the first embodiment.

[0161] In addition, in combination with the three-dimensional light field display optimization method in the first embodiment, the present application can provide a storage medium for implementation. The storage medium stores computer program instructions; when the computer program instructions are executed by the processor, the three-dimensional light field display optimization method in the first embodiment is implemented.

[0162] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0163] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A three-dimensional light field display optimization method, characterized in that: The method comprises: Traversing different observation positions in the space, recording the parallax light field captured at different viewpoint angles of each observation position to collect light field information, and obtaining display information in the three-dimensional light field based on the light field information; Acquire the width of the viewpoint area of ​​the different viewpoints, acquire the light beam divergence angle based on the display information, and acquire the quantitative relationship between the width of the viewpoint area and the light beam divergence angle, and construct the viewpoint model of the three-dimensional light field according to the quantitative relationship; Establishing a dense viewpoint fitting algorithm based on the viewpoint model, fitting information of overlapping areas of adjacent light beams with an ideal light field with dense viewpoints according to the dense viewpoint fitting algorithm, solving the dense viewpoint fitting algorithm to obtain optimal light beam divergence angles under the different viewpoint angles, and obtaining three-dimensional reconstructed light field image information according to the optimal light beam divergence angle; Capturing image data of the three-dimensional reconstructed light field image information, acquiring an initial unit image array and a high-resolution unit image array of the image data by using synthetic image coding, processing the initial unit image array and the high-resolution unit image array, performing matrix multiplication calculation based on the processed initial unit image array and the processed high-resolution unit image array to obtain a unit image array, and loading the unit image array into a display panel; Generate a plurality of weighting matrices based on the viewpoint model, form a weighting matrix array based on the plurality of weighting matrices, and input the unit image array and the weighting matrix array into a convolutional neural network to output an image after a deconvolution operation; Acquire each side information in the image after the deconvolution operation, fold the each side information in a preset order, and update the topological information of the folded each side information to simplify the image after the deconvolution operation.

2. The three-dimensional light field display optimization method according to claim 1, characterized in that: After the step of collecting light field information, the method further includes: Using a light control element to modulate the light in the light field information to obtain modulated light; The parallax light field captured at the different viewpoint angles is decoded, and a position through which the light passes is adjusted according to the observation position and the modulated light.

3. The three-dimensional light field display optimization method according to claim 1, characterized in that: The expression of the viewpoint model of the three-dimensional light field is: In the formula, represents a three-dimensional light field, represents the number of viewpoints, Represents viewpoint information, Indicates the number of pixels, Indicates the width of the viewpoint area, Represents two-dimensional image information.

4. The three-dimensional light field display optimization method according to claim 1, characterized in that: The step of establishing a dense viewpoint fitting algorithm based on the viewpoint model comprises: A dense viewpoint fitting algorithm is established based on the viewpoint model and dense viewpoints with the beam divergence angle as an unknown parameter. The expression of the dense viewpoint fitting algorithm is: In the formula, represents the dense viewpoint fitting algorithm, The disparity sequence diagram representing the ideal light field, represents the unknown parameter matrix with the beam divergence angle, represents the number of viewpoints, Represents two-dimensional image information.

5. The three-dimensional light field display optimization method according to claim 1, characterized in that: The step of obtaining three-dimensional reconstructed light field image information according to the optimal light beam divergence angle comprises: Acquire the overlapping part of the optimal light beam divergence angles of the different viewpoints, so that the overlapping part serves as a light field information carrier; Based on the light field information carrier, an accurate viewpoint image is provided for the three-dimensionally reconstructed light field to obtain three-dimensionally reconstructed light field image information.

6. The three-dimensional light field display optimization method according to claim 1, characterized in that: After the step of outputting the image after the deconvolution operation, the method comprises: Evaluating the image after the deconvolution operation based on peak signal-to-noise ratio and structural similarity to obtain image evaluation parameters; Reconstructing the difference between the image after the deconvolution operation and the ideal image according to the evaluation parameters to obtain a reconstructed image; Wherein, the expression for evaluating the image after the deconvolution operation based on the peak signal-to-noise ratio is: ； The expression for evaluating the image after the deconvolution operation based on the structural similarity is: ； In the formula, represents the peak signal-to-noise ratio, represents the mean square error between the image after the deconvolution operation and the ideal image, Indicates the number of bits of data corresponding to each image sampling pixel, Indicates the structural similarity, , Respectively represent the pixel indexes of the two images after deconvolution operations, Represents the mean of the pixel indices of one of the images after the deconvolution operation, Represents the mean of the pixel index of another image after deconvolution operation, The variance of the corresponding index, express and The covariance of represents the first constant, Represents the second constant.

7. The three-dimensional light field display optimization method according to claim 1, characterized in that: The step of obtaining each side information in the image after the deconvolution operation, folding each side information in a preset order, and updating the topological information of each side information after folding to simplify the image after the deconvolution operation includes: Reading the model data in the image after the deconvolution operation, and obtaining the mesh topology information in the model data; Calculating the folding costs of the vertices in the mesh topology information, and constructing a priority folding queue according to the folding costs; performing edge folding operations on the mesh topology information in sequence based on the priority folding queue to obtain folded mesh topology information; The surrounding topological information of the folded mesh topological information is updated, and it is determined whether the folded mesh topological information meets the preset simplification requirement, and if so, a simplified image is output.

8. A three-dimensional light field display optimization system, characterized in that: The system comprises: A traversal recording module, used to traverse different observation positions in the space, record the parallax light field captured at different viewpoint angles of each observation position, so as to collect light field information, and obtain display information in the three-dimensional light field based on the light field information; An acquisition and construction module is used to acquire the width of the viewpoint area of ​​the different viewpoints, acquire the light beam divergence angle based on the display information, and acquire the quantitative relationship between the width of the viewpoint area and the light beam divergence angle, and construct the viewpoint model of the three-dimensional light field according to the quantitative relationship; Establishing a fitting module, which is used to establish a dense viewpoint fitting algorithm based on the viewpoint model, fitting the information of the overlapping area of ​​adjacent light beams with the ideal light field with dense viewpoints according to the dense viewpoint fitting algorithm, and solving the dense viewpoint fitting algorithm to obtain the optimal light beam divergence angle under the different viewpoint angles, and obtaining three-dimensional reconstructed light field image information according to the optimal light beam divergence angle; a capture processing module, used to capture image data of the three-dimensional reconstructed light field image information, obtain an initial unit image array and a high-resolution unit image array of the image data by using synthetic image coding, process the initial unit image array and the high-resolution unit image array, perform matrix multiplication calculation based on the processed initial unit image array and the processed high-resolution unit image array to obtain a unit image array, and load the unit image array into a display panel; A generating output module is used to generate a plurality of weighting matrices based on the viewpoint model, form a weighting matrix array based on the plurality of weighting matrices, and input the unit image array and the weighting matrix array into a convolutional neural network to output an image after a deconvolution operation; The simplification module is used to obtain each side information in the image after the deconvolution operation, fold the each side information in a preset order, and update the topological information of the folded each side information to simplify the image after the deconvolution operation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the three-dimensional light field display optimization method according to any one of claims 1 to 7 is implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the three-dimensional light field display optimization method according to any one of claims 1 to 7 is implemented.

Images