Optical three-dimensional measurement method, device, equipment and medium

Abstract

The invention discloses an optical three-dimensional measurement method, device and equipment and a medium, relates to the technical field of optical three-dimensional measurement, and is used for solving the problem of low optical three-dimensional measurement precision under the shielding condition in the prior art. The method comprises the following steps: constructing four cost volumes according to a light field sub-aperture image array; by utilizing a self-attention mechanism, respectively endowing corresponding weight values for the unshielded view features and the shielded view features to obtain corresponding feature weight matrixes; according to the feature weight matrix, respectively carrying out weighted summation and feature extraction on the four cost volumes to obtain four depth images; performing guided filtering on the central view angle image and the four depth images of the light field original image to obtain four filtered depth images; and performing cascade operation on the four filtered depth images to obtain a target depth image. The shielding problem is solved through a self-attention mechanism and guide filtering, and the three-dimensional measurement precision is improved.

Description

Technical Field

[0001] The present application relates to the technical field of optical three-dimensional measurement, and provides an optical three-dimensional measurement method, device, equipment and medium. Background Art

[0002] As we all know, light field depth estimation is one of the representative technologies of optical three-dimensional measurement. It has important application value in intelligent manufacturing, robot vision, autonomous driving, industrial inspection, virtual reality, aerial detection and metaverse.

[0003] In the prior art, light field depth estimation is often performed based on frequency domain energy distribution, dynamic fusion network or multiple parallax scales. Among them, the light field depth estimation method based on frequency domain energy distribution can solve the influence of occlusion on light field depth estimation by distinguishing occluded points from non-occluded points and further processing the two types of points separately. However, this technical solution needs to manually design some feature descriptors to distinguish occluded points from non-occluded points, so that the anti-occlusion performance of the solution is seriously affected by the designed feature descriptors; although the light field depth estimation method based on dynamic fusion network builds a dynamic fusion network and combines a dual-stream network and a multimodal dynamic fusion network to realize light field depth estimation, this technical solution does not consider the influence of occlusion on light field depth estimation; in addition, although the fast light field depth estimation method based on multiple parallax scales uses the extracted feature vectors to construct cost volumes of different parallax scales and realizes light field depth estimation through cost volume aggregation, this technical solution also does not consider the influence of occlusion on light field depth estimation.

[0004] Therefore, how to improve the accuracy of optical three-dimensional measurement under occlusion conditions has become a problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides an optical three-dimensional measurement method, device, equipment and medium, which are used to solve the problem of low optical three-dimensional measurement accuracy under occlusion in the prior art.

[0006] In one aspect, an optical three-dimensional measurement method is provided, the method comprising:

[0007] Constructing four cost volumes according to the image features in the vertical, horizontal and two diagonal directions in the light field sub-aperture image array; wherein the light field sub-aperture image array is obtained by decoding the light field original image;

[0008] Using the self-attention mechanism, corresponding weight values ​​are assigned to the unoccluded view features and the occluded view features respectively to obtain the corresponding feature weight matrix;

[0009] According to the feature weight matrix, weighted summation and feature extraction are performed on the four cost volumes respectively to obtain four depth images;

[0010] Performing guided filtering on the central perspective image of the original light field image and the four depth images to obtain four filtered depth images;

[0011] The four filtered depth images are cascaded to obtain a target depth image; wherein the target depth image contains optical three-dimensional information of the original light field image.

[0012] Optionally, the step of constructing four cost volumes according to image features in vertical, horizontal and two diagonal directions in the light field sub-aperture image array includes:

[0013] Decoding a light field raw image into a light field sub-aperture image array;

[0014] Using a VGG-19 pre-trained model, extracting features of the light field sub-aperture image array to obtain features of the light field sub-aperture image array;

[0015] The light field sub-aperture image array features are subjected to feature shifting to construct the four cost volumes.

[0016] Optionally, the step of performing feature shifting on the light field sub-aperture image array features to construct the four cost volumes includes:

[0017] Feature shifting is performed on the image features in the vertical, horizontal and two diagonal directions in the light field sub-aperture image array features to construct the four cost volumes.

[0018] Optionally, the step of using the self-attention mechanism to assign corresponding weight values ​​to unobstructed view features and obstructed view features respectively to obtain a corresponding feature weight matrix includes:

[0019] Using the self-attention mechanism, unoccluded view features in each cost volume are determined;

[0020] Corresponding weight values ​​are assigned to the unobstructed view features and the obstructed view features respectively to obtain the corresponding feature weight matrix.

[0021] Optionally, the step of performing weighted summation and feature extraction on the four cost volumes respectively according to the feature weight matrix to obtain four depth images includes:

[0022] According to the feature weight matrix, weighted summation is performed on the four cost volumes respectively to obtain cost volume features corresponding to the four cost volumes respectively;

[0023] The cost volume features corresponding to each of the four cost volumes are extracted through convolution to obtain the four depth images.

[0024] Optionally, the step of performing guided filtering on the central perspective image of the original light field image and the four depth images to obtain four filtered depth images includes:

[0025] The convolution feature of the central viewing angle image of the original light field image is used as a guiding map for guided filtering, and guided filtering is performed on the four depth images to obtain the four filtered depth images.

[0026] Optionally, after performing a cascade operation on the four filtered depth images to obtain a target depth image, the method further includes:

[0027] Using a preset loss function, calculating the loss value of the target depth image and the real depth image to obtain a target loss value; wherein the real depth image is obtained based on the four cost volumes;

[0028] Update the network parameters of the VGG-19 pre-trained model according to the target loss value.

[0029] In one aspect, an optical three-dimensional measurement device is provided, the device comprising:

[0030] A cost volume construction unit is used to construct four cost volumes according to image features in vertical, horizontal and two diagonal directions in a light field sub-aperture image array; wherein the light field sub-aperture image array is obtained by decoding a light field original image;

[0031] A weight matrix obtaining unit, used to use a self-attention mechanism to assign corresponding weight values ​​to unobstructed view features and obstructed view features, respectively, to obtain a corresponding feature weight matrix;

[0032] A cost volume weighted summation unit, used to perform weighted summation and feature extraction on the four cost volumes according to the feature weight matrix to obtain four depth images;

[0033] A guided filtering unit, configured to perform guided filtering on the central viewing angle image of the original light field image and the four depth images to obtain four filtered depth images;

[0034] The target depth image obtaining unit is used to perform a cascade operation on the four filtered depth images to obtain a target depth image; wherein the target depth image contains optical three-dimensional information of the light field original image.

[0035] On the one hand, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein any one of the above methods is implemented when the processor executes the computer program.

[0036] In one aspect, a computer storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, any of the above methods is implemented.

[0037] Compared with the prior art, the beneficial effects of this application are:

[0038] In an embodiment of the present application, when performing optical three-dimensional measurement, first, four cost volumes can be constructed according to the image features in the vertical, horizontal and two diagonal directions in the light field sub-aperture image array; wherein the light field sub-aperture image array is obtained by decoding the light field original image; then, the self-attention mechanism can be used to assign corresponding weight values ​​to the unobstructed view features and the obstructed view features, respectively, to obtain the corresponding feature weight matrix; next, the four cost volumes can be weighted summed and feature extracted according to the feature weight matrix, respectively, to obtain four depth images; then, the central view image of the light field original image and the four depth images can be guided filtered to obtain four filtered depth images; finally, the four filtered depth images can be cascaded to obtain the target depth image; wherein the optical three-dimensional information of the light field original image in the target depth image. Therefore, in an embodiment of the present application, since the self-attention mechanism is used to assign corresponding weight values ​​to the unobstructed view features and the obstructed view features, respectively, more attention can be paid to the unobstructed view features, thereby improving the three-dimensional measurement accuracy by solving the occlusion problem. In addition, since guided filtering is also performed on the central perspective image and four depth images of the original light field image, the obtained depth map can be filtered with the help of learnable guided filtering to further improve the three-dimensional measurement accuracy by reducing outliers and noise in the depth map. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0040] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0041] Figure 2 A schematic diagram of a flow chart of an optical three-dimensional measurement method provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of the principle of optical three-dimensional measurement provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of a flow chart of a specific implementation process of optical three-dimensional measurement provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of an optical three-dimensional measurement device provided in an embodiment of the present application.

[0045] Markings in the figure: 10 - optical three-dimensional measurement device, 101 - processor, 102 - memory, 103 - I / O interface, 104 - database, 50 - optical three-dimensional measurement device, 501 - cost volume construction unit, 502 - weight matrix acquisition unit, 503 - cost volume weighted summation unit, 504 - guided filtering unit, 505 - target depth image acquisition unit, 506 - network parameter updating unit. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0047] As we all know, light field depth estimation is one of the representative technologies of optical three-dimensional measurement. It has important application value in intelligent manufacturing, robot vision, autonomous driving, industrial inspection, virtual reality, aerial detection and metaverse.

[0048] In the prior art, light field depth estimation is often performed based on frequency domain energy distribution, dynamic fusion network or multiple parallax scales. Among them, the light field depth estimation method based on frequency domain energy distribution can solve the influence of occlusion on light field depth estimation by distinguishing occluded points from non-occluded points and further processing the two types of points separately. However, this technical solution needs to manually design some feature descriptors to distinguish occluded points from non-occluded points, so that the anti-occlusion performance of the solution is seriously affected by the designed feature descriptors; although the light field depth estimation method based on dynamic fusion network builds a dynamic fusion network and combines a dual-stream network and a multimodal dynamic fusion network to realize light field depth estimation, this technical solution does not consider the influence of occlusion on light field depth estimation; in addition, although the fast light field depth estimation method based on multiple parallax scales uses the extracted feature vectors to construct cost volumes of different parallax scales and realizes light field depth estimation through cost volume aggregation, this technical solution also does not consider the influence of occlusion on light field depth estimation.

[0049] Based on this, an embodiment of the present application provides an optical three-dimensional measurement method, in which, first, four cost volumes can be constructed according to the image features in the vertical, horizontal and two diagonal directions in the light field sub-aperture image array; wherein the light field sub-aperture image array is obtained by decoding the light field original image; then, the self-attention mechanism can be used to assign corresponding weight values ​​to the unobstructed view features and the obstructed view features, respectively, to obtain the corresponding feature weight matrix; next, the four cost volumes can be weighted summed and feature extracted according to the feature weight matrix, respectively, to obtain four depth images; then, the central view image of the light field original image and the four depth images can be guided filtered to obtain four filtered depth images; finally, the four filtered depth images can be cascaded to obtain the target depth image; wherein, the optical three-dimensional information of the light field original image in the target depth image. Therefore, in the embodiment of the present application, since the self-attention mechanism is used to assign corresponding weight values ​​to the unobstructed view features and the obstructed view features, respectively, more attention can be paid to the unobstructed view features, thereby improving the three-dimensional measurement accuracy by solving the occlusion problem. In addition, since guided filtering is also performed on the central perspective image and four depth images of the original light field image, the obtained depth map can be filtered with the help of learnable guided filtering to further improve the three-dimensional measurement accuracy by reducing outliers and noise in the depth map.

[0050] After introducing the design ideas of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0051] like Figure 1 FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the present application. The application scenario may include an optical three-dimensional measurement device 10 .

[0052] The optical three-dimensional measuring device 10 can be used for optical three-dimensional measurement, for example, it can be a personal computer (PC), a server and a laptop. The optical three-dimensional measuring device 10 may include one or more processors 101, a memory 102, an I / O interface 103 and a database 104. Specifically, the processor 101 may be a central processing unit (CPU) or a digital processing unit, etc. The memory 102 may be a volatile memory, such as a random-access memory (RAM); the memory 102 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 102 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 102 may be a combination of the above memories. The memory 102 may store some program instructions of the optical 3D measurement method provided in the embodiment of the present application, and these program instructions can be used to implement the steps of the optical 3D measurement method provided in the embodiment of the present application when executed by the processor 101, so as to solve the problem of low optical 3D measurement accuracy under occlusion in the prior art. The database 104 may be used to store data such as the original light field image, the light field sub-aperture image array, the feature weight matrix, the depth image, the target depth image, and the VGG-19 pre-trained model involved in the solution provided in the embodiment of the present application.

[0053] In the embodiment of the present application, the optical 3D measurement device 10 can obtain the original light field image through the I / O interface 103, and then the processor 101 of the optical 3D measurement device 10 will improve the optical 3D measurement accuracy under occlusion according to the program instructions of the optical 3D measurement method provided in the embodiment of the present application in the memory 102. In addition, data such as the original light field image, the light field sub-aperture image array, the feature weight matrix, the depth image, the target depth image, and the VGG-19 pre-trained model can also be stored in the database 104.

[0054] Of course, the method provided in the embodiment of the present application is not limited to Figure 1 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of the present application are not limited thereto. Figure 1 The functions that can be realized by each device in the application scenario shown will be described in the subsequent method embodiments, and will not be described in detail here. Below, the method of the embodiment of the present application will be introduced in conjunction with the accompanying drawings.

[0055] like Figure 2 FIG. 1 is a flow chart of an optical three-dimensional measurement method provided in an embodiment of the present application. The method can be performed by Figure 1 The method is performed by the optical three-dimensional measuring device 10, and specifically, the process of the method is described as follows.

[0056] Step 201: construct four cost volumes according to the image features in the vertical, horizontal and two diagonal directions in the light field sub-aperture image array.

[0057] In the embodiment of the present application, the light field sub-aperture image array is obtained by decoding the light field original image.

[0058] Specifically, first, the light field original image can be decoded into a light field sub-aperture image array, and the light field sub-aperture image array can be parameterized to obtain light field data LF(u,v,x,y), where (u,v) is the angle coordinate used to index each image in each image array, and (x,y) is the pixel coordinate used to index the pixel value in each image; then, the VGG-19 pre-trained model can be used to extract features from the light field sub-aperture image array to obtain the light field sub-aperture image array features, that is, Among them, H, W, and C represent the height, width, and number of feature channels of the image, respectively. Finally, the light field sub-aperture image array features can be feature offset to construct four cost volumes, that is, the features of the light field data can be obtained through the pre-trained network to obtain the light field sub-aperture image array features. By offsetting within a certain parallax range, the corresponding matching cost volume can be obtained, as shown in the following formula (1):

[0059]

[0060] Where (u0, v0) represents the central viewing angle of the light field sub-aperture image array; C d represents the matching cost volume; d represents the disparity; Concat(...,...) represents the cascade operation.

[0061] Furthermore, since the optical three-dimensional measurement method based on light field imaging is susceptible to occlusion, resulting in measurement failure at the occlusion, in order to solve the occlusion problem, in the embodiment of the present application, when the light field sub-aperture image array features are feature offset to construct four cost volumes, the four cost volumes can be constructed by feature offsetting the image features in the vertical, horizontal and two diagonal directions in the light field sub-aperture image array features, as shown in the following formula (2):

[0062]

[0063] in, Represents the cost volume constructed by view features in the horizontal direction; Represents the cost volume constructed by view features in the vertical direction; Denote the cost volumes constructed by the view features on the two diagonal lines respectively; D denotes the parallax range. Meanwhile, the dimension of the light field sub-aperture image array is 9'9.

[0064] Step 202: using the self-attention mechanism, assign corresponding weight values ​​to the unobstructed view features and the obstructed view features respectively, and obtain the corresponding feature weight matrix.

[0065] In the embodiment of the present application, with respect to the occlusion problem of light field imaging, when occlusion occurs, more weight values ​​can be assigned to the unoccluded viewing angle features in the four cost volumes to solve the occlusion problem.

[0066] Specifically, first, the self-attention mechanism can be used to determine the unobstructed view features in each cost volume; then, corresponding weight values ​​can be assigned to the unobstructed view features and the occluded view features, respectively, to obtain the corresponding feature weight matrix, wherein the unobstructed view features can be assigned more weight values, and the corresponding occluded view features can be assigned fewer weight values.

[0067] That is, the present application can use the self-attention mechanism to focus on the unoccluded view features in each cost volume, as shown in the following formula (3):

[0068]

[0069] Among them, W f ,W hrepresents the learnable weight value; Φ(·) represents the dimension transformation function; f(x), h(x) and A * Represents the feature weight matrix in four directions.

[0070] Step 203: According to the feature weight matrix, weighted summation and feature extraction are performed on the four cost volumes respectively to obtain four depth images.

[0071] Specifically, first, the four cost volumes can be weighted and summed according to the feature weight matrix to obtain the cost volume features corresponding to each of the four cost volumes, as shown in the following formula (4):

[0072]

[0073] Among them, Ψ(·) represents the dimension transformation function; α represents the learnable parameter; represents the cost volume feature after weighting by the self-attention mechanism, which can pay more attention to the unobstructed view features and thus solve the occlusion problem. Moreover, the cost volume features in the vertical, horizontal and two diagonal directions are

[0074] Then, the cost volume features corresponding to each of the four cost volumes can be extracted through convolution to obtain four depth images, as shown in the following formula (5):

[0075]

[0076] Among them, w * , b * Represent the convolution kernel and bias of the convolution respectively; Represents the weighted cost volume features in different directions; Depth * represents the depth image; furthermore, the depth image can be calculated by The weighted sum over each disparity is obtained.

[0077] Step 204: performing guided filtering on the central viewing angle image of the original light field image and the four depth images to obtain four filtered depth images.

[0078] In the embodiment of the present application, although the self-attention mechanism has been used to solve the occlusion problem, the depth image Depth * There are still some outliers in the image, resulting in low depth estimation accuracy. Therefore, in the embodiment of the present application, the depth image can also be filtered by means of "learnable guided filtering" to reduce outliers and noise in the depth image and improve the three-dimensional measurement accuracy.

[0079] Specifically, the convolution feature of the central view image of the original light field image can be used as a guide map for guided filtering to perform guided filtering on the four depth images, thereby obtaining four filtered depth images, as shown in the following formula (6):

[0080]

[0081] That is, the input of the learnable guided filter is the depth image Depth * And the central perspective image LF(4,4,x,y) of the original light field image, the output is four filtered depth images The input central view image passes through two convolutional layers to learn a guidance map feature G. Therefore, G can be used to build a local linear model to achieve guided filtering.

[0082] Step 205: cascade the four filtered depth images to obtain a target depth image.

[0083] In the embodiment of the present application, the target depth image is optical three-dimensional information of the light field original image.

[0084] Specifically, the final depth estimation result (target depth image) can be obtained by fusing the filtered depth images in four directions, as shown in the following formula (7):

[0085] Depth=ω*(Concat(Depth_guid 0 ,Depth_guid 90 ,Depth_guid 45 ,Depth_guid 135 ))+b (7)

[0086] Among them, Depth is the final target depth image; ω, b represent the convolution kernel and bias of the convolution respectively. Specific embodiment:

[0088] like Figure 3 As shown, it is a schematic diagram of the principle of optical three-dimensional measurement provided by an embodiment of the present application, such as Figure 4 As shown, it is a schematic diagram of a specific implementation process of optical three-dimensional measurement provided by an embodiment of the present application. The process can be performed by Figure 1 The method is performed by the optical three-dimensional measuring device 10, and specifically, the process of the method is described as follows.

[0089] Step 401: Decode the light field original image into a light field sub-aperture image array.

[0090] Step 402: Use the VGG-19 pre-trained model to extract features of the light field sub-aperture image array to obtain light field sub-aperture image array features.

[0091] Step 403: Use the last three layers of the VGG-19 pre-trained model to perform feature shifts on the vertical, horizontal, and two diagonal image features in the light field sub-aperture image array features to construct four cost volumes.

[0092] That is, the features of the last three layers of the VGG-19 pre-trained model are fused to construct the cost volume through feature offset. The parallax range of the feature offset is [-4, 4], and the resolution of the cost volume is half of the resolution of the original light field image, that is, H = 1 / 2H', W = 1 / 2W'.

[0093] Step 404: using the self-attention mechanism, assign corresponding weight values ​​to the unobstructed view features and the obstructed view features respectively, and obtain the corresponding feature weight matrix.

[0094] Step 405: performing weighted summation on the four cost volumes respectively according to the feature weight matrix to obtain cost volume features corresponding to the four cost volumes respectively.

[0095] Step 406: extracting features of the cost volumes corresponding to the four cost volumes respectively through convolution to obtain four depth images.

[0096] Step 407: performing guided filtering on the central viewing angle image of the original light field image and the four depth images to obtain four filtered depth images.

[0097] Among them, the resolution of the depth image after guided filtering is consistent with the resolution of the original light field image, that is, both are 512×512.

[0098] Step 408: cascade the four filtered depth images to obtain a target depth image.

[0099] Step 409: Using a preset loss function, calculate the loss value of the target depth image and the real depth image to obtain a target loss value.

[0100] In the embodiment of the present application, the real depth image is obtained based on the four cost volumes in step 403. And the L1-norm can be selected as the preset loss function, as shown in the following formula (8):

[0101] Loss=||Depth(H′, W′)-GroundTruth(H′, W′)||1 (8)

[0102] Wherein, GroundTruth(H',W') represents the real depth image; ||·|| represents the L1-norm.

[0103] Step 410: Update the network parameters of the VGG-19 pre-trained model according to the target loss value.

[0104] Among them, the network model of this application can be built using the PyTorch framework and programming language Python, and the optimization function Adam is used to optimize the entire network.

[0105] In summary, in the embodiment of the present application, since the self-attention mechanism is used to assign corresponding weight values ​​to the unobstructed view features and the obstructed view features, more attention can be paid to the unobstructed view features, thereby improving the three-dimensional measurement accuracy by solving the occlusion problem. In addition, since the central view image and the four depth images of the original light field image are also guided filtered, the depth map can be filtered by means of a learnable guided filter to further improve the three-dimensional measurement accuracy by reducing the outliers and noise in the depth map.

[0106] Based on the same inventive concept, the present application embodiment provides an optical three-dimensional measurement device 50, such as Figure 5 As shown, the optical three-dimensional measuring device 50 includes:

[0107] The cost volume construction unit 501 is used to construct four cost volumes according to the image features in the vertical, horizontal and two diagonal directions in the light field sub-aperture image array; wherein the light field sub-aperture image array is obtained by decoding the light field original image;

[0108] A weight matrix obtaining unit 502 is used to use a self-attention mechanism to assign corresponding weight values ​​to unobstructed view features and obstructed view features, respectively, to obtain a corresponding feature weight matrix;

[0109] The cost volume weighted summing unit 503 is used to perform weighted summing and feature extraction on the four cost volumes according to the feature weight matrix to obtain four depth images;

[0110] A guided filtering unit 504 is used to perform guided filtering on the central perspective image and the four depth images of the original light field image to obtain four filtered depth images;

[0111] The target depth image obtaining unit 505 is used to perform a cascade operation on the four filtered depth images to obtain a target depth image; wherein the target depth image contains optical three-dimensional information of the original light field image.

[0112] Optionally, the cost volume construction unit 501 is further configured to:

[0113] Decoding a light field raw image into a light field sub-aperture image array;

[0114] The VGG-19 pre-trained model is used to extract features of the light field sub-aperture image array and obtain the features of the light field sub-aperture image array.

[0115] The light field sub-aperture image array features are feature offset to construct four cost volumes.

[0116] Optionally, the cost volume construction unit 501 is further configured to:

[0117] The image features in the vertical, horizontal and two diagonal directions of the light field sub-aperture image array features are offset to construct four cost volumes.

[0118] Optionally, the weight matrix obtaining unit 502 is further used for:

[0119] Using the self-attention mechanism, unoccluded view features in each cost volume are determined;

[0120] Corresponding weight values ​​are assigned to the unobstructed view features and the obstructed view features respectively to obtain the corresponding feature weight matrix.

[0121] Optionally, the cost volume weighted summing unit 503 is further used for:

[0122] According to the feature weight matrix, the four cost volumes are weighted and summed respectively to obtain the cost volume features corresponding to each of the four cost volumes;

[0123] The cost volume features corresponding to each of the four cost volumes are extracted through convolution to obtain four depth images.

[0124] Optionally, the guiding filtering unit 504 is further configured to:

[0125] The convolution features of the central perspective image of the original light field image are used as the guidance map for guided filtering, and guided filtering is performed on the four depth images to obtain four filtered depth images.

[0126] Optionally, the optical three-dimensional measurement device 50 further includes a network parameter updating unit 506, which is used to:

[0127] Using a preset loss function, the loss values ​​of the target depth image and the real depth image are calculated to obtain a target loss value; wherein the real depth image is obtained based on four cost volumes;

[0128] Update the network parameters of the VGG-19 pre-trained model according to the target loss value.

[0129] The optical three-dimensional measuring device 50 can be used to perform Figure 2-Figure 4 The method is performed in the embodiment shown in FIG. 1 . Therefore, for the functions that can be realized by each functional module of the optical three-dimensional measurement device 50, reference can be made to FIG. Figure 2-Figure 4 The description of the illustrated embodiment is not repeated in detail.

[0130] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program part, which includes a program code. When the program part is run on a computer device, the program code is used to enable the computer device to execute the steps of the method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the following steps: Figure 2-Figure 4 The method performed in the illustrated embodiment.

[0131] It can be understood by those skilled in the art that all or part of the steps of the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above method embodiment are executed; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk. Alternatively, if the above integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent part, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present invention can be essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software part, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. And the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disk or optical disk.

[0132] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0133] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An optical three-dimensional measurement method, characterized in that: The method comprises: Constructing four cost volumes according to the image features in the vertical, horizontal and two diagonal directions in the light field sub-aperture image array; wherein the light field sub-aperture image array is obtained by decoding the light field original image; Using the self-attention mechanism, corresponding weight values ​​are assigned to the unoccluded view features and the occluded view features respectively to obtain the corresponding feature weight matrix; According to the feature weight matrix, weighted summation and feature extraction are performed on the four cost volumes respectively to obtain four depth images; Performing guided filtering on the central perspective image of the original light field image and the four depth images to obtain four filtered depth images; The four filtered depth images are cascaded to obtain a target depth image; wherein the target depth image contains optical three-dimensional information of the original light field image.

2. The method according to claim 1, characterized in that The step of constructing four cost volumes according to the image features in the vertical, horizontal and two diagonal directions in the light field sub-aperture image array comprises: Decoding a light field raw image into a light field sub-aperture image array; Using a VGG-19 pre-trained model, extracting features from the light field sub-aperture image array to obtain features of the light field sub-aperture image array; The light field sub-aperture image array features are subjected to feature shifting to construct the four cost volumes.

3. The method according to claim 2, characterized in that The step of performing feature shifting on the light field sub-aperture image array features to construct the four cost volumes comprises: Feature shifting is performed on the image features in the vertical, horizontal and two diagonal directions in the light field sub-aperture image array features to construct the four cost volumes.

4. The method according to claim 1, characterized in that The step of using the self-attention mechanism to assign corresponding weight values ​​to the unobstructed view features and the obstructed view features respectively to obtain the corresponding feature weight matrix includes: Using the self-attention mechanism, unoccluded view features in each cost volume are determined; Corresponding weight values ​​are assigned to the unobstructed view features and the obstructed view features respectively to obtain the corresponding feature weight matrix.

5. The method according to claim 1, characterized in that The step of performing weighted summation and feature extraction on the four cost volumes respectively according to the feature weight matrix to obtain four depth images includes: According to the feature weight matrix, weighted summation is performed on the four cost volumes respectively to obtain cost volume features corresponding to the four cost volumes respectively; The cost volume features corresponding to each of the four cost volumes are extracted through convolution to obtain the four depth images.

6. The method according to claim 1, characterized in that The step of performing guided filtering on the central perspective image of the original light field image and the four depth images to obtain four filtered depth images comprises: The convolution feature of the central viewing angle image of the original light field image is used as a guiding map for guided filtering, and guided filtering is performed on the four depth images to obtain the four filtered depth images.

7. The method according to claim 1, characterized in that After the four filtered depth images are cascaded to obtain a target depth image, the method further includes: Using a preset loss function, calculating the loss value of the target depth image and the real depth image to obtain a target loss value; wherein the real depth image is obtained based on the four cost volumes; Update the network parameters of the VGG-19 pre-trained model according to the target loss value.

8. An optical three-dimensional measuring device, characterized in that: The device comprises: A cost volume construction unit is used to construct four cost volumes according to image features in vertical, horizontal and two diagonal directions in a light field sub-aperture image array; wherein the light field sub-aperture image array is obtained by decoding a light field original image; A weight matrix obtaining unit, used to use a self-attention mechanism to assign corresponding weight values ​​to unobstructed view features and obstructed view features, respectively, to obtain a corresponding feature weight matrix; A cost volume weighted summation unit, used to perform weighted summation and feature extraction on the four cost volumes according to the feature weight matrix to obtain four depth images; A guided filtering unit, configured to perform guided filtering on the central viewing angle image of the original light field image and the four depth images to obtain four filtered depth images; The target depth image obtaining unit is used to perform a cascade operation on the four filtered depth images to obtain a target depth image; wherein the target depth image contains optical three-dimensional information of the light field original image.

9. An electronic device, characterized in that: The device comprises: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute any method according to claims 1-7 according to the obtained program instructions.

10. A storage medium, characterized in that: The storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute any one of the methods of claims 1-7.

Images