Method and device for three-dimensional reconstruction of light field based on chromatic aberration clues and neural network

Abstract

This application relates to the field of light field 3D reconstruction technology, and particularly to a method and apparatus for light field 3D reconstruction based on chromatic aberration cues and neural networks. The method includes: traversing and iteratively optimizing the number of target wavelengths and their corresponding wavelength values ​​based on a preset wavelength dataset; generating five-dimensional light field data based on the number of target wavelengths and their corresponding wavelength values; training a preset 3D reconstruction neural network model using the five-dimensional light field data to obtain a trained 3D reconstruction neural network model; traversing the preset wavelength dataset in the trained 3D reconstruction neural network model to obtain an optimized 3D reconstruction neural network model, and generating a light field 3D reconstructed image. The embodiments of this application can select the number and values ​​of wavelengths to obtain five-dimensional light field data. By simulating different chromatic aberrations and combining them with neural network algorithms to capture 3D scene information from chromatic aberration cues, more accurate light field 3D reconstruction is achieved, improving the resolution of the light field 3D reconstruction results.

Description

Technical Field

[0001] This application relates to the field of light field three-dimensional reconstruction technology, and in particular to a light field three-dimensional reconstruction method and apparatus based on chromatic aberration cues and neural networks. Background Technology

[0002] Light field 3D reconstruction has important application value in fields such as intelligent manufacturing, robot vision, autonomous driving, industrial inspection, aerial exploration and brain and neuroscience. It can reconstruct 3D scenes from observation images using 4D light field data 3D reconstruction methods.

[0003] However, in related technologies, the joint reconstruction of three-dimensional light field data rarely takes into account the joint reconstruction of multiple wavelengths. It is difficult to completely eliminate optical aberrations and chromatic aberration during the imaging process, and the real scene information contained in the chromatic aberration is not effectively utilized. This leads to a decrease in the accuracy of the three-dimensional reconstruction results, and the joint reconstruction of multiple wavelengths in the three-dimensional light field reconstruction cannot be achieved, which affects the resolution accuracy of the three-dimensional light field reconstruction and urgently needs to be solved. Summary of the Invention

[0004] This application provides a method and apparatus for three-dimensional reconstruction of light field based on chromatic aberration cues and neural networks, in order to solve the problems in related technologies, such as the fact that three-dimensional reconstruction based on four-dimensional light field data rarely considers the joint reconstruction of multiple wavelengths, and it is difficult to completely eliminate optical aberrations and chromatic aberrations during the imaging process, and the real scene information contained in the chromatic aberrations is not effectively utilized, resulting in a decrease in the accuracy of the three-dimensional reconstruction results, the inability to achieve joint reconstruction of multiple wavelengths in the three-dimensional reconstruction of light field, and the impact on the resolution accuracy of the three-dimensional reconstruction of light field.

[0005] The first aspect of this application provides a method for three-dimensional reconstruction of light field based on chromatic aberration cues and neural networks, comprising the following steps: traversing and iteratively optimizing a preset wavelength dataset to generate a target number of wavelengths and corresponding wavelength values; generating five-dimensional light field data based on the target number of wavelengths and corresponding wavelength values; training a preset three-dimensional reconstruction neural network model using the five-dimensional light field data to obtain a trained three-dimensional reconstruction neural network model; traversing the preset wavelength dataset in the trained three-dimensional reconstruction neural network model to obtain an optimized three-dimensional reconstruction neural network model; and generating a three-dimensional reconstructed light field image using the optimized three-dimensional reconstruction neural network model.

[0006] Optionally, in one embodiment of this application, the step of traversing and iteratively optimizing based on a preset wavelength dataset to generate the target number of wavelengths and the corresponding wavelength values ​​includes: traversing the number of wavelengths within a preset interval in the preset wavelength dataset, and selecting the corresponding wavelength values ​​based on the number of wavelengths; combining camera parameters, using a preset first evaluation function to iteratively optimize the number of wavelengths and the corresponding wavelength values ​​until a preset iteration stop condition is reached, thereby obtaining the target number of wavelengths and the corresponding wavelength values.

[0007] Optionally, in one embodiment of this application, training a preset three-dimensional reconstruction neural network model using the five-dimensional light field data includes: inputting the five-dimensional light field data into the preset three-dimensional reconstruction neural network model, and outputting the reconstruction result of the five-dimensional light field data; based on the reconstruction result, training the preset three-dimensional reconstruction neural network model using a preset second evaluation function to obtain the trained three-dimensional reconstruction neural network model.

[0008] Optionally, in one embodiment of this application, the step of traversing the preset wavelength dataset in the trained 3D reconstruction neural network model to obtain an optimized 3D reconstruction neural network model includes: obtaining the selection results of the number of all wavelengths and their corresponding wavelength values ​​in the preset wavelength dataset; inputting the selection results into the trained 3D reconstruction neural network model to obtain the output result of the trained 3D reconstruction neural network model; and optimizing the trained 3D reconstruction neural network model based on the output result using a preset third evaluation function until the trained 3D reconstruction neural network model meets the preset optimization conditions to obtain the optimized 3D reconstruction neural network model.

[0009] A second aspect of this application provides a three-dimensional light field reconstruction device based on chromatic aberration cues and a neural network, comprising: a generation module for iteratively optimizing a preset wavelength dataset to generate a target number of wavelengths and corresponding wavelength values; a training module for generating five-dimensional light field data based on the target number of wavelengths and corresponding wavelength values, and training a preset three-dimensional reconstruction neural network model using the five-dimensional light field data to obtain a trained three-dimensional reconstruction neural network model; and a reconstruction module for iterating through the preset wavelength dataset in the trained three-dimensional reconstruction neural network model to obtain an optimized three-dimensional reconstruction neural network model, and generating a three-dimensional reconstructed light field image using the optimized three-dimensional reconstruction neural network model.

[0010] Optionally, in one embodiment of this application, the generation module includes: a selection unit, configured to traverse the number of wavelengths within a preset interval in the preset wavelength dataset and select the corresponding wavelength value based on the number of wavelengths; and an iteration unit, configured to combine camera parameters and use a preset first evaluation function to iteratively optimize the number of wavelengths and the corresponding wavelength value until a preset iteration stop condition is reached, thereby obtaining the target number of wavelengths and the corresponding wavelength value.

[0011] Optionally, in one embodiment of this application, the training module includes: an output unit, configured to input the five-dimensional light field data into the preset three-dimensional reconstruction neural network model and output the reconstruction result of the five-dimensional light field data; and a training unit, configured to train the preset three-dimensional reconstruction neural network model based on the reconstruction result using a preset second evaluation function to obtain the trained three-dimensional reconstruction neural network model.

[0012] Optionally, in one embodiment of this application, the reconstruction module includes: an acquisition unit, configured to acquire the selection results of the number of all wavelengths and their corresponding wavelength values ​​in the preset wavelength dataset, input the selection results into the trained three-dimensional reconstruction neural network model, and obtain the output result of the trained three-dimensional reconstruction neural network model; and an optimization unit, configured to optimize the trained three-dimensional reconstruction neural network model based on the output result using a preset third evaluation function until the trained three-dimensional reconstruction neural network model meets the preset optimization conditions, and obtain the optimized three-dimensional reconstruction neural network model.

[0013] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the light field three-dimensional reconstruction method based on chromatic aberration cues and neural networks as described in the above embodiments.

[0014] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for three-dimensional reconstruction of light fields based on chromatic aberration cues and neural networks.

[0015] This application embodiment allows for the selection and value of a number of wavelengths to obtain five-dimensional light field data. By simulating different chromatic aberrations and combining this with a neural network algorithm to extract three-dimensional scene information from chromatic aberration cues, more accurate three-dimensional reconstruction of the light field is achieved, improving the resolution of the three-dimensional light field reconstruction results. This solves the problems in related technologies, such as the inability to fully eliminate optical aberrations and chromatic aberrations during imaging due to the limited consideration of multi-wavelength joint reconstruction in three-dimensional light field data, the failure to effectively utilize the real scene information contained in the chromatic aberrations, resulting in decreased accuracy of the three-dimensional reconstruction results, and the inability to achieve multi-wavelength joint reconstruction of the light field, thus affecting the resolution accuracy of the three-dimensional light field reconstruction.

[0016] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0017] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0018] Figure 1 This is a flowchart of a light field 3D reconstruction method based on chromatic aberration cues and neural networks provided according to an embodiment of this application;

[0019] Figure 2 This is a schematic diagram illustrating the iterative initialization principle of the number and value of wavelengths in one embodiment of this application;

[0020] Figure 3 This is a schematic diagram illustrating the training principle of a three-dimensional reconstruction deep neural network according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram illustrating the joint optimization principle of a three-dimensional reconstruction deep neural network according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram illustrating the principle of light field 3D reconstruction based on chromatic aberration cues and neural networks according to an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of the structure of a light field three-dimensional reconstruction device based on color difference cues and neural networks according to an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0025] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0026] The following description, with reference to the accompanying drawings, outlines a method and apparatus for three-dimensional light field reconstruction based on chromatic aberration cues and neural networks, according to embodiments of this application. Addressing the issues raised in the background section regarding the limitations of four-dimensional light field data-based three-dimensional reconstruction, which often fails to adequately address multi-wavelength joint reconstruction, resulting in reduced accuracy and hindering the effective utilization of real-scene information contained within chromatic aberration, thus impacting resolution accuracy, this application provides a method for three-dimensional light field reconstruction based on chromatic aberration cues and neural networks. This method allows for the selection and value of wavelengths to obtain five-dimensional light field data. By simulating different chromatic aberrations and combining this with neural network algorithms to extract three-dimensional scene information from chromatic aberration cues, more accurate three-dimensional light field reconstruction is achieved, improving the resolution of the reconstruction results. This solves the problems in related technologies, such as the lack of consideration for multi-wavelength joint reconstruction in 3D reconstruction based on 4D light field data, the difficulty in completely eliminating optical aberrations and chromatic aberration during imaging, the failure to effectively utilize the real scene information contained in chromatic aberration, the resulting decrease in the accuracy of 3D reconstruction results, the inability to achieve multi-wavelength joint reconstruction of light field 3D reconstruction, and the impact on the resolution accuracy of light field 3D reconstruction.

[0027] Specifically, Figure 1 This is a flowchart illustrating a three-dimensional light field reconstruction method based on chromatic aberration cues and neural networks, provided in an embodiment of this application.

[0028] like Figure 1 As shown, the light field 3D reconstruction method based on chromatic aberration cues and neural networks includes the following steps:

[0029] In step S101, the target number of wavelengths and their corresponding wavelength values ​​are generated by traversing and iteratively optimizing the preset wavelength dataset.

[0030] It should be noted that the preset wavelength dataset can be set by those skilled in the art according to the actual situation, and no specific limitation is made here.

[0031] It is understood that, in the embodiments of this application, wavelength data obtained from the same camera can be used as a preset wavelength dataset. The target wavelength number and corresponding wavelength value are selected in the preset wavelength dataset, and the process is traversed and iteratively optimized to select the optimal number of wavelengths. The number of wavelengths corresponds to specific wavelength values, that is, the total number of wavelengths is N, corresponding to N wavelength values.

[0032] Optionally, in one embodiment of this application, generating the target number of wavelengths and the corresponding wavelength values ​​by traversing and iteratively optimizing a preset wavelength dataset includes: traversing the number of wavelengths within a preset interval in the preset wavelength dataset, and selecting the corresponding wavelength values ​​based on the number of wavelengths; combining camera parameters, using a preset first evaluation function to iteratively optimize the number of wavelengths and the corresponding wavelength values ​​until a preset iteration stop condition is reached, thereby obtaining the target number of wavelengths and the corresponding wavelength values.

[0033] It should be noted that the preset interval, preset first evaluation function, and preset iteration stopping condition can be set by those skilled in the art according to the actual situation, and are not specifically limited here.

[0034] In actual implementation, such as Figure 2 The diagram illustrates the iterative initialization principle of the number and value of wavelengths according to an embodiment of this application. A random integer generator can be used to generate the total number of wavelengths N, selecting an integer within a preset range [2, 8]. Then, selectable wavelength values ​​are determined within the visible and near-infrared bands. N selectable wavelengths are selected from a set of discrete wavelength permutations and combinations, for example, sequentially according to permutations and combinations. Furthermore, combining the selected number of wavelengths, wavelength values, and other camera parameters, the data is input into a light field imaging generation model to generate 5D light field data of size H×W×U×V×N, where H and W represent the height and width of the light field image, respectively, U and V represent the number of viewing angles in two vertical directions, and N is the number of wavelengths.

[0035] The light field imaging generation model used can be simulated and the RL (Richardson-Lucy) deconvolution reconstruction algorithm is used to reconstruct the five-dimensional light field data. By fusing the RL reconstruction results at different wavelengths, a preliminary reconstruction result is obtained. The loss function is used as the preset first evaluation function and compared with the real scene value (true value) to obtain evaluation results such as signal-to-noise ratio (SNR) and similarity metric (SSIM). The number of wavelengths (iteration 2) and the specific wavelength size (iteration 1) are iteratively optimized until the preset iteration stopping condition is reached, and the target number of wavelengths and the corresponding wavelength values ​​are obtained.

[0036] In step S102, five-dimensional light field data is generated based on the number of target wavelengths and their corresponding wavelength values. The five-dimensional light field data is then used to train a preset three-dimensional reconstruction neural network model to obtain the trained three-dimensional reconstruction neural network model.

[0037] It should be noted that the preset 3D reconstruction neural network model can be set by those skilled in the art according to the actual situation, and no specific limitations are made here.

[0038] It is understood that, in the embodiments of this application, a five-dimensional light field image can be simulated based on the number of target wavelengths and the corresponding wavelength values ​​obtained in the above steps. Based on the obtained five-dimensional light field image data, a preset three-dimensional reconstruction neural network model can be trained to obtain a deep neural network that can obtain high-precision three-dimensional reconstruction results from light field data of different wavelengths. The trained deep neural network can utilize the imaging features of specific wavelengths to extract chromatic aberration cues in order to improve the performance of light field three-dimensional reconstruction.

[0039] Optionally, in one embodiment of this application, training a preset three-dimensional reconstruction neural network model using five-dimensional light field data includes: inputting five-dimensional light field data into the preset three-dimensional reconstruction neural network model and outputting the reconstruction result of the five-dimensional light field data; based on the reconstruction result, training the preset three-dimensional reconstruction neural network model using a preset second evaluation function to obtain the trained three-dimensional reconstruction neural network model.

[0040] It should be noted that the preset second evaluation function can be set by those skilled in the art according to the actual situation, and no specific limitation is made here.

[0041] In actual implementation, such as Figure 3 The diagram shown is a schematic diagram of the training principle of a three-dimensional reconstruction deep neural network according to an embodiment of this application. By combining the selected number of wavelengths and wavelength values ​​with other camera parameters, the data is input into a light field imaging generation model to generate 5D light field data. This data is then input into a preset three-dimensional reconstruction neural network model to obtain the reconstruction result of the 5D light field data. The evaluation result is obtained by comparing the data with the real scene value (true value) through a preset second evaluation function. The model is then modified using the evaluation result to obtain the trained three-dimensional reconstruction neural network model.

[0042] In step S103, the preset wavelength dataset is traversed in the trained 3D reconstruction neural network model to obtain the optimized 3D reconstruction neural network model, and the optimized 3D reconstruction neural network model is used to generate a light field 3D reconstruction image.

[0043] It is understood that, in this embodiment of the application, the trained 3D reconstruction neural network model obtained in the above steps can be used to select the number and value of wavelengths again from the preset wavelength dataset to jointly optimize the 3D reconstruction deep neural network. Since the trained 3D reconstruction neural network model is a deep neural network algorithm capable of extracting effective features from 5D light field data for 3D reconstruction, but is limited to fixed wavelength values, a joint optimization method can be used to fine-tune the 3D reconstruction deep neural network in different wavelength types. This further improves the stability and accuracy of 3D reconstruction, enhances the robustness of the 5D light field 3D reconstruction deep neural network, obtains a feature model capable of extracting chromatic aberration more deeply, and uses the obtained optimized 3D reconstruction neural network model to process 5D light field data to obtain a more accurate 3D reconstructed light field image.

[0044] Optionally, in one embodiment of this application, traversing a preset wavelength dataset in the trained 3D reconstruction neural network model to obtain an optimized 3D reconstruction neural network model includes: obtaining the selection results of the number of all wavelengths and their corresponding wavelength values ​​in the preset wavelength dataset; inputting the selection results into the trained 3D reconstruction neural network model to obtain the output results of the trained 3D reconstruction neural network model; and optimizing the trained 3D reconstruction neural network model based on the output results using a preset third evaluation function until the trained 3D reconstruction neural network model meets the preset optimization conditions to obtain the optimized 3D reconstruction neural network model.

[0045] It should be noted that the preset third evaluation function and preset optimization conditions can be set by those skilled in the art according to the actual situation, and no specific limitations are made here.

[0046] In actual implementation, such as Figure 4 The diagram illustrates the joint optimization principle of a 3D reconstruction deep neural network according to an embodiment of this application. A random integer generator can generate the total number of wavelengths and select an integer within a preset range. Then, selectable wavelength values ​​are determined within the visible and near-infrared bands. Selectable wavelengths corresponding to the number of wavelengths are chosen from a set of discrete wavelength permutations and combinations, selected sequentially according to the permutations and combinations. The selected number of wavelengths, wavelength values, and other camera parameters are then combined with the input light field imaging generation model and 5D light field data. This data is then input into a preset 3D reconstruction neural network model to obtain the reconstruction result of the 5D light field data. The result is then compared with the real scene value (true value) through a preset third evaluation function. Optimization training is performed on the number of wavelengths and specific wavelength values ​​within the preset wavelength dataset until the preset optimization conditions are met, resulting in an optimized 3D reconstruction neural network model.

[0047] like Figure 5As shown below, the working content of the embodiment of this application will be described in detail with a specific example. Figure 5 This is a schematic diagram illustrating the principle of 3D light field reconstruction based on chromatic aberration cues and neural networks according to an embodiment of this application. In stage 1, the number and values ​​of selected wavelengths are iteratively initialized. Adding a new wavelength dimension expands the four-dimensional light field data to five dimensions, increasing the amount of light field information acquired and thus improving the accuracy of 3D light field reconstruction. Stage 2 involves training and optimizing the 3D reconstruction deep neural network after selecting the number and values ​​of wavelengths. An initial deep neural network is trained to achieve 5D light field 3D reconstruction. Stage 3 involves joint optimization of the selected number and values ​​of wavelengths and the 3D reconstruction deep neural network to optimize a more robust and accurate 3D reconstruction neural network. By simulating light field images under various wavelengths and evaluating reconstruction effects multiple times, a 3D light field reconstruction model that can further improve reconstruction accuracy is obtained for 3D light field reconstruction.

[0048] The light field 3D reconstruction method based on chromatic aberration cues and neural networks proposed in this application can select the number and values ​​of wavelengths to obtain five-dimensional light field data. By simulating different chromatic aberrations and combining them with neural network algorithms to capture 3D scene information from chromatic aberration cues, more accurate light field 3D reconstruction is achieved, improving the resolution of the light field 3D reconstruction results. This solves the problems in related technologies, such as the lack of consideration for multi-wavelength joint reconstruction in 3D reconstruction based on four-dimensional light field data, the difficulty in completely eliminating optical aberrations and chromatic aberrations during imaging, the failure to effectively utilize the real scene information contained in chromatic aberrations, leading to decreased accuracy of 3D reconstruction results, and the inability to achieve multi-wavelength joint reconstruction of light field 3D reconstruction, thus affecting the resolution accuracy of light field 3D reconstruction.

[0049] Next, referring to the accompanying drawings, a three-dimensional light field reconstruction device based on color difference cues and neural networks is described according to an embodiment of this application.

[0050] Figure 6 This is a schematic diagram of the structure of a light field three-dimensional reconstruction device based on color difference cues and neural networks according to an embodiment of this application.

[0051] like Figure 6 As shown, the light field 3D reconstruction device 10 based on chromatic aberration cues and neural networks includes: a generation module 100, a training module 200, and a reconstruction module 300.

[0052] The generation module 100 is used to traverse and iteratively optimize based on a preset wavelength dataset to generate the target number of wavelengths and their corresponding wavelength values.

[0053] The training module 200 is used to generate five-dimensional light field data based on the number of target wavelengths and their corresponding wavelength values, and to train a preset three-dimensional reconstruction neural network model using the five-dimensional light field data to obtain the trained three-dimensional reconstruction neural network model.

[0054] The reconstruction module 300 is used to traverse the preset wavelength dataset in the trained 3D reconstruction neural network model to obtain the optimized 3D reconstruction neural network model, and to generate a light field 3D reconstruction image using the optimized 3D reconstruction neural network model.

[0055] Optionally, in one embodiment of this application, the generation module 100 includes a selection unit and an iteration unit.

[0056] The selection unit is used to traverse the number of wavelengths within a preset interval in the preset wavelength dataset and select the corresponding wavelength value based on the number of wavelengths.

[0057] The iterative unit is used to combine camera parameters and use a preset first evaluation function to iteratively optimize the number of wavelengths and their corresponding wavelength values ​​until a preset iteration stop condition is reached, thereby obtaining the target number of wavelengths and their corresponding wavelength values.

[0058] Optionally, in one embodiment of this application, the training module 200 includes an output unit and a training unit.

[0059] The output unit is used to input five-dimensional light field data into a preset three-dimensional reconstruction neural network model and output the reconstruction result of the five-dimensional light field data.

[0060] The training unit is used to train a preset 3D reconstruction neural network model based on the reconstruction results using a preset second evaluation function, so as to obtain the trained 3D reconstruction neural network model.

[0061] Optionally, in one embodiment of this application, the reconstruction module 300 includes an acquisition unit and an optimization unit.

[0062] The acquisition unit is used to acquire the selection results of the number of wavelengths and the corresponding wavelength values ​​in the preset wavelength dataset, input the selection results into the trained 3D reconstruction neural network model, and obtain the output results of the trained 3D reconstruction neural network model.

[0063] The optimization unit is used to optimize the trained 3D reconstruction neural network model based on the output results using a preset third evaluation function until the trained 3D reconstruction neural network model meets the preset optimization conditions, thus obtaining the optimized 3D reconstruction neural network model.

[0064] It should be noted that the foregoing explanation of the embodiment of the light field three-dimensional reconstruction method based on color difference cues and neural networks also applies to the light field three-dimensional reconstruction device based on color difference cues and neural networks in this embodiment, and will not be repeated here.

[0065] The light field 3D reconstruction device based on chromatic aberration cues and neural networks proposed in this application can select the number and values ​​of wavelengths to obtain five-dimensional light field data. By simulating different chromatic aberrations and combining them with neural network algorithms to capture 3D scene information from chromatic aberration cues, more accurate light field 3D reconstruction is achieved, improving the resolution of the light field 3D reconstruction results. This solves the problems in related technologies, such as the inability to fully eliminate optical aberrations and chromatic aberrations during imaging due to the lack of consideration for multi-wavelength joint reconstruction in 3D reconstruction based on four-dimensional light field data, the failure to effectively utilize the real scene information contained in the chromatic aberrations, resulting in decreased accuracy of the 3D reconstruction results, and the inability to achieve multi-wavelength joint reconstruction of the light field 3D reconstruction, thus affecting the resolution accuracy of the light field 3D reconstruction.

[0066] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0067] The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0068] When the processor 702 executes the program, it implements the light field three-dimensional reconstruction method based on color difference cues and neural networks provided in the above embodiments.

[0069] Furthermore, electronic devices also include:

[0070] Communication interface 703 is used for communication between memory 701 and processor 702.

[0071] The memory 701 is used to store computer programs that can run on the processor 702.

[0072] The memory 701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0073] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0074] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0075] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0076] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described light field 3D reconstruction method based on chromatic aberration cues and neural networks.

[0077] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0078] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0079] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0080] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0081] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0082] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0083] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0084] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for light field 3D reconstruction based on chromatic aberration cues and neural network, characterized in that, Includes the following steps: Based on the preset wavelength dataset, the target number of wavelengths and their corresponding wavelength values ​​are generated by traversing and iteratively optimizing the dataset. Five-dimensional light field data is generated based on the number of target wavelengths and the corresponding wavelength values. The five-dimensional light field data is then used to train a preset three-dimensional reconstruction neural network model to obtain the trained three-dimensional reconstruction neural network model. The preset wavelength dataset is traversed in the trained 3D reconstruction neural network model to obtain an optimized 3D reconstruction neural network model, and the optimized 3D reconstruction neural network model is used to generate a light field 3D reconstruction image.

2. The method according to claim 1, characterized in that, The step of traversing and iteratively optimizing a preset wavelength dataset to generate the target number of wavelengths and their corresponding wavelength values ​​includes: The number of wavelengths within a preset interval is traversed in the preset wavelength dataset, and the corresponding wavelength value is selected based on the number of wavelengths. By combining camera parameters, the number of wavelengths and their corresponding wavelength values ​​are iteratively optimized using a preset first evaluation function until a preset iteration stop condition is reached, thereby obtaining the target number of wavelengths and their corresponding wavelength values.

3. The method according to claim 1, characterized in that, The step of training a preset three-dimensional reconstruction neural network model using the five-dimensional light field data includes: The five-dimensional light field data is input into the preset three-dimensional reconstruction neural network model, and the reconstruction result of the five-dimensional light field data is output. Based on the reconstruction results, the preset three-dimensional reconstruction neural network model is trained using a preset second evaluation function to obtain the trained three-dimensional reconstruction neural network model.

4. The method according to claim 1, characterized in that, The step of traversing the preset wavelength dataset in the trained 3D reconstruction neural network model to obtain an optimized 3D reconstruction neural network model includes: Obtain the selection results of the number of all wavelengths and their corresponding wavelength values ​​in the preset wavelength dataset, input the selection results into the trained three-dimensional reconstruction neural network model, and obtain the output results of the trained three-dimensional reconstruction neural network model; Based on the output results, the trained 3D reconstruction neural network model is optimized using a preset third evaluation function until the trained 3D reconstruction neural network model meets the preset optimization conditions, thus obtaining the optimized 3D reconstruction neural network model.

5. A three-dimensional light field reconstruction device based on chromatic aberration cues and neural networks, characterized in that, include: The generation module is used to traverse and iteratively optimize based on a preset wavelength dataset to generate the target number of wavelengths and their corresponding wavelength values. The training module is used to generate five-dimensional light field data based on the number of target wavelengths and the corresponding wavelength values, and to train a preset three-dimensional reconstruction neural network model using the five-dimensional light field data to obtain the trained three-dimensional reconstruction neural network model. The reconstruction module is used to traverse the preset wavelength dataset in the trained 3D reconstruction neural network model to obtain an optimized 3D reconstruction neural network model, and to generate a light field 3D reconstruction image using the optimized 3D reconstruction neural network model.

6. The apparatus according to claim 5, characterized in that, The generation module includes: The selection unit is used to traverse the number of wavelengths within a preset interval in the preset wavelength dataset and select the corresponding wavelength value based on the number of wavelengths. An iterative unit is used to combine camera parameters and use a preset first evaluation function to iteratively optimize the number of wavelengths and the corresponding wavelength values ​​until a preset iteration stop condition is reached, thereby obtaining the target number of wavelengths and the corresponding wavelength values.

7. The apparatus according to claim 5, characterized in that, The training module includes: The output unit is used to input the five-dimensional light field data into the preset three-dimensional reconstruction neural network model and output the reconstruction result of the five-dimensional light field data. The training unit is used to train the preset 3D reconstruction neural network model based on the reconstruction results using a preset second evaluation function, so as to obtain the trained 3D reconstruction neural network model.

8. The apparatus according to claim 5, characterized in that, The reconstruction module includes: The acquisition unit is used to acquire the selection results of the number of all wavelengths and the corresponding wavelength values ​​in the preset wavelength dataset, input the selection results into the trained three-dimensional reconstruction neural network model, and obtain the output results of the trained three-dimensional reconstruction neural network model. An optimization unit is used to optimize the trained 3D reconstruction neural network model based on the output result using a preset third evaluation function until the trained 3D reconstruction neural network model meets the preset optimization conditions, thereby obtaining the optimized 3D reconstruction neural network model.

9. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the light field three-dimensional reconstruction method based on chromatic aberration cues and neural networks as described in any one of claims 1-4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the light field 3D reconstruction method based on chromatic aberration cues and neural networks as described in any one of claims 1-4.

Images