Light field imaging method and device, medium, imaging unit and imaging system

Abstract

The invention relates to the technical field of optical imaging, and discloses a light field imaging method and device, a medium, an imaging unit and an imaging system.The method comprises the steps that a first sub-aperture image collected by a first imaging unit at the current moment is obtained; generating first exchange information, transmitting the first exchange information to each second imaging unit, and acquiring second exchange information transmitted by each second imaging unit; the exchange information comprises a sub-aperture image and a reconstructed light field; performing environment light field reconstruction according to the first sub-aperture image, each second sub-aperture image and each second reconstruction light field at the previous moment, and generating a first reconstruction light field at the current moment; and under the condition that the preset imaging target is not completed, performing environment light field reconstruction again at the next moment until the imaging target is completed. According to the method, a centralized light field data processing mode is not needed, the requirements for system computing power and data bandwidth are reduced, and processing is more flexible.

Description

Technical Field

[0001] The present invention relates to the field of optical imaging technology, and particularly relates to a light field imaging method, device, medium, imaging unit and imaging system. Background Art

[0002] The light field refers to the distribution of a certain physical quantity of light in space, which is used to describe the intensity of light rays in any direction from any point in space. As a classic means of light field imaging, the array light field imaging technology has been widely used in the imaging systems of smart phones.

[0003] However, the existing array light field imaging system also adopts a centralized light field reconstruction algorithm, which requires prior calibration of the relative relationship between all sub-apertures and acquisition of images of all sub-apertures to perform complete light field reconstruction, posing very high requirements on the computing power, bandwidth, etc. of the computing system. Summary of the Invention

[0004] In view of this, the present invention provides a light field imaging method, device, medium, imaging unit and imaging system to solve the problem of high requirements for existing light field imaging.

[0005] In a first aspect, the present invention provides a light field imaging method, which is applied to a first imaging unit; the first imaging unit is an imaging unit in an array light field imaging system, and there is at least one second imaging unit adjacent to the first imaging unit in the array light field imaging system, and the first imaging unit is communicatively connected to each of the second imaging units;

[0006] The method includes:

[0007] Obtain a first sub-aperture image collected by the first imaging unit at the current moment;

[0008] Generate first exchange information, transmit the first exchange information to each of the second imaging units, and obtain second exchange information transmitted by each of the second imaging units; the first exchange information includes the first sub-aperture image and the first reconstructed light field of the first imaging unit at the previous moment, and the second exchange information includes the second sub-aperture image collected by the second imaging unit at the current moment and the second reconstructed light field of the second imaging unit reconstructed at the previous moment; the first exchange information is used to instruct the second imaging unit to execute the light field imaging method;

[0009] Perform environmental light field reconstruction according to the first sub-aperture image, each of the second sub-aperture images and each of the second reconstructed light fields at the previous moment to generate a first reconstructed light field at the current moment;

[0010] In the case where the preset imaging target is not completed, the ambient light field reconstruction is performed again at the next moment until the imaging target is completed.

[0011] In some alternative embodiments, the reconstructing the ambient light field based on the first sub-aperture image, each of the second sub-aperture images, and each of the second reconstructed light fields at the previous moment to generate the first reconstructed light field at the current moment includes:

[0012] Inputting the first sub-aperture image and each of the second sub-aperture images into a preset light field reconstruction neural network to obtain a first undetermined light field at the current moment;

[0013] Performing ambient light field registration and fusion based on the first undetermined light field at the current moment, the first reconstructed light field at the previous moment, and each of the second reconstructed light fields at the previous moment to generate the first reconstructed light field at the current moment.

[0014] In some alternative embodiments, the performing ambient light field registration and fusion based on the first undetermined light field at the current moment, the first reconstructed light field at the previous moment, and each of the second reconstructed light fields at the previous moment to generate the first reconstructed light field at the current moment includes:

[0015] Performing ambient light field point cloud registration based on the first undetermined light field at the current moment, the first reconstructed light field at the previous moment, and each of the second reconstructed light fields at the previous moment to determine the first pose of the first imaging unit at the current moment;

[0016] Overlaying the registered point clouds together to generate a fused point cloud;

[0017] Performing point cloud representation optimization on the first undetermined light field at the current moment according to the first pose at the current moment and the fused point cloud to obtain the first reconstructed light field at the current moment.

[0018] In some alternative embodiments, the objective function of the ambient light field point cloud registration is:

[0019]

[0020] where i and j are indices of the imaging units, and k is the index of the point cloud; represents the k-th point cloud data in the determined ambient light field when the i-th imaging unit is the first imaging unit and the j-th imaging unit is the second imaging unit, represents the k-th point cloud data of the confidence; represents the k-th point cloud data in the determined ambient light field when the j-th imaging unit is the first imaging unit and the i-th imaging unit is the second imaging unit, represents the k-th point cloud data Confidence;

[0021] s is a scaling parameter, and P is a rigid body transformation parameter; s j→i represents the scaling parameter from the point cloud of the j-th imaging unit to the point cloud of the i-th imaging unit, P j→i represents the rigid body transformation parameter from the point cloud of the j-th imaging unit to the point cloud of the i-th imaging unit; s i→j represents the scaling parameter from the point cloud of the i-th imaging unit to the point cloud of the j-th imaging unit, P i→j represents the rigid body transformation parameter from the point cloud of the i-th imaging unit to the point cloud of the j-th imaging unit; the rigid body transformation parameter P and the scaling parameter s are used to determine the first pose of the first imaging unit at the current moment.

[0022] In some alternative embodiments, the first exchange information further includes the first pose of the first imaging unit at the previous moment, and the second exchange information further includes the second pose of the second imaging unit at the previous moment;

[0023] The method further includes:

[0024] Inputting the first reconstructed light field at the current moment, the first pose determined at the current moment, and each second pose at the previous moment into a preset shape feedback neural network to obtain the first planned pose at the next moment;

[0025] Determining the deformation parameter of the first imaging unit according to the first planned pose at the next moment;

[0026] Adjusting the pose of the first imaging unit according to the deformation parameter.

[0027] In some alternative embodiments, the determining the deformation parameter of the first imaging unit according to the first planned pose at the next moment includes:

[0028] Obtaining the second planned pose of each second imaging unit at the next moment;

[0029] Performing joint optimization according to the first planned pose at the next moment and each second planned pose at the next moment to determine the deformation parameter of the first imaging unit.

[0030] In a second aspect, the present invention provides a light field imaging device applied to a first imaging unit; the first imaging unit is an imaging unit in an array light field imaging system, and there is at least one second imaging unit adjacent to the first imaging unit in the array light field imaging system, and the first imaging unit is communicatively connected to each second imaging unit; the device includes:

[0031] An image acquisition module, configured to acquire a first sub-aperture image captured by the first imaging unit at the current moment;

[0032] A switching module, configured to generate first switching information, transmit the first switching information to each of the second imaging units, and acquire second switching information transmitted by each of the second imaging units; the first switching information includes the first sub-aperture image and the first reconstructed light field of the first imaging unit at the previous moment, and the second switching information includes the second sub-aperture image captured by the second imaging unit at the current moment and the second reconstructed light field of the second imaging unit reconstructed at the previous moment; the first switching information is used to instruct the second imaging unit to execute the light field imaging method;

[0033] A light field reconstruction module, configured to perform environmental light field reconstruction based on the first sub-aperture image, each of the second sub-aperture images, and each of the second reconstructed light fields at the previous moment, and generate a first reconstructed light field at the current moment;

[0034] A loop module, configured to perform environmental light field reconstruction again at the next moment until the preset imaging target is completed when the preset imaging target has not been completed.

[0035] In a third aspect, the present invention provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to cause a computer to execute the light field imaging method according to the first aspect or any corresponding embodiment thereof.

[0036] In a fourth aspect, the present invention provides an imaging unit, including: a memory and a processor, which are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to execute the light field imaging method according to the first aspect or any corresponding embodiment thereof.

[0037] In a fifth aspect, the present invention provides an array light field imaging system, including a plurality of imaging units according to the fourth aspect or any corresponding embodiment thereof;

[0038] The plurality of imaging units are arranged in an array, and each of the imaging units is communicatively connected to other adjacent imaging units.

[0039] In the present invention, information is transmitted between the imaging unit in the array light field imaging system and other adjacent imaging units, enabling each imaging unit to perform intelligent calculations independently without the need for a centralized light field data processing method. This not only reduces the requirements for system computing power and data bandwidth but also allows the imaging unit to reconstruct the light field without relying on the strict calibration between light field sub-apertures, making the processing more flexible. Moreover, the image of each light field sub-aperture can continuously transmit information and interact with the images of adjacent apertures to reconstruct the complete ambient light field information. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the related art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0041] Figure 1 is a schematic diagram of a 4D model of the light field;

[0042] Figure 2 is a schematic diagram of the architecture of an array light field imaging system according to an embodiment of the present invention;

[0043] Figure 3 is a schematic flowchart of a light field imaging method according to an embodiment of the present invention;

[0044] Figure 4 is a schematic diagram of the principle of a light field reconstruction neural network according to an embodiment of the present invention;

[0045] Figure 5 is a schematic diagram of the process of information transmission between each imaging unit according to an embodiment of the present invention;

[0046] Figure 6 is a schematic flowchart of generating a first reconstructed light field according to an embodiment of the present invention;

[0047] Figure 7 is a schematic diagram of the effect of registering and fusing two reconstructed light fields according to an embodiment of the present invention;

[0048] Figure 8 is a schematic diagram of the information transmission process between adjacent imaging units according to an embodiment of the present invention;

[0049] Figure 9 is a schematic diagram of the process of determining deformation parameters according to an embodiment of the present invention;

[0050] Figure 10Schematic diagram of the architectures of two light field imaging systems according to embodiments of the present invention;

[0051] Figure 11 Block diagram of the structure of a light field imaging device according to embodiments of the present invention;

[0052] Figure 12 Schematic diagram of the hardware structure of an imaging unit according to embodiments of the present invention. Detailed implementation manners

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0054] The light field, as the name implies, is the distribution of a certain physical quantity of light in space. This concept was first clearly proposed in 1939, and was gradually improved later, and the form of the plenoptic function was given. Simply put, the light field describes the intensity of the light rays in any direction from any point in space, and the complete plenoptic function that describes the light field is a 7-dimensional function, including the position (x, y, z) of any point, any direction (Θ, Φ in polar coordinates), wavelength (λ), and time (t). In practical applications, the information of color (related to wavelength) and time dimension is usually represented by RGB channels and different frames. Therefore, in terms of the light field, only the direction and position of the light rays need to be concerned, so it is reduced from 7 dimensions to 5 dimensions.

[0055] The plenoptic function can be simplified and dimensionally reduced, and it can be simplified to a 4D light field model of (u, v, s, t). Specifically, two non-coplanar planes (u, v) and (s, t) are assumed. If a light ray has an intersection with each of these two planes, then the light ray can be uniquely represented by these two intersections. This 4D light field model has an important prerequisite assumption: the light rays collected at any position along the propagation direction of the light ray are the same. In other words, it is assumed that the light intensity does not decay and the wavelength remains unchanged during the propagation of any light ray. Considering that the propagation distance of the light ray from the scene surface to the human eye in daily life is very limited and the attenuation of the light ray in the air is negligible, the above assumption is completely reasonable.

[0056] This 4D light field model cannot completely describe all the light rays in the three-dimensional space. Among them, the light rays parallel to the (u, v) or (s, t) plane cannot be represented by this 4D model. For example Figure 1The light marked in red cannot be represented by the 4D model. Although the 4D model cannot completely describe all the light in three-dimensional space, it can completely describe the light received by the human eye. Because when the light is perpendicular to the forward viewing direction of the human eye, the light will not enter the human eye. Therefore, this part of the light does not affect the visual imaging of the human eye. Since the 4D model not only reduces the dimension required to represent the light field but also can completely represent all the light required for human eye imaging, the 4D light field model has been widely recognized in the academic community, and a large number of studies on the light field have been carried out on this basis.

[0057] With the development and breakthrough of artificial intelligence technologies represented by deep learning, the light field imaging technology has made great breakthroughs in imaging dimension, scale, performance, and robustness. The array sensor light field imaging (hereinafter referred to as "array light field imaging") technology, as a classic means of light field imaging, has been widely used in the imaging systems of smart phones and has shown great application potential in fields such as scientific observation, smart city, public safety, and unmanned systems.

[0058] Array light field imaging is a technology that uses an array sensor to capture the three-dimensional object scene. It combines the advantages of light field imaging and array imaging and can obtain complete three-dimensional information in a single fast shot. The array light field imaging system is composed of multiple cameras forming a fixed-shaped camera array, which can simultaneously collect spatial information and viewing angle information, realizing the efficient acquisition of three-dimensional information.

[0059] However, the existing rigid array light field imaging system is developed based on the 4D light field model proposed by Professor Marc Levoy, and adopts a fixed planar (or spherical) array structure design, and the mutual relationship between the sub-aperture sensors of the light field is fixed. When the aperture and baseline of the light field imaging system increase, the volume of the imaging system will also increase accordingly, seriously affecting the flexibility of the imaging system. Correspondingly, the existing array light field imaging system also adopts a centralized light field reconstruction algorithm, which requires prior calibration of the relative relationship between all sub-apertures and acquisition of images of all sub-apertures to perform complete light field reconstruction, posing very high requirements on the computing power, bandwidth, etc. of the computing system.

[0060] The light field imaging method provided by the embodiments of the present invention enables information transmission between the imaging units in the array light field imaging system and other adjacent imaging units, so that each imaging unit can independently perform intelligent calculations to achieve light field imaging.

[0061] According to the embodiments of the present invention, an embodiment of a light field imaging method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0062] In this embodiment, a light field imaging method is provided, which is applied to a first imaging unit. The first imaging unit is an imaging unit in an array light field imaging system, and there is at least one second imaging unit adjacent to the first imaging unit in the array light field imaging system. The first imaging unit is communicatively connected to each second imaging unit.

[0063] Figure 2 Fig. shows a schematic architecture diagram of an array light field imaging system. Among them, the array light field imaging system includes a plurality of imaging units, and each imaging unit is arranged in an array and communicatively connected to other adjacent imaging units, so that information exchange can be realized. As Figure 2 shown, the array light field imaging system includes 9 imaging units and is arranged in a 3×3 array form, so that each imaging unit has other adjacent imaging units.

[0064] Since the array light field imaging system includes a plurality of imaging units, each imaging unit therein can be used as the first imaging unit and execute the light field imaging method. Moreover, for two adjacent imaging units, both of these two imaging units can be used as the first imaging unit simultaneously, but for each imaging unit, its corresponding second imaging unit is different.

[0065] Figure 3 is a flowchart of the light field imaging method according to an embodiment of the present invention. As Figure 3 shown, the process includes the following steps.

[0066] Step S301, obtain a first sub-aperture image collected by the first imaging unit at the current moment.

[0067] In this embodiment, each imaging unit corresponds to a sub-aperture of the light field, and all imaging units together form a light field imaging system. At the hardware level, each imaging unit includes a CMOS image sensor module, which can collect optical signals at each moment and convert the optical signals into sub-aperture images of electrical signals to achieve imaging. Moreover, the imaging unit further includes an information interaction and intelligent calculation module, which can realize information transmission between imaging units and calculate and reconstruct the environmental light field to achieve perception of the body and the environment.

[0068] For ease of description, the sub-aperture image collected by the first imaging unit is called the first sub-aperture image. It can be understood that at each moment (each frame), the first imaging unit can collect the corresponding first sub-aperture image. For the current moment, let A represent the index of the first imaging unit and t represent the current moment, then the first sub-aperture image collected by the first imaging unit at the current moment is represented as P A (t).

[0069] Similarly, the first sub-aperture image captured by the first imaging unit at the previous moment is P A (t - 1), and the first sub-aperture image captured by the first imaging unit at the next moment is P A (t + 1).

[0070] Step S302: Generate first exchange information, transmit the first exchange information to each second imaging unit, and obtain the second exchange information transmitted by each second imaging unit; the first exchange information includes the first sub-aperture image and the first reconstructed light field of the first imaging unit at the previous moment, and the second exchange information includes the second sub-aperture image captured by the second imaging unit at the current moment and the second reconstructed light field of the second imaging unit at the previous moment; the first exchange information is used to instruct the second imaging unit to execute the light field imaging method.

[0071] In this embodiment, information exchange occurs between the first imaging unit and the second imaging unit. For ease of description, the information transmitted from the first imaging unit to the second imaging unit is referred to as the first exchange information, and the information transmitted from the second imaging unit to the first imaging unit is referred to as the second exchange information.

[0072] Among them, the first exchange information and the second exchange information are similar. Specifically, the first exchange information includes the first sub-aperture image P A (t), and the second exchange information includes the second sub-aperture image captured by the second imaging unit at the current moment. That is, for the second imaging unit, it will also execute the light field imaging method provided in this embodiment, and after capturing the sub-aperture image at the current moment based on its own CMOS image sensor module, it will send the sub-aperture image to other adjacent imaging units, which includes the first imaging unit.

[0073] For ease of description, the sub-aperture image captured by the second imaging unit is referred to as the second sub-aperture image. Similar to the above-mentioned first sub-aperture image, if B represents the index of the second imaging unit, the second sub-aperture image captured by the second imaging unit at the current moment t can be expressed as P B (t). That is, the second sub-aperture image P B (t) is included in the second exchange information obtained by the first imaging unit.

[0074] Moreover, the captured sub-aperture image is used to reconstruct the environmental light field, so as to obtain the corresponding reconstructed environmental light field; for ease of description, the environmental light field reconstructed by the first imaging unit is referred to as the first reconstructed light field, and the environmental light field reconstructed by the second imaging unit is referred to as the second reconstructed light field.

[0075] At each moment, each imaging unit can reconstruct the corresponding reconstructed light field, and at a subsequent moment, use this reconstructed light field as part of the exchanged information and transmit it to adjacent other imaging units. Specifically, at the previous moment, the first imaging unit can reconstruct the first reconstructed light field FIELD A (t - 1) at the previous moment t - 1, and the second imaging unit can reconstruct the second reconstructed light field FIELD B (t - 1) at the previous moment t - 1. At the current moment, the first imaging unit can use this first reconstructed light field FIELD A (t - 1) as part of the first exchanged information, and the second imaging unit can also use this second reconstructed light field FIELD B (t - 1) as part of the second exchanged information and perform information exchange.

[0076] It can be understood that at the initial first moment, since light field reconstruction has not been performed yet, the exchanged information at this time may not include the reconstructed light field.

[0077] Since the first imaging unit may be adjacent to multiple other imaging units, that is, the number of second imaging units is multiple, at this time the first imaging unit needs to send the first exchanged information to each second imaging unit; correspondingly, the first imaging unit can receive the second exchanged information sent by each second imaging unit.

[0078] Step S303, perform environmental light field reconstruction based on the first sub - aperture image, each second sub - aperture image, and each second reconstructed light field at the previous moment to generate the first reconstructed light field at the current moment.

[0079] In this embodiment, after the first imaging unit obtains the second sub - aperture images P B (t) transmitted by each second imaging unit, combined with the first sub - aperture image P A (t) collected by itself, it can perform environmental light field reconstruction to obtain the reconstructed environmental light field, which is the light field corresponding to the current moment. And based on each second reconstructed light field at the previous moment, the environmental light field obtained by image reconstruction can be optimized to finally obtain the reconstructed light field at the current moment, that is, the first reconstructed light field at the current moment.

[0080] Specifically, a light field reconstruction neural network can be preset, and based on this light field reconstruction neural network, the sub - aperture images collected by multiple imaging units are reconstructed into an environmental light field.

[0081] Figure 4 Shows a schematic diagram of the principle of the light field reconstruction neural network. As Figure 4As shown, if the number of imaging units adjacent to the first imaging unit is 1, that is, there is only 1 second imaging unit, the input of the light field reconstruction neural network is two images, including an image of this imaging unit and an image of the adjacent imaging unit, that is, the first sub-aperture image P A (t) and the second sub-aperture image P B (t). The output of the light field reconstruction neural network is a three-dimensional point cloud representation corresponding one-to-one with the image pixels, that is, each pixel in the two images will correspond to 1 point on the three-dimensional point cloud, and moreover, the attributes of each point include the three-dimensional position, color, and confidence of each point, etc. This three-dimensional point cloud is an ambient light field, and thus the first reconstructed light field FIELD A (t) at the current moment can be generated.

[0082] Among them, when reconstructing the ambient light field, the relative pose (including position and attitude) between the imaging units can also be input into the light field reconstruction neural network, or the relative pose between the imaging units may not be required, which is specifically determined based on the actual situation.

[0083] In terms of network structure, the light field reconstruction neural network can adopt a Transformer network architecture, which is constructed based on a standard Transformer encoder and decoder. The Transformer network architecture allows the model to utilize a powerful pre-trained model to learn rich geometric and appearance information from the input images without explicit geometric constraints (the relative pose between the imaging units), so as to achieve the reconstruction of the ambient light field. The light field reconstruction neural network can be trained in an end-to-end manner and directly learn three-dimensional points from the image pairs without the need for complex multi-step processing such as feature matching and triangulation.

[0084] As mentioned above, each imaging unit in the light field imaging system can execute the above method; for the second imaging unit, after obtaining the first exchange information, it can obtain the sub-aperture images collected by other adjacent imaging units, and combine the sub-aperture images collected by itself, such as the second sub-aperture image P B (t), and can also reconstruct the reconstructed light field at its own current moment, which will not be elaborated in this embodiment.

[0085] Step S304, in the case that the preset imaging target is not completed, perform ambient light field reconstruction again at the next moment until the imaging target is completed.

[0086] In this embodiment, the imaging target required currently can be set in advance. For example, in a large scene, the imaging target can be that the imaging unit can synthesize a complete light field; or, when imaging a three-dimensional sphere, information from all angles needs to be collected and the light field is reconstructed. The imaging target is specifically set based on actual requirements, and this embodiment does not limit this.

[0087] If the first imaging unit has not completed the imaging target currently, the environmental light field needs to be reconstructed again at the next moment. That is, at the next moment after that, based on the newly acquired image (the first sub-aperture image at the next moment) and the second exchange information (the second exchange information transmitted by the second imaging unit at the next moment), the above steps S301 to S303 are executed again until the imaging target is completed.

[0088] If the first imaging unit has completed the imaging target currently, it can be determined that the imaging is completed. At this time, the process can be ended, or the next imaging target can be continued to be executed. This embodiment does not limit this.

[0089] In this embodiment, at each moment, light field reconstruction can be performed through information transmission between adjacent imaging units. Thus, after multiple information transmissions, each imaging unit can reconstruct the complete environmental light field information. As Figure 2 shown, after the imaging is completed, each imaging unit can output a complete imaging result, which contains the complete environmental light field information.

[0090] Figure 5 shows a schematic diagram of a process of information transmission between imaging units. As Figure 5 shown, the light field imaging system includes five imaging units connected in sequence. In the initial state, each imaging unit only has its own information. That is, imaging unit 1 and imaging unit 2 only contain their own information 1 and information 2 respectively, and the other imaging units are similar.

[0091] After the first information transmission, information transmission will occur between adjacent imaging units. That is, imaging unit 1 will obtain the information of adjacent imaging units 2 and 3, and perform fusion through light field calculation and reconstruction. The information it contains becomes 123. Similarly, the information contained in imaging unit 2 and imaging unit 3 becomes 125 and 134 respectively. The information contained in imaging unit 4 and imaging unit 5 becomes 34 and 25.

[0092] After the second information transmission, imaging unit 1 will obtain the information of all imaging units and become information 12345. Through continuous information transmission, each imaging unit can obtain the information of all imaging units and continue to be updated through information transmission, so that each imaging unit can reconstruct the complete environmental light field information.

[0093] In the light field imaging method provided in this embodiment, information is transmitted between the imaging unit in the array light field imaging system and other adjacent imaging units, enabling each imaging unit to perform intelligent calculations independently without the need for a centralized light field data processing method. This not only reduces the requirements for system computing power and data bandwidth but also allows the imaging unit to reconstruct the light field without relying on strict calibration between light field sub-apertures, making the processing more flexible. Moreover, the image of each light field sub-aperture can continuously transmit information and provide mutual feedback with the images of adjacent apertures to reconstruct the complete ambient light field information.

[0094] In some alternative embodiments, step S303 “Reconstruct the ambient light field based on the first sub-aperture image, each second sub-aperture image, and each second reconstructed light field at the previous moment to generate the first reconstructed light field at the current moment” may include steps A1 to A2.

[0095] Step A1: Input the first sub-aperture image and each second sub-aperture image into a preset light field reconstruction neural network to obtain the first tentative light field at the current moment.

[0096] Step A2: Perform ambient light field registration and fusion based on the first tentative light field at the current moment, the first reconstructed light field at the previous moment, and each second reconstructed light field at the previous moment to generate the first reconstructed light field at the current moment.

[0097] In this embodiment, the first sub-aperture image P A (t) and each second sub-aperture image P B (t) are input into a preset light field reconstruction neural network to preliminarily obtain the ambient light field reconstructed based on the images, that is, the first tentative light field at the current moment. In this embodiment, field A (t) represents the first tentative light field at the current moment t.

[0098] After that, based on the first tentative light field field A (t) at the current moment, the first reconstructed light field FIELD A (t - 1) determined by itself, and the second reconstructed light field FIELD B (t - 1) based on each second exchange information at the previous moment, perform ambient light field registration and fusion to finally obtain the light field at the current moment that incorporates more information, that is, the first reconstructed light field FIELD A (t).

[0099] Figure 6 Shows a schematic flowchart of generating the first reconstructed light field. As Figure 6 shown, through information transmission, the first imaging unit can obtain the second sub-aperture image P B(t) and the second reconstructed light field FIELD at the previous moment t-1 B (t-1). First, based on the light field reconstruction neural network, determine the first undetermined light field field A (t) at the current moment t. Then, combine the first reconstructed light field FIELD A (t-1) reconstructed by itself at the previous moment t-1, and the second reconstructed light field FIELD B (t-1) transmitted by the second imaging unit, and the registration and fusion of the ambient light field can be performed. Finally, the first undetermined light field field A (t) can be updated to obtain the updated first reconstructed light field FIELD A (t).

[0100] Optionally, as Figure 6 shown, when performing the registration and fusion of the ambient light field, position registration can be achieved, so as to determine the pose (position and attitude) of the first imaging unit. Specifically, the above step A2 "According to the first undetermined light field at the current moment, the first reconstructed light field at the previous moment, and each second reconstructed light field at the previous moment, perform the registration and fusion of the ambient light field to generate the first reconstructed light field at the current moment" can specifically include the following steps A21 to A23.

[0101] Step A21: According to the first undetermined light field at the current moment, the first reconstructed light field at the previous moment, and each second reconstructed light field at the previous moment, perform the point cloud registration of the ambient light field to determine the first pose of the first imaging unit at the current moment.

[0102] In this embodiment, after determining the first undetermined light field field A (t), the first reconstructed light field FIELD A (t-1) at the previous moment, and each second reconstructed light field FIELD B (t-1) and other ambient light fields at the previous moment, when performing the registration and fusion, perform the registration of the point clouds of each ambient light field, so as to update the pose of the first imaging unit and determine the pose of the first imaging unit at the current moment, that is, the first pose. In this embodiment, Pose A (t) represents the first pose of the first imaging unit at the current moment; similarly, the pose of the second imaging unit is called the second pose, and can be represented by Pose B (t) represents the second pose of the first imaging unit at the current moment.

[0103] Optionally, the objective function of the ambient light field point cloud registration can be set, and the pose update can be achieved by optimizing the objective function. Specifically, the objective function is:

[0104]

[0105] where i and j are indices of imaging units, and k is an index of point clouds; represents the k-th point cloud data in the determined ambient light field when the i-th imaging unit is the first imaging unit and the j-th imaging unit is the second imaging unit; represents the k-th point cloud data confidence; represents the k-th point cloud data in the determined ambient light field when the j-th imaging unit is the first imaging unit and the i-th imaging unit is the second imaging unit; represents the k-th point cloud data confidence;

[0106] s is a scaling parameter, and P is a rigid body transformation parameter; s j→i represents the scaling parameter from the point cloud of the j-th imaging unit to the point cloud of the i-th imaging unit, and P j→i represents the rigid body transformation parameter from the point cloud of the j-th imaging unit to the point cloud of the i-th imaging unit; s i→j represents the scaling parameter from the point cloud of the i-th imaging unit to the point cloud of the j-th imaging unit, and P i→j represents the rigid body transformation parameter from the point cloud of the i-th imaging unit to the point cloud of the j-th imaging unit; the rigid body transformation parameter P and the scaling parameter s are used to determine the first pose Pose A (t) of the first imaging unit at the current moment.

[0107] In this embodiment, for the first imaging unit A, if it is the i-th imaging unit, and for the second imaging unit B, it is the j-th imaging unit, then is specifically the k-th point cloud data in the first reconstructed light field. Correspondingly, is specifically the k-th point cloud data in the second reconstructed light field. The confidence can be determined based on the output result of the light field reconstruction neural network. Both s and P are transformations (Transform) from point cloud to point cloud, where the former represents scaling and the latter represents rigid body transformation.

[0108] By optimizing the above objective function, the optimal rigid body transformation parameter P and scaling parameter s can be obtained. Based on this, the relative pose between the two imaging units can be obtained, and then the pose of the first imaging unit itself, that is, the first pose Pose A (t) at the current moment can be determined.

[0109] For the case where there are multiple second imaging units, it is necessary to match and fuse the point clouds of multiple ambient light fields. The above objective function can be extended to be applied to all adjacent imaging unit pairs to achieve registration.

[0110] Step A22: Superimpose the registered point clouds together to generate a fused point cloud.

[0111] Step A23: Optimize the point cloud representation of the first undetermined light field at the current moment according to the first pose at the current moment and the fused point cloud to obtain the first reconstructed light field at the current moment.

[0112] In this embodiment, after the point cloud registration is completed, the point clouds of all ambient light fields are superimposed together. Among them, the point clouds with a distance less than a preset distance (indicating a very small distance) can be merged to obtain the fused point cloud, that is, the fused point cloud.

[0113] After obtaining the fused point cloud, further optimize the 3D point cloud representation using the point cloud and the position and pose of the first imaging unit. Specifically, the 3D point cloud representation of the first undetermined light field field A (t) at the current moment can be optimized according to the first pose Pose A (t) at the current moment to obtain a 3D Gaussian sphere representation. In this optimization process, the correspondence between the 3D representation and the 2D image can be used to eliminate part of the error introduced when the light field reconstruction neural network outputs the point cloud and during point cloud registration, thereby further improving the effect of light field reconstruction and making the finally reconstructed first reconstructed light field FIELD A (t) more accurate.

[0114] Among them, the 3D Gaussian sphere algorithm is a mature technology for 3D representation, reconstruction, and optimization, which will not be elaborated in this embodiment.

[0115] Figure 7 Shows a schematic diagram of the effect of registering and fusing two reconstructed light fields. As Figure 7 shown, register and fuse the first reconstructed light field of the first imaging unit and the second reconstructed light field of the second imaging unit. For example, register and fuse the first reconstructed light field FIELD A (t - 1) and the second reconstructed light field FIELD B (t - 1) to obtain the first pose Pose A (t) of the first imaging unit, and then optimize to obtain the first reconstructed light field FIELD A (t) at the current moment.

[0116] In some alternative embodiments, since the imaging units no longer need to rely on the strict calibration between the light field sub-apertures, during the reconstruction of the light field information, the light field imaging system can also perform self-adjustment to change its body form, that is, the relative relationship between the sub-apertures (imaging units), so as to achieve better perception of the environment. Specifically, each imaging unit can adjust its own pose based on actual needs to better perceive the surrounding environment; for example, through deformation, the light field information in a narrow area can be perceived, etc.

[0117] Specifically, the exchanged information between the imaging units also includes the corresponding poses. Among them, the first exchanged information further includes the first pose Pose A (t - 1) of the first imaging unit at the previous moment, and the second exchanged information further includes the second pose Pose B (t - 1) of the second imaging unit at the previous moment.

[0118] Figure 8 The schematic diagram of the information transfer process between adjacent imaging units is shown. As Figure 8 shown, the exchanged information between adjacent imaging units mainly includes two parts: 1) the sub-aperture image collected by the CMOS image sensor in this imaging unit at the current moment; 2) the light field reconstruction result at the previous moment (including the reconstructed environmental light field characterization, such as point cloud, three-dimensional Gaussian sphere, etc., and the body pose information, that is, the pose information of the imaging unit involved in the environmental light field reconstruction result. These light field reconstruction results can be used as the basis for subsequent iterative processing.

[0119] And this method further includes the following steps B1 to B3.

[0120] Step B1: Input the first reconstructed light field at the current moment, the first pose determined at the current moment, and each second pose at the previous moment into a preset form feedback neural network to obtain the first planned pose at the next moment.

[0121] Step B2: Determine the deformation parameters of the first imaging unit according to the first planned pose at the next moment.

[0122] Step B3: Adjust the pose of the first imaging unit according to the deformation parameters.

[0123] In this embodiment, a form feedback neural network for realizing pose prediction is preset in advance. This form feedback network can output the next position and pose of this imaging unit, that is, the planned pose, according to the position and pose of this imaging unit itself at the current moment, the positions and poses of adjacent imaging units, and the reconstructed environmental light field information (point cloud or three-dimensional Gaussian sphere characterization).

[0124] Specifically, the input of the morphological feedback neural network includes the first reconstructed light field FIELD A (t) at the current moment, the first pose Pose A (t) determined at the current moment, and each second pose Pose B (t - 1) at the previous moment. Its output is the next pose of the first imaging unit, that is, the first planned pose PE A (t + 1) at the next moment. This first planned pose PE A (t + 1) is a predicted position. By planning the pose of the first imaging unit according to this first planned pose PE A (t + 1), pose control can be achieved.

[0125] Among them, the morphological feedback neural network can also adopt the Transformer network architecture and be trained in an end-to-end manner. The data required for training can be obtained in large quantities in a virtual simulation environment. During the training process, the quality of the light field information reconstruction of the current pose can be judged according to indicators such as the integrity of the scene and the clarity of scene details, and the morphological feedback neural network can be trained according to this evaluation result.

[0126] After predicting the first planned pose PE A (t + 1) at the next moment, deformation parameters corresponding to this first planned pose PE A (t + 1) can be generated. These deformation parameters are used to adjust the pose of the first imaging unit so as to adjust the actual pose of the first imaging unit to this first planned pose PE A (t + 1) as much as possible at the next moment.

[0127] Optionally, the above step B2 "determine the deformation parameters of the first imaging unit according to the first planned pose (PE A (t + 1)) at the next moment" can include steps B21 to B22.

[0128] Step B21, obtain the second planned poses of each second imaging unit at the next moment.

[0129] Step B22, perform joint optimization according to the first planned pose at the next moment and the second planned poses of each second imaging unit at the next moment, and determine the deformation parameters of the first imaging unit.

[0130] In this embodiment, the imaging units can also transmit the predicted planned poses to each other. Figure 9 Shows a schematic process diagram for determining the deformation parameters.

[0131] As Figure 9As shown, based on the morphological feedback neural network, the first planned pose PE of the next step of this imaging unit (i.e., the first imaging unit) is determined. A After (t + 1), through information exchange, the planned pose of itself predicted by the adjacent imaging unit (i.e., the second imaging unit), that is, the second planned pose PE at the next moment, can be determined. B (t + 1). Among them, the morphological feedback neural network can output multiple planned poses and output corresponding scores; when performing joint optimization, based on the planned poses and scores of each imaging unit, the joint optimization of poses can be realized based on a preset neural network, etc., and finally the deformation parameters of the first imaging unit are generated.

[0132] Among them, when the planned poses are transmitted between imaging units, they can be transmitted separately; or, the planned pose is also located in the corresponding exchanged information, such as Figure 8 each of the exchanged information shown also includes the corresponding planned pose, and this embodiment does not limit this.

[0133] This embodiment can be applied to a light field imaging system whose body shape can be adaptively adjusted. It no longer depends on the strict calibration between each imaging unit (optical sub-aperture), and the entire light field imaging system can be very flexible.

[0134] Figure 10 The schematic diagrams of the architectures of two light field imaging systems are shown. As Figure 10 shown, the light field imaging system can be a flexible and stretchable intelligent light field imaging system. The imaging units are connected by flexible and extensible interconnection wires, and information transmission between adjacent imaging units can be realized through the interconnection wires. Or, the light field imaging system can also be a multi-agent light field imaging system. Each agent corresponds to an imaging unit, and the agent is, for example, a drone, etc., and information transmission is carried out by means of the wireless communication ability between agents.

[0135] The light field imaging method provided by this embodiment registers and fuses the ambient light field reconstructed by the network, the ambient light field reconstructed by this imaging unit before, and the ambient light field reconstructed by the adjacent imaging units obtained through information transmission, and obtains the updated reconstructed ambient light field and the positions and postures of each imaging unit after update, which can save computing power and bandwidth and improve scalability. Moreover, the mutual relationship between imaging units (optical sub-apertures) is no longer fixed, but can be adaptively adjusted according to the changes of the environment and imaging targets, so as to achieve better perception of the environment.

[0136] In this embodiment, an optical field imaging device is further provided. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0137] This embodiment provides an optical field imaging device, which is applied to a first imaging unit; the first imaging unit is an imaging unit in an array optical field imaging system, and there is at least one second imaging unit adjacent to the first imaging unit in the array optical field imaging system. The first imaging unit is communicatively connected to each of the second imaging units; as Figure 11 shown, the device includes:

[0138] An image acquisition module 1101, configured to acquire a first sub-aperture image acquired by the first imaging unit at the current moment;

[0139] An exchange module 1102, configured to generate first exchange information, transmit the first exchange information to each of the second imaging units, and acquire second exchange information transmitted by each of the second imaging units; the first exchange information includes the first sub-aperture image and the first reconstructed optical field of the first imaging unit at the previous moment, and the second exchange information includes the second sub-aperture image acquired by the second imaging unit at the current moment and the second reconstructed optical field of the second imaging unit reconstructed at the previous moment; the first exchange information is used to instruct the second imaging unit to execute the optical field imaging method;

[0140] An optical field reconstruction module 1103, configured to perform ambient optical field reconstruction based on the first sub-aperture image, each of the second sub-aperture images, and each of the second reconstructed optical fields at the previous moment, and generate a first reconstructed optical field at the current moment;

[0141] A loop module 1104, configured to perform ambient optical field reconstruction again at the next moment until the preset imaging target is completed when the preset imaging target is not completed. That is, the loop module 1104 instructs the image acquisition module 1101, the exchange module 1102, and the optical field reconstruction module 1103 to perform corresponding processing in a loop until the imaging target is completed.

[0142] In some optional implementation manners, the optical field reconstruction module 1103 performs ambient optical field reconstruction based on the first sub-aperture image, each of the second sub-aperture images, and each of the second reconstructed optical fields at the previous moment, and generates a first reconstructed optical field at the current moment, including:

[0143] Input the first sub-aperture image and each of the second sub-aperture images into a preset light field reconstruction neural network to obtain a first undetermined light field at the current moment;

[0144] Perform ambient light field registration and fusion based on the first undetermined light field at the current moment, the first reconstructed light field at the previous moment, and each of the second reconstructed light fields at the previous moment to generate the first reconstructed light field at the current moment.

[0145] In some optional implementation manners, the light field reconstruction module 1103 performs ambient light field registration and fusion based on the first undetermined light field at the current moment, the first reconstructed light field at the previous moment, and each of the second reconstructed light fields at the previous moment to generate the first reconstructed light field at the current moment, including:

[0146] Perform ambient light field point cloud registration based on the first undetermined light field at the current moment, the first reconstructed light field at the previous moment, and each of the second reconstructed light fields at the previous moment to determine the first pose of the first imaging unit at the current moment;

[0147] Overlay the registered point clouds together to generate a fused point cloud;

[0148] Perform point cloud representation optimization on the first undetermined light field at the current moment according to the first pose at the current moment and the fused point cloud to obtain the first reconstructed light field at the current moment.

[0149] In some optional implementation manners, the objective function of the ambient light field point cloud registration is:

[0150]

[0151] where i and j are indices of imaging units, and k is an index of a point cloud; represents the k-th point cloud data in the determined ambient light field when the i-th imaging unit is the first imaging unit and the j-th imaging unit is the second imaging unit, represents the k-th point cloud data confidence; represents the k-th point cloud data in the determined ambient light field when the j-th imaging unit is the first imaging unit and the i-th imaging unit is the second imaging unit, represents the k-th point cloud data confidence;

[0152] s is a scaling parameter, and P is a rigid body transformation parameter; s j→i represents the scaling parameter from the point cloud of the j-th imaging unit to the point cloud of the i-th imaging unit, and P j→i represents the rigid body transformation parameter from the point cloud of the j-th imaging unit to the point cloud of the i-th imaging unit; s i→jrepresents the scaling parameter from the point cloud of the i-th imaging unit to the point cloud of the j-th imaging unit, P i→j represents the rigid body transformation parameter from the point cloud of the i-th imaging unit to the point cloud of the j-th imaging unit; the rigid body transformation parameter P and the scaling parameter s are used to determine the first pose of the first imaging unit at the current moment.

[0153] In some alternative embodiments, the first exchange information further includes the first pose of the first imaging unit at the previous moment, and the second exchange information further includes the second pose of the second imaging unit at the previous moment;

[0154] The device further includes a pose processing module, configured to:

[0155] input the first reconstructed light field at the current moment, the first pose determined at the current moment, and each second pose at the previous moment into a preset shape feedback neural network to obtain the first planned pose at the next moment;

[0156] determine the deformation parameter of the first imaging unit according to the first planned pose at the next moment;

[0157] adjust the pose of the first imaging unit according to the deformation parameter.

[0158] In some alternative embodiments, the pose processing module determines the deformation parameter of the first imaging unit according to the first planned pose at the next moment, including:

[0159] obtain the second planned pose of each second imaging unit at the next moment;

[0160] perform joint optimization according to the first planned pose at the next moment and each second planned pose at the next moment to determine the deformation parameter of the first imaging unit.

[0161] The further function descriptions of the above-mentioned various modules and units are the same as those in the corresponding foregoing embodiments, and will not be elaborated herein.

[0162] The light field imaging device in this embodiment is presented in the form of functional units. Here, the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, including a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0163] The embodiment of the present invention further provides an imaging unit having the above-mentioned Figure 11 shown light field imaging device.

[0164] Please refer to Figure 12 , Figure 12FIG. 0 is a schematic structural diagram of an imaging unit provided by an alternative embodiment of the present invention. As Figure 12 shown, the imaging unit includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the imaging unit, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some alternative embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple imaging units can be connected, and each device provides some necessary operations (such as an array of servers, a set of blade servers, or a multi-processor system). Figure 12 In FIG. 1, one processor 10 is taken as an example.

[0165] The processor 10 can be a central processing unit, a network processor, or a combination thereof. Among them, the processor 10 can further include a hardware chip. The above-mentioned hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above-mentioned programmable logic device can be a complex programmable logic device, a field-programmable gate array, a generic array logic, or any combination thereof.

[0166] Among them, the memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiments.

[0167] The memory 20 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the imaging unit, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 can optionally include a memory remotely set relative to the processor 10, and these remote memories can be connected to the imaging unit through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0168] The memory 20 can include a volatile memory, such as a random access memory; the memory can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid-state drive; the memory 20 can also include a combination of the above types of memories.

[0169] The imaging unit further includes a communication interface 30 for the imaging unit to communicate with other devices or a communication network.

[0170] The embodiments of the present invention further provide an array light field imaging system, which includes the imaging unit provided in any of the above embodiments; a plurality of the imaging units are arranged in an array, and each of the imaging units is communicatively connected to other adjacent imaging units. For the specific structure of the array light field imaging system, reference can be made to Figure 2 or Figure 10 as shown, which will not be elaborated here.

[0171] The embodiments of the present invention also provide a computer-readable storage medium. The methods according to the embodiments of the present invention can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented as computer code that is originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the methods described herein can be processed by such software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the methods shown in the above embodiments are implemented.

[0172] A part of the present invention can be applied as a computer program product, such as computer program instructions, which when executed by a computer, can call or provide the methods and / or technical solutions according to the present invention through the operation of the computer. Those skilled in the art should understand that the forms of existence of computer program instructions in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0173] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations should be covered within the protection scope of the present invention.

Claims

1. A light field imaging method, characterized in that: Applied to a first imaging unit; the first imaging unit is an imaging unit in an array light field imaging system, and in the array light field imaging system there is at least one second imaging unit adjacent to the first imaging unit, and the first imaging unit is communicatively connected with each of the second imaging units; The method comprises: Acquire a first sub-aperture image acquired by the first imaging unit at a current moment; Generate first exchange information, transmit the first exchange information to each of the second imaging units, and obtain second exchange information transmitted by each of the second imaging units; the first exchange information includes the first sub-aperture image and the first reconstructed light field of the first imaging unit at a previous moment, and the second exchange information includes the second sub-aperture image collected by the second imaging unit at a current moment and the second reconstructed light field reconstructed by the second imaging unit at a previous moment; the first exchange information is used to instruct the second imaging unit to execute the light field imaging method; Reconstructing the ambient light field according to the first sub-aperture image, each of the second sub-aperture images, and each of the second reconstructed light fields at a previous moment to generate a first reconstructed light field at a current moment; In the case where the preset imaging target is not completed, the ambient light field reconstruction is performed again at the next moment until the imaging target is completed.

2. The method according to claim 1, characterized in that: The step of reconstructing the ambient light field according to the first sub-aperture image, each of the second sub-aperture images, and each of the second reconstructed light fields at the previous moment to generate the first reconstructed light field at the current moment includes: Inputting the first sub-aperture image and each of the second sub-aperture images into a preset light field reconstruction neural network to obtain a first pending light field at a current moment; According to the first pending light field at the current moment, the first reconstructed light field at the previous moment and each second reconstructed light field at the previous moment, the ambient light field registration and fusion are performed to generate the first reconstructed light field at the current moment.

3. The method according to claim 2, characterized in that The step of performing ambient light field registration and fusion according to the first pending light field at the current moment, the first reconstructed light field at the previous moment, and each second reconstructed light field at the previous moment to generate the first reconstructed light field at the current moment includes: Performing ambient light field point cloud registration according to the first pending light field at the current moment, the first reconstructed light field at the previous moment, and each second reconstructed light field at the previous moment to determine the first pose of the first imaging unit at the current moment; Superimpose the registered point clouds together to generate a fused point cloud; The point cloud representation optimization is performed on the first undetermined light field at the current moment according to the first pose at the current moment and the fused point cloud to obtain the first reconstructed light field at the current moment.

4. The method according to claim 3, characterized in that The objective function of the ambient light field point cloud registration is: Among them, i and j are the indexes of the imaging unit, and k is the index of the point cloud; represents the kth point cloud data in the determined ambient light field when the i-th imaging unit is used as the first imaging unit and the j-th imaging unit is used as the second imaging unit, Represents the kth point cloud data Confidence level; represents the kth point cloud data in the determined ambient light field when the jth imaging unit is used as the first imaging unit and the ith imaging unit is used as the second imaging unit, Represents the kth point cloud data Confidence level; s is the scaling parameter, P is the rigid body transformation parameter; s j→i represents the scaling parameter from the point cloud of the jth imaging unit to the point cloud of the i-th imaging unit, P j→i represents the rigid body transformation parameters from the point cloud of the jth imaging unit to the point cloud of the ith imaging unit; s i→j represents the scaling parameter from the point cloud of the i-th imaging unit to the point cloud of the j-th imaging unit, P i→j Represents the rigid body transformation parameters from the point cloud of the i-th imaging unit to the point cloud of the j-th imaging unit; the rigid body transformation parameters P and the scaling parameters s are used to determine the first position of the first imaging unit at the current moment.

5. The method according to any one of claims 1 to 4, characterized in that The first exchange information further includes a first pose of the first imaging unit at a previous moment, and the second exchange information further includes a second pose of the second imaging unit at a previous moment; The method further comprises: The first reconstructed light field at the current moment, the first pose determined at the current moment, and each second pose at the previous moment are input into a preset morphological feedback neural network to obtain the first planned pose at the next moment; Determine a deformation parameter of the first imaging unit according to a first planned posture at a next moment; The posture of the first imaging unit is adjusted according to the deformation parameter.

6. The method according to claim 5, characterized in that The step of determining the deformation parameter of the first imaging unit according to the first planned posture at the next moment includes: Acquire a second planned position and posture of each of the second imaging units at a next moment; The deformation parameters of the first imaging unit are determined by performing joint optimization according to the first planned posture at the next moment and each second planned posture at the next moment.

7. A light field imaging device, characterized in that: Applied to a first imaging unit; the first imaging unit is an imaging unit in an array light field imaging system, and there is at least one second imaging unit adjacent to the first imaging unit in the array light field imaging system, and the first imaging unit is communicatively connected with each of the second imaging units; the device comprises: An image acquisition module, used to acquire a first sub-aperture image acquired by the first imaging unit at a current moment; an exchange module, configured to generate first exchange information, transmit the first exchange information to each of the second imaging units, and obtain second exchange information transmitted by each of the second imaging units; the first exchange information includes the first sub-aperture image and the first reconstructed light field of the first imaging unit at a previous moment, and the second exchange information includes the second sub-aperture image collected by the second imaging unit at a current moment and the second reconstructed light field reconstructed by the second imaging unit at a previous moment; the first exchange information is used to instruct the second imaging unit to execute the light field imaging method; A light field reconstruction module, configured to reconstruct the ambient light field according to the first sub-aperture image, each of the second sub-aperture images and each of the second reconstructed light fields at a previous moment, to generate a first reconstructed light field at a current moment; The loop module is used to reconstruct the ambient light field at the next moment if the preset imaging target is not completed until the imaging target is completed.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the light field imaging method according to any one of claims 1 to 6.

9. An imaging unit, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the light field imaging method according to any one of claims 1 to 6 by executing the computer instructions.

10. An array light field imaging system, characterized in that: comprising a plurality of imaging units as claimed in claim 9; A plurality of the imaging units are arranged in an array, and each of the imaging units is communicatively connected with other adjacent imaging units.

Images