Flexible self-deformable array light field intelligent imaging system and device

Abstract

The present disclosure relates to the field of array light field imaging technology, and in particular to a flexible self-deformable array light field intelligent imaging system and device. The system includes: a flexible image sensor array module, the flexible image sensor array module includes a plurality of rigid image sensors and a plurality of flexible stretchable wires; between the plurality of rigid image sensors and the plurality of flexible stretchable wires, the rigid image sensors are used as islands, and the flexible stretchable wires are used as bridges to form an island-bridge structure. The present disclosure using the above scheme can break through the limitations of the rigid morphological design on the imaging system, solve the contradiction between the performance and flexibility of the array light field imaging system, and realize light field intelligent imaging with high performance, high flexibility, and high adaptability.

Description

Technical Field

[0001] The present disclosure relates to the technical field of array light field imaging, and in particular to a flexible self-deformable array light field intelligent imaging system and device. Background Art

[0002] With the development and breakthrough of artificial intelligence technology represented by deep learning, light field imaging technology has made great breakthroughs in imaging dimension, scale, performance and robustness. As a classic method of light field imaging, array sensor light field imaging (hereinafter referred to as "array light field imaging") technology has been widely used in smartphone imaging systems and has shown great application potential in scientific observation, smart cities, public safety, unmanned systems and other fields.

[0003] In the related technologies, although the array light field imaging system directly or indirectly improves the performance of the imaging system, it also brings about a more complex optoelectronic system design, a larger volume and weight, and more stringent working conditions, and has to make a compromise between performance and portability and flexibility, thus limiting the further development and application of the array sensor imaging system. Summary of the invention

[0004] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the first purpose of the present invention is to propose a flexible self-deformable array light field intelligent imaging system to break through the limitations of the rigid form design on the imaging system, solve the contradiction between the performance and flexibility of the array light field imaging system, and achieve light field intelligent imaging with high performance, high flexibility and high adaptability.

[0006] The second objective of the present disclosure is to provide a flexible self-deformable array light field intelligent imaging device.

[0007] To achieve the above-mentioned purpose, the first embodiment of the present disclosure proposes a flexible self-deformable array light field intelligent imaging system, comprising:

[0008] A flexible image sensor array module, the flexible image sensor array module comprising a plurality of rigid image sensors and a plurality of flexible stretchable wires;

[0009] The plurality of rigid image sensors and the plurality of flexible stretchable wires are connected to each other by using the rigid image sensors as islands and the flexible stretchable wires as bridges, thereby forming an island-bridge structure.

[0010] Optionally, the target surface of the rigid image sensor is smaller than a target surface threshold.

[0011] Optionally, the flexible stretchable conductor is a serpentine flexible stretchable conductor; the serpentine flexible stretchable conductor includes multiple parallel serpentine flexible stretchable sub-conductors; the serpentine flexible stretchable sub-conductors include a first polyimide layer, a copper layer and a second polyimide layer stacked in sequence.

[0012] Optionally, the serpentine flexible stretchable conductor comprises four parallel serpentine flexible stretchable sub-conductors, and the conductor width of the serpentine flexible stretchable sub-conductors corresponds to a width range of 0.2 mm to 0.3 mm.

[0013] Optionally, the flexible image sensor array module further includes a first polydimethylsiloxane film and a second polydimethylsiloxane film; wherein,

[0014] In the island bridge structure, the interconnected rigid image sensors and the rigid image sensors between the flexible stretchable wires are welded on the first side of the flexible stretchable wires;

[0015] The first polydimethylsiloxane film is bonded to the first side of the flexible stretchable wire by pressure, the second polydimethylsiloxane film is bonded to the second side of the flexible stretchable wire by pressure, and the first polydimethylsiloxane film includes a plurality of holes, and the positions of the holes correspond to the positions of the rigid image sensors one by one, so that the rigid image sensor passes through the first polydimethylsiloxane film through the holes;

[0016] Polydimethylsiloxane material is poured around the rigid image sensor that passes through the first polydimethylsiloxane film.

[0017] Optionally, the flexible self-deformable array light field intelligent imaging system further includes a data readout module and a light field processing module; wherein,

[0018] The data reading module is used to read the image data set collected by the multiple rigid image sensors and send the image data set to the light field processing module;

[0019] The light field processing module is used to reconstruct the ambient light field according to the image data set to obtain ambient light field information.

[0020] Optionally, wherein

[0021] Each of the rigid image sensors is connected to the data readout module via a flexible stretchable wire, so that the data readout module reads out data from all rigid image sensors in parallel to obtain the image data set;

[0022] or,

[0023] The plurality of rigid image sensors are connected to the data readout module via a flexible stretchable wire, so that the data readout module serially reads out data from all rigid image sensors to obtain the image data set.

[0024] Optionally, the data readout module includes a sensor control submodule, a multi-sensor synchronization submodule and an image signal processing submodule; wherein,

[0025] The sensor control submodule is used to control sensor parameters of the rigid image sensor;

[0026] The multi-sensor synchronization submodule is used to read the image data set collected by the multiple rigid image sensors and synchronize the data between the multiple rigid image sensors;

[0027] The image signal processing submodule is used to optimize the image data set to obtain and send the optimized image data set to the light field processing module.

[0028] Optionally, the light field processing module includes an environment perception submodule and a body perception submodule; wherein,

[0029] The environment perception submodule is used to extract semantic information of each image data in the image data set according to the visual basic model, and estimate first three-dimensional depth information corresponding to each image data according to the semantic information;

[0030] The body perception submodule is used to establish a feature point association relationship between different image data in the image data set; perform array image-light field three-dimensional joint optimization according to the first three-dimensional depth information and the feature point association relationship to obtain the posture information corresponding to each image data and the second three-dimensional depth information; and reconstruct the body posture corresponding to the flexible image sensor array module according to the posture information;

[0031] The environment perception submodule is further used to obtain the environment light field information by fusing the body posture and the second three-dimensional depth information.

[0032] To achieve the above-mentioned purpose, a second aspect of the present disclosure provides a flexible self-deformable array light field intelligent imaging device, including: a flexible self-deformable array light field intelligent imaging system as shown in any one of the above-mentioned first aspects.

[0033] In summary, the system and device provided by the present invention, by adopting a rigid-flexible "island-bridge" technical route, not only ensures the image pixel resolution, but also takes into account the flexible and extensible characteristics of the system as a whole, and can give the imaging system soft and deformable, flexible and reconfigurable functions, thereby breaking through the limitations of the rigid morphological design on the imaging system, solving the contradiction between the performance and flexibility of the array light field imaging system, and realizing light field intelligent imaging with high performance, high flexibility and high adaptability.

[0034] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0036] Figure 1 A schematic diagram of the principle of a light field 4D model provided by an embodiment of the present disclosure;

[0037] Figure 2 A schematic diagram of the structure of a flexible self-deformable array light field intelligent imaging system provided by an embodiment of the present disclosure;

[0038] Figure 3 A schematic diagram of the structure of a flexible stretchable wire provided in an embodiment of the present disclosure;

[0039] Figure 4 A cross-sectional view of a flexible stretchable sub-conductor provided in an embodiment of the present disclosure;

[0040] Figure 5 A preparation flow chart of a flexible image sensor array module provided in an embodiment of the present disclosure;

[0041] Figure 6 A schematic diagram of a design of a flexible stretchable wire provided in an embodiment of the present disclosure;

[0042] Figure 7 A structural diagram of a data readout module provided by an embodiment of the present disclosure;

[0043] Figure 8 A workflow diagram of a light field processing module provided in an embodiment of the present disclosure;

[0044] Fig. 9 A schematic diagram of imaging results of a flexible self-deformable array light field intelligent imaging system provided by an embodiment of the present disclosure;

[0045] Fig.10A schematic diagram of an application of a flexible self-deformable array light field intelligent imaging system provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0047] Light field, as the name implies, is the distribution of a physical quantity of light in space. Simply put, the light field describes the intensity of light in any direction at any point in space. The plenoptic function that fully describes the light field is a 7-dimensional function, which includes the position of any point (x, y, z), any direction (Θ, Φ in polar coordinates), wavelength (λ) and time (t). In practical applications, information in color and time dimensions is usually represented by RGB channels and different frames, so as far as the light field is concerned, we only need to pay attention to the direction and position of the light, so that the plenoptic function can be reduced from 7 dimensions to 5 dimensions.

[0048] Next, we can continue to simplify and reduce the dimension of the full light function. Assume two non-coplanar planes (u, v) and (s, t). If a light ray has an intersection with each of these two planes, then the light ray can be uniquely represented by these two intersection points, thus obtaining a (u, v, s, t) light field 4D model. This light field 4D model has an important premise assumption: the light collected at any position along the propagation direction of the light is the same. In other words, it is assumed that the light intensity of any light ray does not attenuate and the wavelength remains unchanged during the propagation process. Considering that the propagation distance of light from the scene surface to the human eye in daily life is very limited, and the attenuation of light in the air is negligible, the above assumption is completely reasonable.

[0049] However, Figure 1 The schematic diagram of the principle of a light field 4D model provided by the embodiment of the present disclosure is as follows. Figure 1 As shown, the light field 4D model cannot fully describe all the light in the three-dimensional space. The light parallel to the (u, v) or (s, t) plane cannot be represented by the light field 4D model. Figure 1 The light is marked by the dashed line.

[0050] Although the light field 4D model cannot fully describe all the light in three-dimensional space, it can fully describe the light received by the human eye. Because when the light is perpendicular to the forward viewing direction of the human eye, the light will not enter the human eye. Therefore, this part of the light does not affect the visual imaging of the human eye. In other words, the light field 4D model not only reduces the dimension required to represent the light field, but also can fully represent all the light required for the human eye to form an image. Therefore, the light field 4D model has been widely recognized by the academic community, and a large number of studies on light fields have been carried out on this basis.

[0051] Array light field imaging is a technology that uses array sensors to capture three-dimensional object scenes. It combines the advantages of light field imaging and array imaging, and can obtain complete three-dimensional information in a single fast shot. An array light field imaging system, for example, can be a camera array composed of multiple cameras in a fixed shape, which can simultaneously collect spatial information and perspective information to achieve efficient acquisition of three-dimensional information.

[0052] However, the existing rigid array light field imaging system is developed based on the above light field 4D model, adopts a fixed planar (or spherical) array structure design, and the relationship between each sub-aperture sensor of the light field is fixed. When the aperture and baseline of the light field imaging system increase, the volume of the imaging system will also increase, which seriously affects the flexibility of the imaging system.

[0053] In recent years, emerging technologies such as flexible electronics have developed rapidly, providing new ideas for solving the contradiction between performance and flexibility of array sensor imaging systems. Flexible electronics technology can make electronic circuits soft, stretchable and foldable, just like umbrellas and satellite solar panels, which can be opened to increase the working area when needed and folded and stored when not needed.

[0054] However, although flexible electronic technology has made great breakthroughs, its absolute performance is still far behind that of mature rigid electronic technology. For example, due to the limitations of the preparation process and integration of flexible electronics, the number of pixels in flexible image sensors can only reach a few hundred (16x16) or thousands (32x32), while the number of pixels in rigid image sensors can reach tens of millions or even hundreds of millions, which is much higher than that of fully flexible image sensors.

[0055] Therefore, since these emerging technologies are still in the development stage, there are still two key challenges when applying them to array light field imaging: 1) The existing array sensor light field calculation theory and algorithm rely on the strict calibration of rigid geometric structures, which is difficult to support the computational reconstruction of flexible deformable array sensors. 2) There is still a gap between the performance of current flexible electronic components and mature technologies, making it difficult to completely replace existing sensors and optical lenses.

[0056] The present disclosure is described in detail below with reference to specific embodiments.

[0057] Figure 2 A schematic diagram of the structure of a flexible self-deformable array light field intelligent imaging system provided in an embodiment of the present disclosure.

[0058] like Figure 2 As shown, the flexible self-deformable array light field intelligent imaging system comprises:

[0059] The flexible image sensor array module 10 includes a plurality of rigid image sensors 11 and a plurality of flexible stretchable wires 12 .

[0060] An island-bridge structure is formed between the plurality of rigid image sensors 11 and the plurality of flexible stretchable wires 12 , with the rigid image sensors 11 serving as islands and the flexible stretchable wires 12 serving as bridges.

[0061] According to some embodiments, the rigid image sensor 11 is used to sense ambient light field information, and the flexible stretchable wire 12 is used to transmit the signal of the rigid image sensor 11 .

[0062] It is easy to understand that the flexible image sensor array module 10 adopts an "island-bridge" hybrid structure, with the rigid image sensor 11 as the island, and can make use of the mature process of the rigid image sensor 11 to make the flexible self-deformable array light field intelligent imaging system have high pixel resolution; with the flexible stretchable wire 12 as the bridge, the flexible image sensor array module 10 as a whole can have the characteristics of being soft, stretchable and deformable.

[0063] Optionally, the target surface of the rigid image sensor 11 is smaller than the target surface threshold. Therefore, the influence of the rigid image sensor 11 on the overall softness and stretchability of the flexible self-deformable array light field intelligent imaging system can be reduced.

[0064] Optionally, the rigid image sensor 11 may be, for example, a rigid CMOS image sensor, which may be, for example, a small-target-area rigid CMOS image sensor.

[0065] For example, the rigid CMOS image sensor may adopt an OV6946 (1 / 18") or OV9734 (1 / 9") module.

[0066] Optionally, the flexible stretchable wire 12 may be, for example, a serpentine flexible stretchable wire.

[0067] According to some embodiments, key parameters of the serpentine flexible stretchable wire include the wire arc radius R, the wire straight segment length L, and the wire width w.

[0068] The greater the length L of the straight line segment of the wire, the stronger the stretchability of the serpentine flexible stretchable wire. However, correspondingly, the overall area of ​​the serpentine flexible stretchable wire will also be larger, and the arrangement density of the flexible image sensor array module 10 will be limited.

[0069] Among them, the larger the wire width w, the stronger the performance of the serpentine flexible stretchable wire, but its flexible stretchability will be weakened.

[0070] Optionally, when a rigid image sensor 11 requires multiple signal line outputs, the flexible stretchable conductor 12 includes multiple parallel flexible stretchable sub-conductors.

[0071] Take a scenario as an example. Figure 3 This is a schematic diagram of the structure of a flexible stretchable wire provided by an embodiment of the present disclosure. Figure 3 As shown, the flexible stretchable conductor is a serpentine flexible stretchable conductor, which includes four parallel serpentine flexible stretchable sub-conductors, and the conductor arc radii R corresponding to the four parallel serpentine flexible stretchable sub-conductors are R1, R2, R3, and R4 respectively.

[0072] According to some embodiments, after testing, when the wire width w of each serpentine flexible stretchable sub-conductor in the four parallel serpentine flexible stretchable sub-conductors corresponds to a width range of 0.2 mm to 0.3 mm, the performance and flexible stretchability of the serpentine flexible stretchable conductor can be taken into account.

[0073] Optionally, Figure 4 A cross-sectional view of a flexible and stretchable sub-conductor provided by an embodiment of the present disclosure. Figure 4 As shown, the flexible stretchable sub-conductor includes a first polyimide layer 121, a copper layer 122 and a second polyimide layer 123 which are stacked in sequence.

[0074] According to some embodiments, the flexible stretchable sub-conductor is composed of three layers of polyimide (PI), copper (Cu), and polyimide (PI), and is processed by a flexible circuit board process and then obtained by laser cutting. Therefore, it can take into account both the performance of the serpentine flexible stretchable conductor and the flexible stretchability.

[0075] Optionally, the flexible image sensor array module 10 further includes a first polydimethylsiloxane film 13 and a second polydimethylsiloxane film 14; wherein,

[0076] In the island-bridge structure, the rigid image sensors 11 connected to each other and the rigid image sensors 11 between the flexible stretchable wires 12 are welded to the first side of the flexible stretchable wires 12;

[0077] The first polydimethylsiloxane film 13 is bonded to the first side of the flexible stretchable wire 12 by pressure, and the second polydimethylsiloxane film 14 is bonded to the second side of the flexible stretchable wire 12 by pressure, and the first polydimethylsiloxane film 13 includes a plurality of holes, and the positions of the holes correspond to the positions of the rigid image sensors 11 one by one, so that the rigid image sensors 11 pass through the first polydimethylsiloxane film 13 through the holes;

[0078] The rigid image sensor 11 passing through the first PDMS film 13 is poured with PDMS material around it.

[0079] According to some embodiments, in the flexible image sensor array module 10 , upper and lower elastic polydimethylsiloxane (PDMS) films are used as a substrate and a protective layer, respectively, which can be used to disperse stress to protect the rigid image sensor 11 and the flexible stretchable wire 12 .

[0080] According to some embodiments, Figure 5 The following is a flow chart of preparing a flexible image sensor array module provided by an embodiment of the present disclosure. Figure 5 As shown, the specific steps include:

[0081] S101, such as Figure 5 As shown in (a), the flexible stretchable wire 12 is designed and prepared, wherein a pad is provided at one end of each flexible stretchable wire connected to the rigid image sensor 11;

[0082] S102, such as Figure 5 As shown in (b), the rigid image sensor 11 is welded to the pad corresponding to the flexible stretchable wire 12;

[0083] S103, such as Figure 5 As shown in (c), two PDMS films are taken, and a hole is punched at the corresponding position of the rigid image sensor 11 in the upper PDMS film to obtain a first polydimethylsiloxane film 13, and the other is used as a second polydimethylsiloxane film 14; then, an oxygen plasma surface treatment device is used to clean and activate the surfaces of the first polydimethylsiloxane film 13, the second polydimethylsiloxane film 14 and the flexible stretchable conductor 12; finally, pressurization is performed to achieve bonding between the first polydimethylsiloxane film 13, the second polydimethylsiloxane film 14 and the flexible stretchable conductor 12;

[0084] S104, such as Figure 5 As shown in (d) in FIG. 1 , PDMS is poured around the rigid image sensor 11 to reinforce the local area.

[0085] It should be noted that if Figure 5As shown in (d) in FIG. 8 , a thin lens 15 is provided on a side of the rigid image sensor 11 away from the flexible stretchable wire 12 .

[0086] In some embodiments, the focal length of the thin lens 15 can be adjusted according to the actual working scenario of the flexible self-deformable array light field intelligent imaging system.

[0087] Optionally, the flexible self-deformable array light field intelligent imaging system further includes a data readout module 20 and a light field processing module 30; wherein,

[0088] A data reading module 20 is used to read the image data set collected by the plurality of rigid image sensors 11 and send the image data set to the light field processing module 30;

[0089] The light field processing module 30 is used to reconstruct the ambient light field according to the image data set to obtain ambient light field information.

[0090] According to some embodiments, the data readout module 20 may also be responsible for synchronizing the signal reading of the rigid image sensor 11 and controlling the rigid image sensor 11 .

[0091] Optionally, Figure 6 This is a schematic diagram of a design of a flexible stretchable wire provided by an embodiment of the present disclosure. Figure 6 As shown, in the design of the flexible stretchable conductor 12, two modes of parallel readout and serial readout can be adopted.

[0092] like Figure 6 As shown in (a), each rigid image sensor 11 is connected to the data readout module 20 via a flexible stretchable wire 12, so that the data readout module 20 reads out data of all rigid image sensors 11 in parallel to obtain an image data set. Therefore, the data readout speed of the flexible image sensor array module 10 can be maximized, and high-speed parallel readout of sensor data can be achieved.

[0093] or,

[0094] The plurality of rigid image sensors 11 are connected to the data readout module 20 via a flexible stretchable wire 12, so that the data readout module 20 serially reads data from all the rigid image sensors 11 to obtain an image data set. Figure 6 As shown in (b), there are six rigid image sensors 11 in total, and every three rigid image sensors 11 are connected to the data readout module 20 via a flexible stretchable wire 12. Therefore, the area of ​​the flexible stretchable wire 12 can be reduced, and the density of the flexible image sensor array module 10 can be increased.

[0095] According to some embodiments, when the data readout module 20 performs serial data readout on all rigid image sensors 11 , the data readout may be performed in a time division multiplexing manner.

[0096] Optionally, Figure 7 Schematic diagram of the structure of a data reading module provided by an embodiment of the present disclosure. Figure 7 As shown, the data readout module 20 includes a sensor control submodule 21, a multi-sensor synchronization submodule 22 and an image signal processing submodule 23; wherein,

[0097] A sensor control submodule 21, used to control sensor parameters of the rigid image sensor 11;

[0098] A multi-sensor synchronization submodule 22 is used to read the image data set collected by the multiple rigid image sensors 11 and synchronize the data between the multiple rigid image sensors 11;

[0099] The image signal processing submodule 23 is used to optimize the image data set, obtain the optimized image data set and send it to the light field processing module 30 .

[0100] According to some embodiments, sensor parameters include but are not limited to parameters such as exposure and gain.

[0101] In some embodiments, when the multi-sensor synchronization submodule 22 reads the image data set collected by the multiple rigid image sensors 11, the interface form used includes but is not limited to analog, MIPI and other interface forms.

[0102] According to some embodiments, when the image signal processing submodule 23 performs optimization processing on the image data set, the image data set may be processed and optimized in real time, thereby improving the image quality corresponding to the image data set and enhancing the details.

[0103] In some embodiments, the optimization processing methods include but are not limited to denoising, white balance, etc.

[0104] According to some embodiments, when the image signal processing submodule 23 sends the optimized image data set to the light field processing module 30, the interface used includes but is not limited to universal interfaces such as Cameralink and USB.

[0105] In some embodiments, the optimized image data set may be, for example, a light field array image (also called an array sensor image).

[0106] Optionally, the light field processing module 30 includes an environment perception submodule 31 and a body perception submodule 32; wherein,

[0107] The environment perception submodule 31 is used to extract semantic information of each image data in the image data set according to the visual basic model, and estimate the first three-dimensional depth information corresponding to each image data according to the semantic information;

[0108] The proprioceptive perception submodule 32 is used to establish a feature point association relationship between different image data in the image data set; perform array image-light field three-dimensional joint optimization according to the first three-dimensional depth information and the feature point association relationship to obtain the posture information corresponding to each image data and the second three-dimensional depth information; and reconstruct the proprioceptive posture corresponding to the flexible image sensor array module according to the posture information;

[0109] The environment perception submodule 31 is further used to obtain the environment light field information by fusing the main body posture and the second three-dimensional depth information.

[0110] According to some embodiments, the visual base model may be, for example, a large model.

[0111] In some embodiments, Figure 8 This is a workflow diagram of a light field processing module provided by an embodiment of the present disclosure. Figure 8 As shown, the environment perception submodule 31 and the proprioception submodule 32 provide feedback to each other, and the specific process is as follows:

[0112] S201, the environment perception submodule 31 extracts semantic information from the array sensor image using a visual basic model;

[0113] S202, the environment perception submodule 31 uses semantic information to perform light field 3D estimation based on the monocular image to obtain monocular 3D depth information of each image, wherein the monocular image refers to any image in the array sensor image; for example, this can be achieved by using visual basic model technologies such as monocular image depth estimation, or by training a special monocular 3D estimation neural network;

[0114] S203, the proprioceptive perception submodule 32 establishes a characteristic point association relationship between array sensor images;

[0115] S204, the proprioceptive perception submodule 32 establishes a physical constraint relationship according to the feature association between the array sensor images and the monocular 3D depth information of each image, performs array image-light field 3D joint optimization based on the camera projection model, and jointly optimizes the posture information of each rigid image sensor 11 and the corresponding monocular image 3D depth information;

[0116] S205, the body perception submodule 32 reconstructs the body posture of the flexible self-deformable array light field intelligent imaging system according to the posture information of the rigid image sensor 11;

[0117] S206, the environment perception submodule 31 obtains the environment light field information by fusing the body posture and the optimized three-dimensional depth information of the monocular image.

[0118] According to some embodiments, Fig. 9 A schematic diagram of imaging results of a flexible self-deformable array light field intelligent imaging system provided in an embodiment of the present disclosure. Fig. 9 (a) in FIG. 1 shows a set of image data collected by the rigid image sensor 11. Fig. 9 (b) in the figure shows the light field reconstruction result obtained by reconstructing the ambient light field from the image data set.

[0119] It is easy to understand that the light field processing module 30 extracts the semantics and three-dimensional information of scene and instance-level objects, establishes physical constraint relationships, and simultaneously optimizes the system body posture and ambient light field information based on the camera projection model, thereby realizing variable system morphology and intelligent reconstruction of the light field without calibration parameters.

[0120] Take a scenario as an example. Fig.10 This is a schematic diagram of an application of a flexible self-deformable array light field intelligent imaging system provided by an embodiment of the present disclosure. Fig.10 As shown, the flexible self-deformable array light field intelligent imaging system has the characteristics of soft and deformable system and flexible and reconfigurable functions.

[0121] like Fig.10 As shown in (a), thanks to the soft and stretchable characteristics of the flexible stretchable wire 12, the flexible self-deformable array light field intelligent imaging system has the characteristics that the entire system is still flexible, stretchable and deformable, can be intelligently transformed into different shapes, and can adapt to surfaces of complex shapes. It has great potential in the fields of robots and wearables.

[0122] like Fig.10 As shown in (b) and (c) in the figure, in the robot application scenario, the flexible self-deformable array light field intelligent imaging system can be used to switch between wide field of view search, high-precision reconstruction, target positioning and tracking and other functions by changing its own shape. At the same time, the flexible self-deformable array light field intelligent imaging system has very high flexibility. It can use its deformation ability to pass through various narrow spaces, and then expand to improve the imaging ability, so as to achieve high integrity and high precision reconstruction of the ambient light field.

[0123] In summary, the system provided in this embodiment adopts a rigid-flexible "island-bridge" technology route, with a small-target rigid CMOS image sensor as the "island", and with the help of mature processes, the system has high pixel resolution, and a flexible stretchable wire as the "bridge", so that the system as a whole has the characteristics of being soft, stretchable and deformable. Secondly, through the data readout module and the light field processing module, combined with the monocular depth estimation capability of the visual basic model (large model) and the traditional camera projection model, variable forms are achieved, and proprioception (flexible self-deformation system proprioception posture reconstruction) and environmental perception (environmental light field reconstruction) without calibration parameters can be achieved, which can achieve high integrity and high precision reconstruction of the ambient light field.

[0124] To implement the above embodiments, the present disclosure also provides a flexible self-deformable array light field intelligent imaging device, including: the flexible self-deformable array light field intelligent imaging system provided by the above embodiments. The flexible self-deformable array light field intelligent imaging device can be, for example, an electronic device.

[0125] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this disclosure shall comply with the relevant laws and regulations and shall not violate public order and good morals.

[0126] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0127] The present disclosure anticipates providing implementation schemes for users to selectively block the use or access of personal information data. That is, the present disclosure anticipates providing hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, risks can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0128] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0129] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0130] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.

[0131] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0132] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0133] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0134] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0135] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.

Claims

1. A flexible self-deformable array light field intelligent imaging system, characterized in that: include: A flexible image sensor array module, the flexible image sensor array module comprising a plurality of rigid image sensors and a plurality of flexible stretchable wires; The plurality of rigid image sensors and the plurality of flexible stretchable wires are connected to each other, with the rigid image sensors serving as islands and the flexible stretchable wires serving as bridges, to form an island-bridge structure; The flexible self-deformable array light field intelligent imaging system also includes a data readout module and a light field processing module; wherein, The data reading module is used to read the image data set collected by the multiple rigid image sensors and send the image data set to the light field processing module; The light field processing module is used to reconstruct the ambient light field according to the image data set to obtain ambient light field information; The light field processing module includes an environment perception submodule and a body perception submodule; wherein, The environment perception submodule is used to extract semantic information of each image data in the image data set according to the visual basic model, and estimate first three-dimensional depth information corresponding to each image data according to the semantic information; The body perception submodule is used to establish a feature point association relationship between different image data in the image data set; perform array image-light field three-dimensional joint optimization according to the first three-dimensional depth information and the feature point association relationship to obtain the posture information corresponding to each image data and the second three-dimensional depth information; and reconstruct the body posture corresponding to the flexible image sensor array module according to the posture information; The environment perception submodule is further used to obtain the environment light field information by fusing the body posture and the second three-dimensional depth information.

2. The system according to claim 1, characterized in that The target surface of the rigid image sensor is smaller than a target surface threshold.

3. The system according to claim 1, characterized in that The flexible stretchable conductor is a serpentine flexible stretchable conductor; the serpentine flexible stretchable conductor includes multiple parallel serpentine flexible stretchable sub-conductors; the serpentine flexible stretchable sub-conductors include a first polyimide layer, a copper layer and a second polyimide layer stacked in sequence.

4. The system according to claim 3, characterized in that The serpentine flexible stretchable conductor comprises four parallel serpentine flexible stretchable sub-conductors, and the conductor width of the serpentine flexible stretchable sub-conductors corresponds to a width range of 0.2 mm to 0.3 mm.

5. The system according to claim 1, characterized in that The flexible image sensor array module further includes a first polydimethylsiloxane film and a second polydimethylsiloxane film; wherein, In the island bridge structure, the interconnected rigid image sensors and the rigid image sensors between the flexible stretchable wires are welded on the first side of the flexible stretchable wires; The first polydimethylsiloxane film is bonded to the first side of the flexible stretchable wire by pressure, the second polydimethylsiloxane film is bonded to the second side of the flexible stretchable wire by pressure, and the first polydimethylsiloxane film includes a plurality of holes, and the positions of the holes correspond to the positions of the rigid image sensors one by one, so that the rigid image sensor passes through the first polydimethylsiloxane film through the holes; Polydimethylsiloxane material is poured around the rigid image sensor that passes through the first polydimethylsiloxane film.

6. The system according to claim 1, characterized in that in, Each of the rigid image sensors is connected to the data readout module via a flexible stretchable wire, so that the data readout module reads out data from all rigid image sensors in parallel to obtain the image data set; or, The plurality of rigid image sensors are connected to the data readout module via a flexible stretchable wire, so that the data readout module serially reads out data from all rigid image sensors to obtain the image data set.

7. The system according to claim 1, characterized in that The data readout module includes a sensor control submodule, a multi-sensor synchronization submodule and an image signal processing submodule; wherein, The sensor control submodule is used to control sensor parameters of the rigid image sensor; The multi-sensor synchronization submodule is used to read the image data set collected by the multiple rigid image sensors and synchronize the data between the multiple rigid image sensors; The image signal processing submodule is used to optimize the image data set to obtain and send the optimized image data set to the light field processing module.

8. A flexible self-deformable array light field intelligent imaging device, characterized in that: include: A flexible self-deformable array light field intelligent imaging system as claimed in any one of claims 1 to 7.

Images