Image acquisition device, image acquisition method, electronic equipment and storage medium

Abstract

The invention relates to the technical field of image acquisition, in particular to an image acquisition device, an image acquisition method, electronic equipment and a storage medium. According to the embodiment of the invention, camera calibration parameters need to be acquired through the image acquisition calibration module, the trigger signals are generated in the multi-camera control module, and the trigger signals are respectively sent to the light field camera imaging module and the event camera imaging module; the light field camera imaging module and the event camera imaging module carry out data acquisition in the same detection time slot; in a light field camera imaging module, collecting light field data of a target scene in a detection time slot according to the trigger signal; in an event camera imaging module, event data of a target scene are collected in a detection time slot according to the trigger signal; and in the data fusion module, image acquisition fusion data is generated according to the camera calibration parameters, the light field data and the event data. Therefore, high-quality real-time imaging can be realized.

Description

Technical Field

[0001] The present application relates to the technical field of image acquisition, and in particular, to an image acquisition device, an image acquisition method, an electronic device, and a storage medium. Background Art

[0002] The light field camera and the event camera respectively correspond to two new imaging technologies, each with unique advantages. The light field camera can record the angular information of light and realize functions such as multi-angle imaging, depth recovery, and three-dimensional imaging. The event camera is extremely sensitive to motion or light changes and has high sensitivity and event capture capabilities.

[0003] In related technologies, attempts have been made to combine the advantages of the light field camera and the event camera to form an event light field camera with an integrated optical path and sensor. It is precisely because such an event light field camera shares the optical path and the image sensor that, restricted by the event camera image sensor process, the resolution is low, and the light field imaging field of view and depth information capture capabilities are limited, making it difficult to meet the actual application requirements. Summary of the Invention

[0004] The present application aims to at least solve one of the technical problems existing in the prior art. For this purpose, the present application provides an image acquisition device, an image acquisition method, an electronic device, and a storage medium, which can achieve high-quality real-time imaging.

[0005] An image acquisition device according to an embodiment of the first aspect of the present application includes an image acquisition calibration module, a multi-camera control module, a light field camera imaging module, an event camera imaging module, and a data fusion module;

[0006] The image acquisition calibration module is used to obtain camera calibration parameters;

[0007] The multi-camera control module is used to generate a trigger signal and send the trigger signal to the light field camera imaging module and the event camera imaging module respectively, so that the light field camera imaging module and the event camera imaging module perform data acquisition in the same detection time slot;

[0008] The light field camera imaging module is used to collect light field data of the target scene in the detection time slot according to the trigger signal;

[0009] The event camera imaging module is used to collect event data of the target scene in the detection time slot according to the trigger signal;

[0010] The data fusion module is used to generate image acquisition fusion data according to the camera calibration parameters, the light field data, and the event data.

[0011] According to some embodiments of the present application, it includes: a base, a limit plate, a support slide rail, and a multi-camera control module;

[0012] The multi-camera control module is fixed to the base;

[0013] The support slide rails are respectively fixed on both sides of the base, and the light field camera imaging module and the event camera imaging module are arranged on the support slide rails;

[0014] The limit plate is fixed at one end of the support slide rail to prevent the light field camera imaging module and the event camera imaging module from overtraveling and falling off when sliding on the support slide rail.

[0015] An image acquisition method according to an embodiment of the second aspect of the present application is applied to an image acquisition device, and includes:

[0016] Obtain camera calibration parameters through an image acquisition calibration module;

[0017] In the multi-camera control module, generate a trigger signal and send the trigger signal to the light field camera imaging module and the event camera imaging module respectively, so that the light field camera imaging module and the event camera imaging module perform data acquisition in the same detection time slot;

[0018] In the light field camera imaging module, collect light field data of the target scene according to the trigger signal in the detection time slot;

[0019] In the event camera imaging module, collect event data of the target scene according to the trigger signal in the detection time slot;

[0020] In the data fusion module, generate image acquisition fusion data according to the camera calibration parameters, the light field data, and the event data.

[0021] According to some embodiments of the present application, generating the image acquisition fusion data according to the camera calibration parameters, the light field data, and the event data includes:

[0022] Perform the same formatting process on the light field data and the event data of the target scene to obtain multi-dimensional image acquisition data; wherein, the multi-dimensional image acquisition data includes multi-dimensional light field information and multi-dimensional event information;

[0023] Perform scene depth estimation based on the camera calibration parameters and the multi-dimensional light field information to obtain scene depth estimation information;

[0024] Perform enhanced fusion processing on the multi-dimensional image acquisition data according to the camera calibration parameters, the scene depth estimation information, and the multi-dimensional event information to generate the image acquisition fusion data.

[0025] According to some embodiments of the present application, the multi-dimensional light field information includes light intensity acquisition sub-data, direction acquisition sub-data, and position acquisition sub-data. The scene depth estimation is performed based on the camera calibration parameters and the multi-dimensional light field information to obtain scene depth estimation information, including:

[0026] Generate multi-view parallax images of the target scene according to the camera calibration parameters, the light intensity acquisition sub-data, the direction acquisition sub-data, and the position acquisition sub-data;

[0027] Perform parallax depth estimation based on the multi-view parallax images to generate the scene depth estimation information.

[0028] According to some embodiments of the present application, the multi-dimensional image acquisition data further includes event acquisition sub-data. The parallax depth estimation is performed based on the multi-view parallax images to generate the scene depth estimation information, including:

[0029] Perform scene depth analysis according to the multi-view parallax images to obtain scene depth analysis information;

[0030] Correct the scene depth analysis information according to the event acquisition sub-data to obtain the scene depth estimation information.

[0031] According to some embodiments of the present application, the multi-dimensional image acquisition data is enhanced and fused based on the camera calibration parameters, the scene depth estimation information, and the multi-dimensional event information to generate the image acquisition fusion data, including:

[0032] Fuse the multi-dimensional light field information and the multi-dimensional event information according to the camera calibration parameters to obtain preliminary acquisition image fusion data;

[0033] Perform depth of field optimization processing on the preliminary image fusion data according to the scene depth estimation information to obtain intermediate acquisition image fusion data;

[0034] Perform depth of field restoration processing on the intermediate acquisition image fusion data according to the multi-dimensional event information to obtain the image acquisition fusion data.

[0035] According to some embodiments of the present application, the multi-dimensional light field information corresponds to the light field camera coordinate system, and the multi-dimensional event information corresponds to the event camera coordinate system. The multi-dimensional light field information and the multi-dimensional event information are fused according to the camera calibration parameters to obtain preliminary acquisition image fusion data, including:

[0036] Perform time alignment processing on the multi-dimensional light field information and the multi-dimensional event information;

[0037] After time alignment processing, the multi-dimensional light field information is mapped into the event camera coordinate system according to the camera calibration parameters, or the multi-dimensional event information is mapped into the light field camera coordinate system, so as to perform fusion processing on the multi-dimensional light field information and the multi-dimensional event information, and obtain the preliminary acquisition image fusion data.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the image acquisition method according to any one of the embodiments in the second aspect of the present application.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, the storage medium stores a program, and when the program is executed by a processor, it implements the image acquisition method according to any one of the embodiments in the second aspect of the present application.

[0040] The image acquisition device, image acquisition method, electronic device, and storage medium according to the embodiments of the present application at least have the following beneficial effects:

[0041] In the embodiment of the present application, it is necessary to obtain camera calibration parameters through an image acquisition calibration module. In the multi-camera control module, a trigger signal is generated and sent to the light field camera imaging module and the event camera imaging module respectively, so that the light field camera imaging module and the event camera imaging module perform data acquisition in the same detection time slot; in the light field camera imaging module, light field data of the target scene is acquired in the detection time slot according to the trigger signal; in the event camera imaging module, event data of the target scene is acquired in the detection time slot according to the trigger signal; in the data fusion module, image acquisition fusion data is generated according to the camera calibration parameters, the light field data, and the event data. In this way, high-quality real-time imaging can be achieved.

[0042] The additional aspects and advantages of the present application will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present application. Description of the Drawings

[0043] The above and / or additional aspects and advantages of the present application will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, where:

[0044] Figure 1 is a schematic structural diagram of an image acquisition device according to some embodiments of the present application from one perspective;

[0045] Figure 2 is a schematic structural diagram of an image acquisition device according to some embodiments of the present application from another perspective;

[0046] Figure 3 Structural schematic diagram of the moving frame in the image acquisition device according to some embodiments of the present application;

[0047] Figure 4 A flowchart of an image acquisition method provided by an embodiment of the present application;

[0048] Figure 5 Another flowchart of the image acquisition method provided by an embodiment of the present application;

[0049] Figure 6 Another flowchart of the image acquisition method provided by an embodiment of the present application;

[0050] Figure 7 Another flowchart of the image acquisition method provided by an embodiment of the present application;

[0051] Figure 8 Another flowchart of the image acquisition method provided by an embodiment of the present application;

[0052] Figure 9 Another flowchart of the image acquisition method provided by an embodiment of the present application;

[0053] Figure 10 Data fusion schematic diagram between the light field data and the event data provided by an embodiment of the present application;

[0054] Figure 11 Structural schematic diagram of the electronic device provided by an embodiment of the present application. Detailed implementation manners

[0055] The embodiments of the present application are described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and should not be construed as a limitation to the present application.

[0056] In the description of the present application, the meaning of several is one or more, the meaning of multiple is more than two, greater than, less than, exceeding, etc. are understood as not including the present number, and above, below, within, etc. are understood as including the present number. If the first and the second are described only for the purpose of distinguishing technical features, they should not be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence of the indicated technical features.

[0057] In the description of the present application, it should be understood that for the orientation description, such as the orientation or positional relationship indicated by up, down, left, right, front, back, etc., is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present application.

[0058] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "schematic embodiments", "examples", "specific examples", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0059] In the description of the present application, it should be noted that unless otherwise clearly defined, terms such as "set", "installed", "connected", etc. should be understood in a broad sense. Those skilled in the art can reasonably determine the specific meanings of the above terms in the present application in combination with the specific content of the technical solution. In addition, the identification of specific steps hereinafter does not represent a limitation on the step sequence and execution logic. The execution sequence and execution logic between each step should be understood and inferred with reference to the content described in the embodiments.

[0060] A light field camera is a new type of imaging technology. Traditional imaging can only obtain the brightness values recorded by the light captured by the lens on the image sensor. Compared with traditional imaging, a light field camera can also record the angular information of light. Therefore, a light field camera can achieve multiple functions including multi-angle imaging, depth recovery, and three-dimensional imaging. Among them, a microlens array is a common implementation of a light field camera.

[0061] An event camera is a new type of imaging technology. It records the change information of the same pixel on the image sensor at the previous and subsequent moments due to movement or light changes, and has extremely high sensitivity and event capture ability.

[0062] How to combine the advantages of event camera imaging and light field camera imaging? An advanced imaging device that combines the advantages of event cameras and light field cameras can simultaneously capture the light intensity, direction information, and dynamic changes of a scene, and has extremely high temporal and spatial resolutions. However, it is quite difficult to achieve a good combination of event camera imaging and light field camera imaging.

[0063] In some related technologies, an attempt is made to place the microlens array of a light field camera between the event camera sensor and the main lens, use the microlens array to perform secondary imaging on the light information, and use the same optical path and image sensor. However, precisely because this related technology shares an optical path and an image sensor between the event camera and the light field camera, due to the process limitations of the event camera image sensor, the resolution of the image sensor is only in the range of several hundred thousand pixels to several million pixels, and it is currently difficult for the industry to provide event camera image sensors with higher resolutions. Moreover, the limitation of the number of microlenses in the light field microlens array also affects the field of view and the ability to capture depth information of light field imaging, further reducing the performance of the system in monitoring fast-moving objects and making it unable to have practical application value.

[0064] In some other related technologies, a relay imaging system is used to first reduce the field of view of the scene and then perform secondary imaging on the event camera sensor through the main lens and the microlens. However, this related technology still uses the same optical path and image sensor and has not broken through the limitations of the event camera image sensor process and the number of microlens arrays.

[0065] In summary, in related technologies, the imaging method of the event light field camera with integrated optical path and sensor has many defects in application scenarios such as high resolution, high speed, and high dynamic range, and is restricted by the event camera image sensor process, making it difficult to be put into practical use.

[0066] This application aims to solve at least one of the technical problems existing in the prior art. For this purpose, this application proposes an image acquisition device, an image acquisition method, an electronic device, and a storage medium, which can achieve high-quality real-time imaging.

[0067] The following will be further described with reference to the accompanying drawings.

[0068] The image acquisition device according to an embodiment of this application may include an image acquisition calibration module, a multi-camera control module, a light field camera imaging module, an event camera imaging module, and a data fusion module;

[0069] The image acquisition calibration module is used to obtain camera calibration parameters;

[0070] The multi-camera control module is used to generate a trigger signal and send the trigger signal to the light field camera imaging module and the event camera imaging module respectively, so that the light field camera imaging module and the event camera imaging module perform data acquisition in the same detection time slot;

[0071] The light field camera imaging module is used to collect light field data of the target scene in the detection time slot according to the trigger signal;

[0072] The event camera imaging module is used to collect event data of the target scene in the detection time slot according to the trigger signal;

[0073] A data fusion module, configured to generate image acquisition fusion data according to camera calibration parameters, light field data, and event data.

[0074] According to some more specific embodiments of the present application, the image acquisition device includes: a base, a limit plate, support slide rails, and a multi-camera control module;

[0075] The multi-camera control module is fixed to the base;

[0076] Support slide rails are respectively fixed on both sides of the base, and the light field camera imaging module and the event camera imaging module are arranged on the support slide rails;

[0077] The limit plate is fixed to one end of the support slide rail to prevent the light field camera imaging module and the event camera imaging module from falling off due to overtravel when sliding on the support slide rail.

[0078] Referring to Figures 1 to 3 , in some more specific embodiments of the present application, the image acquisition device may include:

[0079] A camera assembly, a base 101, a limit plate 102, support slide rails 103, a fixed seat 104, a folding frame 105, a hinge seat 106, a multi-camera control module 107, a display screen 108, a first adjustment knob 109, a second adjustment knob 110, and a device power supply 111. Among them, the camera assembly includes a light field camera imaging module, an event camera imaging module, two adjusters 301, a coarse adjustment knob 306, and a fine adjustment knob 307. Among them, the light field camera imaging module may include a light field camera 303 and a light field camera lens 304, and the event camera imaging module may include an event camera 302 and an event camera lens 305. It should be noted that the two adjusters 301 are respectively fixed to the upper ends of the two moving frames 201, the event camera 302 is fixed to the upper end of the adjuster 301, the event camera lens 305 is installed on the event camera 302, the light field camera 303 is fixed to the other adjuster 301, the light field camera lens 304 is installed on the light field camera 303, and the coarse adjustment knob 306 and the fine adjustment knob 307 are respectively fixed to the adjuster 301, and the longitudinal displacement of the camera is completed through the coarse adjustment knob 306 and the fine adjustment knob 307.

[0080] It should be noted that the two support slide rails 103 are respectively fixed on both sides of the base 101, the limit plate 102 is fixed to one end of the support slide rail 103, the fixed seat 104 is fixed at the geometric center of the other side of the base 101, one end of the folding frame 105 is hinged to the fixed seat 104, the other end of the folding frame 105 is hinged to the hinge seat 106, and the multi-camera control module 107 is fixed to the hinge seat 106.

[0081] It should be noted that the display screen 108 is fixed on the multi-camera control module 107. The lower knobs are installed in the order of the first adjustment knob 109, the second adjustment knob 110, and the device power supply 111. The limit plate 102 prevents the camera assembly from falling off due to overtravel when sliding on the support slide rail 103. The multi-camera control module 107 can be moved through the folding frame 105, so that the movement of the multi-camera control module 107 can better adapt to the working environment.

[0082] The moving frame 201, the transverse roller 202, the first roller 203, the set screw 204, the second roller 205, and the roller 206. A pair of transverse rollers 202 are respectively fixed on both sides of the moving frame 201. A pair of first rollers 203 are respectively rotatably connected to the transverse rollers 202. A pair of set screws 204 cooperate with the threaded holes on both sides of the moving frame 201. A pair of rollers 206 are respectively fixed at the geometric center of the lower end of the moving frame 201. A pair of rollers 205 are respectively rotatably connected to the rollers 206. The moving frame 201 is located above the base 101. The first roller 203 is in contact with the support slide rail 103. The roller 205 is tangent to the base 101 of the support slide rail 103 at the same time. Sliding the moving frame 201 can be used to adjust the camera assembly to better fit the working environment. The first roller 203 supports the entire camera assembly. The second roller 205 plays a limiting role to prevent the moving frame 201 from generating circumferential offset in the support slide rail. The set screw 204 is responsible for restricting the axial movement of the moving frame. After adjusting the position of the camera assembly, tighten the set screw 204 to complete the fixing work of the camera assembly.

[0083] Referring to Figure 4 , according to the image acquisition method of the second aspect embodiment of the present application, which is applied to an image acquisition device, it may include:

[0084] Step S401, obtaining camera calibration parameters through the image acquisition calibration module;

[0085] Step S402, in the multi-camera control module, generating a trigger signal and respectively sending the trigger signal to the light field camera imaging module and the event camera imaging module, so that the light field camera imaging module and the event camera imaging module perform data acquisition in the same detection time slot;

[0086] Step S403, in the light field camera imaging module, collecting light field data of the target scene according to the trigger signal in the detection time slot;

[0087] Step S404, in the event camera imaging module, collecting event data of the target scene according to the trigger signal in the detection time slot;

[0088] Step S405, in the data fusion module, generating image acquisition fusion data according to the camera calibration parameters, the light field data, and the event data.

[0089] In step S401 of some embodiments, camera calibration parameters are obtained through an image acquisition and calibration module;

[0090] In the embodiments of the present application, obtaining camera calibration parameters through an image acquisition and calibration module can solve the problems of low resolution, limited light field imaging field of view, and limited depth information capture ability caused by the shared optical path and image sensor when combining a light field camera and an event camera. Specifically, the image acquisition and calibration module can accurately obtain the internal and external parameters of the light field camera and the event camera, which may include focal length, principal point position, rotation matrix, translation vector, etc. These internal and external parameters of the light field camera and the event camera ensure the geometric relationship and time synchronization during imaging of the light field camera and the event camera, thereby improving the overall performance of the image acquisition device.

[0091] It should be noted that according to some more specific embodiments of the present application, in step S401, obtaining camera calibration parameters through an image acquisition and calibration module can utilize the multi-camera control module and adopt the checkerboard calibration method to perform data acquisition work on the checkerboard at the same moment, so as to calibrate the light field camera imaging module and the event camera imaging module using the Zhang Zhengyou calibration method to ensure the geometric relationship and internal and external parameter information between different imaging modules, and ensure the accurate alignment of subsequent event information and light field information.

[0092] Specifically, an event camera records events based on pixel-level brightness changes, and the data it captures does not depend on frames but is recorded in the form of events (xe, ye, te, pe), where xe and ye are the pixel coordinates of the event. te is the timestamp of the event occurrence. pe is the polarity of the event, indicating an increase (positive) or decrease (negative) in brightness.

[0093] Although an event camera does not have a traditional frame, an image can be simulated by constructing an event surface (by accumulating events or mapping them to a virtual frame), thereby introducing a camera model similar to that of a light field camera.

[0094] It should be understood that due to the different working principles of the event camera and the light field camera, it is necessary to ensure their time synchronization first:

[0095] The data generated by the event camera has extremely high time resolution, so a high-precision timestamp alignment mechanism is required. Let the time offset between the two be Δt, and through high-precision hardware synchronization, ensure that the data of the event camera and the light field camera can be synchronized in the time dimension.

[0096]

[0097] Among them, represents the timestamp of the light field camera, represents the timestamp of the event camera.

[0098] Furthermore, for spatial alignment: The purpose of calibration is to solve the relative pose between the event camera and the light field camera, that is, the rotation matrix and the translation vector. It can be described by the following formula:

[0099]

[0100] where (X event , Y event , Z event ) are the three-dimensional point coordinates of the event camera, and (X RGB , Y RGB , Z RGB ) are the corresponding light field camera coordinates.

[0101] Since the camera adopts the pinhole imaging principle, for camera imaging, the feature points (u, v) on the image can be directly extracted and back-projected according to the internal parameters of the camera to obtain the three-dimensional point coordinates:

[0102]

[0103] By collecting a large number of corresponding point pairs (u RGB , v RGB ) and (x event , y event ). Then, by minimizing the reprojection error, the external parameters of the event camera and the light field camera can be solved, and the relative pose between the event camera and the light field camera can be obtained. The reprojection error is:

[0104]

[0105] In step S402 of some embodiments, in the multi-camera control module, a trigger signal is generated and sent to the light field camera imaging module and the event camera imaging module respectively, so that the light field camera imaging module and the event camera imaging module perform data acquisition in the same detection time slot;

[0106] It should be noted that in the multi-camera control module, a trigger signal is generated and sent to the light field camera imaging module and the event camera imaging module respectively, so that the light field camera imaging module and the event camera imaging module perform data acquisition in the same detection time slot. This process ensures the synchronous acquisition of light field data and event data, avoiding errors caused by asynchronous data acquisition times. Through precise trigger signal control, the light field camera and the event camera can perform data acquisition on the target scene at the same moment, thereby improving the accuracy and reliability of the data.

[0107] In step S403 of some embodiments, in the light field camera imaging module, light field data of the target scene is collected according to the trigger signal in the detection time slot;

[0108] It should be noted that in the light field camera imaging module, light field data of the target scene is collected in the detection time slot according to the trigger signal. The light field camera can record the light intensity, direction, and position information of light rays through the main lens and the microlens array, generating high-resolution light field data. The light field data collected from these target scenes contains rich spatial and light information of the target scene, providing a basis for subsequent depth recovery and three-dimensional imaging.

[0109] In some more specific embodiments, after the light field data is collected, it needs to be decoded.

[0110] Specifically, due to the secondary imaging of the light field camera on the microlens array, first, the relationship between the pixel and the corresponding microlens needs to be located:

[0111] It should be noted that for the calibration of the optical center of the microlens, the Canny edge detection algorithm can be used to extract the edge points in the light field data. The result of edge detection is a binary image. For example, the value of the edge points is 255, and the value of other points is 0; for each edge point (x i , y i ), the possible parameters of the circle are represented as:

[0112] (a, b, r)

[0113] where a and b are the coordinates of the center of the circle, and r is the radius. According to the equation of the circle, each edge point can be mapped to the parameter space.

[0114] Furthermore, when an edge point (x i , y i ) of a light field data matches a certain circle parameter (a, b, r), the voting number of this circle is increased. This process can be implemented using a three-dimensional array R(a, b, r), where each element represents the voting number of the corresponding parameter.

[0115] Still further, the local maximum is found through voting, and then the parameters of the circle can be determined. By corresponding the pixel coordinates of the center of the microlens, the relationship between each microlens and the pixel can be located, and then the relationship between the microlens array and the pixel can be determined. In this way, the calibration of the microlens center is completed.

[0116] For the convenience of subsequent processing, the microlens array can be numbered, the position of the optical center of the image is calculated, and the distance from the center of each microlens to the optical center is calculated using the Euclidean distance. The microlens closest to the optical center is numbered (0, 0), and a coordinate system is established with the right side in the horizontal direction as the positive x direction and the upper side in the vertical direction as the positive y direction to number all the microlenses. In this way, the decoding process of the light field data in some embodiments is completed.

[0117] In step S404 of some embodiments, in the event camera imaging module, event data of the target scene is collected according to a trigger signal during a detection time slot.

[0118] It should be noted that in the event camera imaging module, event data of the target scene is collected according to a trigger signal during a detection time slot. The event camera can record event data of brightness changes in the target scene, which may include the polarity, position, and timestamp of the event. These event information has extremely high temporal resolution, can capture rapid changes in the scene, and provides important support for depth recovery of dynamic scenes.

[0119] In step S405 of some embodiments, in the data fusion module, image acquisition fusion data is generated according to camera calibration parameters, light field data, and event data.

[0120] It should be noted that in the data fusion module, image acquisition fusion data needs to be generated according to camera calibration parameters, light field data, and event data. In some embodiments, the light field data and event data can be effectively integrated through an all-optical data fusion algorithm. Among them, the light field data provides high-resolution static image information, while the event data provides high-temporal-resolution dynamic change information. The fusion of the two not only improves the imaging quality and resolution, but also helps to improve the depth recovery accuracy of dynamic scenes in some embodiments. Through data fusion, it helps the image acquisition device of the present application to generate a more accurate and detailed three-dimensional model, providing strong support for subsequent three-dimensional reconstruction and scene understanding.

[0121] In summary, in the embodiments of the present application, the camera calibration parameters are obtained through the image acquisition calibration module, ensuring the precise alignment and synchronous acquisition of the light field camera and the event camera. The trigger signal is generated through the multi-camera control module to achieve the synchronous acquisition of the light field data and the event data. Through the data fusion module, the light field data and the event data are effectively integrated to generate high-quality image acquisition fusion data. This process not only solves the problems of low resolution, limited field of view, and limited depth information capture ability when the light field camera and the event camera are combined, but also improves the overall performance and application value of the image acquisition device.

[0122] Referring to Figure 5 , according to some embodiments of the present application, step S405 generates image acquisition fusion data according to camera calibration parameters, light field data, and event data, which may include:

[0123] Step S501, performing the same formatting process on the light field data and event data of the target scene to obtain multi-dimensional image acquisition data; wherein, the multi-dimensional image acquisition data includes multi-dimensional light field information and multi-dimensional event information;

[0124] Step S502: Perform scene depth estimation based on camera calibration parameters and multi-dimensional light field information to obtain scene depth estimation information;

[0125] Step S503: According to the camera calibration parameters, scene depth estimation information, and multi-dimensional event information, perform enhanced fusion processing on the multi-dimensional image acquisition data to generate image acquisition fusion data.

[0126] In step S501 of some embodiments, perform the same formatting process on the light field data and event data of the target scene to obtain multi-dimensional image acquisition data; among them, the multi-dimensional image acquisition data includes multi-dimensional light field information and multi-dimensional event information;

[0127] It should be noted that in the embodiments of the present application, the light field data and event data of the target scene are subjected to the same formatting process to obtain multi-dimensional image acquisition data. This step ensures the consistency of the formats of the light field data and event data, facilitating subsequent processing and fusion. The multi-dimensional image acquisition data includes multi-dimensional light field information and multi-dimensional event information. Among them, the light field information records the light intensity, direction, and position of the light rays, while the event information records the events of brightness changes in the scene, which can include the polarity, position, and timestamp of the events. Through the formatting process, these two data sources can be integrated into a unified data framework, laying a foundation for subsequent depth estimation and image enhancement fusion.

[0128] In step S502 of some embodiments, perform scene depth estimation based on camera calibration parameters and multi-dimensional light field information to obtain scene depth estimation information;

[0129] It should be noted that in the embodiments of the present application, scene depth estimation is performed based on camera calibration parameters and multi-dimensional light field information to obtain scene depth estimation information. The camera calibration parameters include the internal and external parameters of the light field camera and the event camera, such as focal length, principal point position, rotation matrix, and translation vector, etc. These camera calibration parameters ensure the geometric relationship and time synchronization of the light field camera and the event camera during imaging. By utilizing the multi-dimensional information in the light field data, a high-resolution depth map can be generated, providing a basis for subsequent image enhancement fusion. The depth estimation process infers the depth information of the objects in the scene by analyzing the parallax information in the light field data, thereby generating scene depth estimation information.

[0130] Refer to Figure 6 , according to some embodiments of the present application, the multi-dimensional light field information includes light intensity acquisition sub-data, direction acquisition sub-data, and position acquisition sub-data. Step S502 performs scene depth estimation based on camera calibration parameters and multi-dimensional light field information to obtain scene depth estimation information, which may include:

[0131] Step S601: Generate a multi-view disparity image of the target scene based on the camera calibration parameters, light intensity acquisition sub-data, direction acquisition sub-data, and position acquisition sub-data.

[0132] Step S602: Perform disparity depth estimation based on the multi-view disparity image to generate scene depth estimation information.

[0133] In some embodiments of the present application, the multi-dimensional light field information encompasses light intensity acquisition sub-data, direction acquisition sub-data, and position acquisition sub-data. These sub-data together constitute the multi-dimensional information of the scene captured by the light field camera, providing a rich data basis for subsequent scene depth estimation. Specifically, the light intensity acquisition sub-data records the brightness information of each pixel in the scene, the direction acquisition sub-data records the incident angle of the light rays, and the position acquisition sub-data records the specific position of the pixel on the image sensor. The combination of these information enables the light field camera to capture the multi-dimensional characteristics of the scene such as light intensity, direction, and position, making it possible to generate high-quality multi-view disparity images.

[0134] In some embodiments, Step S601: Generate a multi-view disparity image of the target scene based on the camera calibration parameters, light intensity acquisition sub-data, direction acquisition sub-data, and position acquisition sub-data.

[0135] It should be noted that the camera calibration parameters ensure the geometric relationship and time synchronization between the light field camera and the event camera during imaging, enabling the accurate alignment of images from different perspectives. By combining the light intensity, direction, and position information, the embodiments of the present application can generate multi-view images with disparity, which reflect the depth information of the objects in the scene and provide a basis for subsequent depth estimation.

[0136] In some embodiments, Step S602: Perform disparity depth estimation based on the multi-view disparity image to generate scene depth estimation information.

[0137] It should be noted that perform disparity depth estimation based on the multi-view disparity image to generate scene depth estimation information. Disparity depth estimation infers the depth of the objects in the scene by analyzing the disparity between images from different perspectives. Specifically, the embodiments of the present application calculate the distance between the object and the camera by comparing the position differences of the same object in images from different perspectives.

[0138] Refer to Figure 7 , according to some embodiments of the present application, the multi-dimensional image acquisition data further includes event acquisition sub-data. Step S602 performing disparity depth estimation based on the multi-view disparity image to generate scene depth estimation information may include:

[0139] Step S701: Perform scene depth analysis based on the multi-view disparity image to obtain scene depth analysis information.

[0140] Step S702: Correct the scene depth parsing information based on the event acquisition sub-data to obtain the scene depth estimation information.

[0141] In some embodiments of the present application, the multi-dimensional image acquisition data further includes event acquisition sub-data, which plays an important role in improving the accuracy of scene depth estimation. The event acquisition sub-data records the event information of the brightness change in the scene, including the polarity, position, and timestamp of the event. These information can capture the rapid changes in the scene and provide additional dynamic information for depth estimation.

[0142] In some embodiments, step S701: Perform scene depth parsing based on the multi-view parallax images to obtain the scene depth parsing information;

[0143] It should be noted that, by performing scene depth parsing based on the multi-view parallax images to obtain the scene depth parsing information. In this step, the embodiments of the present application infer the depth information of the objects in the scene by analyzing the parallax between the multi-view parallax images, and generate the preliminary scene depth parsing information.

[0144] In some embodiments, step S702: Correct the scene depth parsing information based on the event acquisition sub-data to obtain the scene depth estimation information.

[0145] It should be noted that the preliminary scene depth parsing information may be affected by various factors, such as light changes, object movements, etc., resulting in insufficient accuracy of depth estimation. To improve the accuracy of depth estimation, the embodiments of the present application further use the event acquisition sub-data to correct the scene depth parsing information. The brightness change event information in the event acquisition sub-data can provide accurate motion trajectories and brightness change information, which are crucial for correcting the depth parsing information. For example, in a fast dynamic scene, the event camera can capture the accurate motion trajectory of an object. By combining this motion information with the preliminary depth parsing information, the embodiments of the present application can correct the depth information to obtain a more accurate scene depth estimation information.

[0146] Specifically, the embodiments of the present application can determine the motion direction and speed of the objects in the scene according to the timestamp and position information in the event acquisition sub-data, and then adjust the preliminary depth parsing information. For example, if the event data indicates that an object is moving rapidly towards the camera, the embodiments of the present application can correspondingly adjust the depth estimation value of the object to make it more consistent with the actual motion situation. In addition, the event data can also be used to handle the influence of light changes on depth estimation. In a scene with drastic light changes, the event camera can capture the rapid change of brightness, and the embodiments of the present application can use this information to correct the depth estimation error caused by light changes.

[0147] Through the process shown in steps S701 to S702, the embodiments of the present application can generate more accurate and reliable scene depth estimation information by combining multi-view parallax images and event acquisition sub-data. This process not only makes full use of the multi-dimensional information of the light field camera, but also improves the accuracy and dynamic adaptability of depth estimation through the high temporal resolution data of the event camera, providing a solid foundation for subsequent image enhancement and fusion processing.

[0148] The process shown in steps S601 to S602 not only makes full use of the multi-dimensional information of the light field camera, but also ensures the accuracy and reliability of the data through the camera calibration parameters, providing a solid foundation for subsequent image enhancement and fusion processing.

[0149] In step S503 of some embodiments, according to the camera calibration parameters, scene depth estimation information, and multi-dimensional event information, the multi-dimensional image acquisition data is enhanced and fused to generate image acquisition fusion data.

[0150] It should be noted that in the embodiments of the present application, according to the camera calibration parameters, scene depth estimation information, and multi-dimensional event information, the multi-dimensional image acquisition data is enhanced and fused to generate image acquisition fusion data. In this step, it is necessary to effectively integrate the light field data and event data, and use the high temporal resolution of the event information and the high spatial resolution of the light field data to generate higher-quality image data. The event information can capture the rapid changes in the scene, provide accurate motion trajectories and brightness change information, thereby improving the accuracy and speed of depth recovery. Through the enhanced fusion processing, image acquisition fusion data containing rich details and dynamic information can be generated, providing strong support for subsequent high-level applications such as 3D reconstruction and object recognition.

[0151] Referring to Figure 8 , according to some embodiments of the present application, according to the camera calibration parameters, scene depth estimation information, and multi-dimensional event information, step S503 enhances and fuses the multi-dimensional image acquisition data to generate image acquisition fusion data, which may include:

[0152] Step S801, according to the camera calibration parameters, fuse the multi-dimensional light field information and multi-dimensional event information to obtain preliminary acquisition image fusion data;

[0153] Step S802, according to the scene depth estimation information, perform depth of field optimization processing on the preliminary image fusion data to obtain intermediate acquisition image fusion data;

[0154] Step S803, according to the multi-dimensional event information, perform depth of field recovery processing on the intermediate acquisition image fusion data to obtain image acquisition fusion data.

[0155] In some embodiments of the present application, step S503 generates image acquisition fusion data by performing enhanced fusion processing on multi-dimensional image acquisition data. This process makes full use of camera calibration parameters, scene depth estimation information, and multi-dimensional event information to improve the quality of image data and the accuracy of depth information.

[0156] In step S801 of some embodiments, according to the camera calibration parameters, the multi-dimensional light field information and the multi-dimensional event information are fused to obtain preliminary acquisition image fusion data;

[0157] It should be noted that according to the camera calibration parameters, the multi-dimensional light field information and the multi-dimensional event information are fused to obtain preliminary acquisition image fusion data. The camera calibration parameters ensure the geometric relationship and time synchronization between the light field camera and the event camera during imaging, enabling the data of both to be fused in the same coordinate system. The multi-dimensional light field information provides high-resolution static image data, including light intensity, direction, and position information, while the multi-dimensional event information provides high-time-resolution dynamic change data, including the polarity, position, and timestamp of the event. Through preliminary fusion, the embodiments of the present application can generate preliminary image fusion data containing static and dynamic information, laying a foundation for subsequent optimization processing.

[0158] Refer to Figure 9 , according to some embodiments of the present application, the multi-dimensional light field information corresponds to the light field camera coordinate system, the multi-dimensional event information corresponds to the event camera coordinate system, and step S801 fuses the multi-dimensional light field information and the multi-dimensional event information according to the camera calibration parameters to obtain preliminary acquisition image fusion data, which may include:

[0159] Step S901, perform time alignment processing on the multi-dimensional light field information and the multi-dimensional event information;

[0160] After the time alignment processing in step S902, according to the camera calibration parameters, the multi-dimensional light field information is mapped into the event camera coordinate system, or the multi-dimensional event information is mapped into the light field camera coordinate system to fuse the multi-dimensional light field information and the multi-dimensional event information to obtain preliminary acquisition image fusion data.

[0161] In step S901 of some embodiments, perform time alignment processing on the multi-dimensional light field information and the multi-dimensional event information;

[0162] It should be noted that time alignment processing is performed on the multi-dimensional light field information and multi-dimensional event information to ensure the consistency of the two data sources in the time dimension. The time alignment processing is achieved by calculating the time difference and performing normalization processing. Specifically, in the embodiments of the present application, the deviation between the timestamp of the event data and the frame time of the light field data is calculated, and the event data is adjusted in time to align it with the frame time of the light field data. This step is crucial for subsequent fusion processing because only data that is aligned in time can ensure the accuracy and consistency of the fused image data in the time dimension.

[0163] In some embodiments, in step S902, after the time alignment processing, the multi-dimensional light field information is mapped into the event camera coordinate system according to the camera calibration parameters, or the multi-dimensional event information is mapped into the light field camera coordinate system to perform fusion processing on the multi-dimensional light field information and multi-dimensional event information, and obtain preliminary acquisition image fusion data.

[0164] It should be noted that after the time alignment processing is completed, in the embodiments of the present application, the multi-dimensional light field information is mapped into the event camera coordinate system according to the camera calibration parameters, or the multi-dimensional event information is mapped into the light field camera coordinate system. The camera calibration parameters include the internal and external parameters of the light field camera and the event camera, such as focal length, principal point position, rotation matrix, and translation vector, etc. These parameters ensure the consistency of the two data sources in the spatial dimension. Specifically, in the embodiments of the present application, the rotation matrix and translation vector in the camera calibration parameters are used to convert the light field information from the light field camera coordinate system to the event camera coordinate system, or convert the event information from the event camera coordinate system to the light field camera coordinate system. Through this coordinate system conversion, the embodiments of the present application can align the two data sources in space, thereby realizing the fusion processing of multi-dimensional light field information and multi-dimensional event information. Finally, the embodiments of the present application generate preliminary acquisition image fusion data, which contains the fusion result of the light field information and the event information, providing a basis for subsequent depth of field optimization and depth of field recovery processing.

[0165] In step S802 of some embodiments, according to the scene depth estimation information, the depth of field of the preliminary image fusion data is optimized to obtain intermediate acquisition image fusion data;

[0166] It should be noted that, according to the scene depth estimation information, the initial image fusion data is processed for depth of field optimization to obtain the intermediate acquired image fusion data. The scene depth estimation information provides the depth information of the objects in the scene, enabling the embodiments of the present application to perform depth of field optimization on the initial fusion data. Specifically, the embodiments of the present application can adjust the depth of field of the image according to the depth information, making the details of foreground and background objects clearer, while reducing the blur and distortion caused by depth differences. This process not only improves the visual quality of the image, but also enhances the accuracy of the depth information, providing a better data basis for subsequent depth of field restoration processing.

[0167] In step S803 of some embodiments, according to the multi-dimensional event information, the intermediate acquired image fusion data is processed for depth of field restoration to obtain the image acquisition fusion data.

[0168] It should be noted that, according to the multi-dimensional event information, the intermediate acquired image fusion data is processed for depth of field restoration to obtain the final image acquisition fusion data. The multi-dimensional event information records the events of brightness changes in the scene. These multi-dimensional event information has extremely high time resolution and can capture the rapid changes in the scene. By using the multi-dimensional event information, the embodiments of the present application can perform depth of field restoration on the intermediate image fusion data. Especially in dynamic scenes, the multi-dimensional event information can provide accurate motion trajectories and brightness change information, thereby improving the accuracy and speed of depth restoration. The finally generated image acquisition fusion data not only contains high-resolution static image information, but also contains high-time-resolution dynamic change information, providing strong support for subsequent advanced applications such as 3D reconstruction and object recognition.

[0169] Through the above three steps of step S801 to step S803, the embodiments of the present application can effectively perform enhanced fusion processing on the multi-dimensional image acquisition data to generate high-quality image acquisition fusion data. This process not only improves the quality of the image data and the accuracy of the depth information, but also provides more accurate and detailed data support for subsequent advanced applications.

[0170] The embodiments of the present application shown through step S501 to step S503 can effectively solve the problems of low resolution, limited field of view and depth information capture ability caused by the combination of the light field camera and the event camera due to the shared optical path and image sensor. The present application generates high-quality image acquisition fusion data through formatting processing, scene depth estimation and enhanced fusion processing, improving the overall performance and application value of the image acquisition device. This process not only improves the imaging quality and resolution, but also improves the depth restoration accuracy of dynamic scenes, providing more accurate data support for subsequent 3D reconstruction and scene understanding.

[0171] In some more specific embodiments, step S405 generates image acquisition fusion data according to the camera calibration parameters, light field data, and event data, and can be specifically implemented in the following manner:

[0172] In the embodiments of the present application, by integrating the event data captured by the event camera and the light field data of the light field camera (the light field data may specifically include light field image information, depth information, point cloud data, etc.), comprehensive data processing and analysis are carried out. Through the all-optical data fusion algorithm, the event data with light time information and the light field data with direction and position relationship are effectively integrated to improve the final imaging quality and resolution, and fast reconstruction and subsequent analysis are realized based on the fast event information.

[0173] To fuse the light field data and event data, time alignment is first required:

[0174] The event camera has an extremely high time resolution, usually at the microsecond level, while the frame rate of the light field camera is relatively low. Therefore, the event data can help capture the dynamic changes of the scene between the frames of the light field camera.

[0175] Assume that the frame rate of the light field camera is f LF , and the event timestamp of the event camera is t e , then the acquisition time of each light field camera frame can be defined as:

[0176]

[0177] where n is the frame number.

[0178] For the event camera, the event timestamp te of the event data needs to be normalized within the frame time range of the light field camera to ensure time synchronization:

[0179]

[0180] Here, Δt e is the deviation between the event occurrence time and the corresponding light field frame. Through this time alignment, the event data can be mapped into a specific light field frame.

[0181] In terms of space:

[0182] Since the imaging principles of the light field camera and the event camera are different, it is necessary to obtain the relative position and attitude between the two through calibration, which can be specifically described by the external parameters (rotation matrix R and translation vector t) of the camera.

[0183] If the coordinates of the imaging point of the light field camera in the three-dimensional space are (X, Y, Z), then the projection coordinates (x e ′, y e ′) of this point in the event camera can be converted through the external parameter matrix:

[0184]

[0185] Among them, Ke is the internal parameter matrix of the event camera, which describes the internal parameter information such as the focal length and the position of the principal point of the event camera.

[0186] Through this geometric transformation, the three-dimensional points in the light field camera can be mapped to the pixel coordinate system of the event camera, and at the same time, the event points of the event camera can be converted into the coordinate system of the light field camera, so as to complete the data fusion strategy and achieve more accurate and rapid three-dimensional reconstruction or scene understanding.

[0187] Although the light field camera itself can estimate depth through parallax information, in a fast dynamic scene, the movement of objects may cause blurring. The event camera can capture the precise movement trajectory of the object. Combining the movement information of the event with the light field depth estimation can improve the reconstruction accuracy of the dynamic scene. Using the movement information of the event camera, constraint conditions can be established:

[0188]

[0189] Among them, δL(x,y,u,v) is the change rate of the light field brightness. The brightness change captured by the event camera can be directly used to guide the depth optimization of the light field data. A schematic diagram of data fusion is shown in Figure 10 as follows.

[0190] Referring to Figure 11 , Figure 11 shows the hardware structure of an electronic device according to another embodiment. The electronic device includes:

[0191] A processor 1101, which can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0192] A memory 1102, which can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1102 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1102 and are called by the processor 1101 to execute the image acquisition method of the embodiments of the present application;

[0193] An input / output interface 1103 for implementing information input and output;

[0194] A communication interface 1104 for implementing communication interaction between this device and other devices, which can achieve communication through a wired manner (such as USB, network cable, etc.) or through a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0195] A bus 1105 for transmitting information between various components of the device (such as a processor 1101, a memory 1102, an input / output interface 1103, and a communication interface 1104);

[0196] Among them, the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104 achieve communication connections with each other inside the device through the bus 1105.

[0197] The embodiment of the present application also provides a computer program product, which includes a computer program. The processor of the computer device reads and executes this computer program, so that the computer device executes to implement the above image acquisition method.

[0198] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "comprising" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0199] It should be understood that in this disclosure, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (one)" or a similar expression thereof means any combination of these items, which can include any combination of a single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0200] It should be understood that in the description of the embodiments of this application, the meaning of a plurality (or multiple items) is more than two. Understandings such as greater than, less than, exceeding, etc. do not include the present number, and understandings such as above, below, within, etc. include the present number.

[0201] In several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in an electrical, mechanical, or other form.

[0202] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0203] In addition, the functional units in each embodiment of this disclosure can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0204] When an integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and may include several instructions to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present disclosure. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store program codes.

[0205] It should also be understood that the various embodiments provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0206] The above is a specific description of the embodiments of the present disclosure, but the present disclosure is not limited to the above embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present disclosure, and these equivalent deformations or substitutions are all included within the scope defined by the claims of the present disclosure.

Claims

1. An image acquisition device, characterized in that: It includes image acquisition and calibration module, multi-camera control module, light field camera imaging module, event camera imaging module and data fusion module; The image acquisition and calibration module is used to obtain camera calibration parameters; The multi-camera control module is used to generate a trigger signal and send the trigger signal to the light field camera imaging module and the event camera imaging module respectively, so that the light field camera imaging module and the event camera imaging module perform data acquisition in the same detection time slot; The light field camera imaging module is used to collect light field data of the target scene in the detection time slot according to the trigger signal; The event camera imaging module is used to collect event data of the target scene in the detection time slot according to the trigger signal; The data fusion module is used to generate image acquisition fusion data according to the camera calibration parameters, the light field data and the event data.

2. The image acquisition device according to claim 1, characterized in that: include: Base, limit plate, support rail, multi-camera control module; The multi-camera control module is fixed to the base; The supporting rails are fixed on both sides of the base, and the light field camera imaging module and the event camera imaging module are arranged on the supporting rails; The limiting plate is fixed to one end of the supporting slide rail to prevent the light field camera imaging module and the event camera imaging module from overtravel and falling off when the supporting slide rail slides.

3. An image acquisition method, characterized in that: Applicable to an image acquisition device, comprising: Obtain camera calibration parameters through the image acquisition and calibration module; In the multi-camera control module, a trigger signal is generated, and the trigger signal is sent to the light field camera imaging module and the event camera imaging module respectively, so that the light field camera imaging module and the event camera imaging module perform data acquisition in the same detection time slot; In the light field camera imaging module, light field data of the target scene is collected in the detection time slot according to the trigger signal; In the event camera imaging module, event data of the target scene is collected in the detection time slot according to the trigger signal; In the data fusion module, image acquisition fusion data is generated according to the camera calibration parameters, the light field data and the event data.

4. The method according to claim 3, characterized in that The step of generating image acquisition fusion data according to the camera calibration parameters, the light field data and the event data comprises: The light field data and the event data of the target scene are subjected to the same formatting process to obtain multi-dimensional image acquisition data; wherein the multi-dimensional image acquisition data includes multi-dimensional light field information and multi-dimensional event information; Perform scene depth estimation based on the camera calibration parameters and the multi-dimensional light field information to obtain scene depth estimation information; According to the camera calibration parameters, the scene depth estimation information and the multi-dimensional event information, enhanced fusion processing is performed on the multi-dimensional image acquisition data to generate the image acquisition fusion data.

5. The method according to claim 4, characterized in that The multidimensional light field information includes light intensity acquisition sub-data, direction acquisition sub-data and position acquisition sub-data, and the scene depth estimation is performed based on the camera calibration parameters and the multidimensional light field information to obtain the scene depth estimation information, including: Generate a multi-view parallax image of the target scene according to the camera calibration parameters, the light intensity acquisition sub-data, the direction acquisition sub-data and the position acquisition sub-data; Disparity depth estimation is performed based on the multi-view disparity images to generate the scene depth estimation information.

6. The method according to claim 5, characterized in that The multi-dimensional image acquisition data also includes event acquisition sub-data, and the performing of disparity depth estimation based on the multi-view disparity image to generate the scene depth estimation information includes: Performing scene depth analysis according to the multi-view parallax images to obtain scene depth analysis information; The scene depth analysis information is corrected according to the event collection sub-data to obtain scene depth estimation information.

7. The method according to claim 4, characterized in that The step of performing enhanced fusion processing on the multi-dimensional image acquisition data according to the camera calibration parameters, the scene depth estimation information and the multi-dimensional event information to generate the image acquisition fusion data includes: According to the camera calibration parameters, the multi-dimensional light field information and the multi-dimensional event information are fused to obtain preliminary collected image fusion data; According to the scene depth estimation information, performing depth of field optimization processing on the preliminary image fusion data to obtain intermediate collected image fusion data; According to the multi-dimensional event information, depth of field recovery processing is performed on the intermediate acquired image fusion data to obtain the image acquisition fusion data.

8. The method according to claim 7, characterized in that The multi-dimensional light field information corresponds to a light field camera coordinate system, the multi-dimensional event information corresponds to an event camera coordinate system, and the multi-dimensional light field information and the multi-dimensional event information are fused according to the camera calibration parameters to obtain preliminary collected image fusion data, including: Performing time alignment processing on the multi-dimensional light field information and the multi-dimensional event information; After the time alignment process, the multi-dimensional light field information is mapped into the event camera coordinate system according to the camera calibration parameters, or the multi-dimensional event information is mapped into the light field camera coordinate system, so as to fuse the multi-dimensional light field information and the multi-dimensional event information to obtain the preliminary collected image fusion data.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor implements the image acquisition method according to any one of claims 2 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the image acquisition method according to any one of claims 2 to 8.

Images