An image reconstruction method and device, electronic equipment and storage medium

Abstract

The application provides an image reconstruction method and device, electronic equipment and storage medium, and relates to the technical field of automatic driving sensor synchronization. The method comprises the following steps: acquiring a point cloud and a plurality of continuous exposure images for a target scene; the point cloud is collected by a laser radar at a first frame rate within a unit time; the plurality of continuous exposure images are collected by a camera at a second frame rate within a unit time; a target pixel base block corresponding to each point in the point cloud is acquired from the plurality of continuous exposure images based on a first exposure time of each point in the point cloud; a difference between a second exposure time of the target pixel base block and the first exposure time is smaller than a difference between an exposure time of other pixel base blocks in the plurality of continuous exposure images and the first exposure time; and a target image is reconstructed based on the target pixel base block corresponding to each point in the point cloud. Thus, an image closer to the actual physical situation can be obtained, and the time error between the point cloud collected by the laser radar and the image collected by the camera is minimized as much as possible.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving sensor synchronization technology, and more specifically, to an image reconstruction method, apparatus, electronic device, and storage medium. Background Technology

[0002] Sensors used in autonomous driving, such as LiDAR and cameras, suffer from inconsistencies in sampling frequencies and timing of messages received by the computing unit from different sensors, leading to problems such as inaccurate obstacle recognition. Existing technologies for sensor time synchronization can be broadly categorized into: unified clock source, hardware synchronization, and software synchronization. By unifying the time sources of both the LiDAR and camera to GPS time, the camera is triggered when the laser beam rotates to the center of its field of view, ensuring synchronized data acquisition by both the LiDAR and camera.

[0003] However, due to the different data acquisition methods of LiDAR and cameras, only approximately guaranteed synchronization between them can be ensured. The acquisition time for point clouds at the center of the camera's field of view is consistent with the image acquisition time. However, for point clouds at the edges of the camera's field of view, the acquisition time deviates from the image acquisition time, varying depending on the size of the camera's field of view and the point cloud acquisition frame rate. Assuming the point cloud acquisition time is used as a reference value (the true value), aligning the corresponding image acquisition time with the point cloud acquisition time will result in a discrepancy between the image exposure time and the actual physical reality because the exposure time of each point in the point cloud is inconsistent with the exposure time of the camera's pixel array. Consequently, the time error between the point cloud acquired by the LiDAR and the image acquired by the camera is significant. Summary of the Invention

[0004] To address the problems in the prior art, the present invention provides an image reconstruction method, apparatus, electronic device, and storage medium, the solutions of which are as follows:

[0005] On the one hand, an image reconstruction method is provided, including:

[0006] The system acquires point clouds and multi-frame images of continuous exposure for a target scene; the point clouds are acquired by a lidar at a first frame rate within a unit time; the multi-frame images are acquired by a camera at a second frame rate within a unit time.

[0007] Based on the first exposure time of each point in the point cloud, the target pixel block corresponding to the point is obtained from the multi-frame image; the difference between the second exposure time of the target pixel block and the first exposure time is less than the difference between the exposure time of other pixel blocks in the multi-frame image and the first exposure time;

[0008] The target image is reconstructed based on the target pixel block corresponding to each point in the point cloud.

[0009] Optionally, obtaining the target pixel base block corresponding to the point from the multi-frame image based on the first exposure time of each point in the point cloud includes:

[0010] Based on the number of points in the point cloud, each frame of the image is divided to obtain pixel blocks corresponding to each point;

[0011] Determine the second exposure time for each pixel block in each frame of the image;

[0012] For each point in the point cloud, determine the difference between the first exposure time of the point and the second exposure time of the corresponding pixel block in each frame image;

[0013] The pixel base block corresponding to the minimum difference is determined to obtain the target pixel base block corresponding to the point.

[0014] Optionally, determining the second exposure time for each pixel block in each frame image includes:

[0015] Determine the number of pixel rows contained in each pixel block in each frame image;

[0016] Mark the exposure time of each row of pixels in each pixel block;

[0017] The second exposure time of each pixel block in each frame image is determined based on the number of pixel rows contained in each pixel block and the exposure time of each row of pixels.

[0018] Optionally, determining the second exposure time of each pixel block in each frame image based on the number of pixel rows contained in each pixel block and the exposure time of each row of pixels includes:

[0019] For each pixel block, the exposure times of the pixels in each row are summed and then averaged based on the number of pixel rows to obtain the average exposure time of the pixels in each pixel block;

[0020] The average exposure time is used as the second exposure time for each pixel block.

[0021] Optionally, the step of dividing each frame of the image based on the number of points in the point cloud to obtain pixel base blocks corresponding to each point includes:

[0022] The first number of each point in the point cloud is determined according to a preset order;

[0023] The pixel blocks in each frame image are equally divided based on the number of points in the point cloud, and the number of pixel blocks is the same as the number of points in the point cloud.

[0024] For each frame of image, the pixel base block corresponding to each point is determined based on the first number of each point in the point cloud, and a second number of each pixel base block in each frame of image is generated.

[0025] Optionally, determining the difference between the first exposure time of each point in the point cloud and the second exposure time of the corresponding pixel block in each frame image includes:

[0026] The sequence identification information of the second exposure time corresponding to the pixel block in each frame image is determined based on the second number of each pixel block in each frame image and the image frame number of the multi-frame image; the image frame number is determined based on the time order of the continuous exposure of the multi-frame image.

[0027] The difference between the first exposure time of the point and the second exposure time of the corresponding pixel block in each frame image is determined based on the sequence identification information;

[0028] The step of determining the pixel base block corresponding to the minimum difference to obtain the target pixel base block corresponding to the point includes:

[0029] When the minimum difference is taken, the target sequence identification information corresponding to the second exposure time is obtained, and the pixel base block indicated by the second target number and the target image frame number in the target sequence identification information is taken as the target pixel base block corresponding to the point.

[0030] Optionally, the number of frames in the multi-frame image is determined based on the ratio of the second frame rate to the first frame rate.

[0031] On the other hand, an image reconstruction apparatus is provided, comprising:

[0032] The acquisition module acquires point clouds and multi-frame images of continuous exposure for the target scene; the point clouds are acquired by the lidar at a first frame rate within a unit time; the multi-frame images are acquired by the camera at a second frame rate within a unit time.

[0033] The calculation module is used to obtain the target pixel base block corresponding to each point from the multi-frame image based on the first exposure time of each point in the point cloud; the difference between the second exposure time of the target pixel base block and the first exposure time is less than the difference between the exposure time of other pixel base blocks in the multi-frame image and the first exposure time.

[0034] The image reconstruction module is used to reconstruct the target image based on the target pixel block corresponding to each point in the point cloud.

[0035] Optionally, the computing module includes:

[0036] The image processing module is used to divide each frame of the image based on the number of points in the point cloud to obtain pixel base blocks corresponding to each point;

[0037] The second exposure time determination module is used to determine the second exposure time of each pixel block in each frame image;

[0038] The difference calculation module is used to determine the difference between the first exposure time of each point in the point cloud and the second exposure time of the corresponding pixel block in each frame image for each point.

[0039] The target pixel base block determination module is used to determine the pixel base block corresponding to the minimum difference, and obtain the target pixel base block corresponding to the point.

[0040] Optionally, the second exposure time determination module includes:

[0041] A pixel partitioning unit is used to determine the number of pixel rows contained in each pixel base block in each frame image;

[0042] A pixel exposure time determination unit is used to mark the exposure time of each row of pixels in each pixel block;

[0043] The second exposure time calculation unit is used to determine the second exposure time of each pixel block in each frame image based on the number of pixel rows contained in each pixel block and the exposure time of each row of pixels.

[0044] Optionally, the second exposure time calculation unit includes:

[0045] The average exposure time calculation unit is used to sum the exposure times of each row of pixels for each pixel block and then average them based on the number of pixel rows to obtain the average exposure time of the pixels in each pixel block.

[0046] The second exposure time determination unit is used to use the average exposure time as the second exposure time for each pixel block.

[0047] Optionally, the image processing module includes:

[0048] A point cloud numbering unit is used to determine the first number of each point in the point cloud according to a preset order.

[0049] A pixel block partitioning unit is used to divide the pixel blocks in each frame image equally based on the number of points in the point cloud, wherein the number of pixel blocks is the same as the number of points in the point cloud.

[0050] The pixel base block numbering unit is used to determine the pixel base block corresponding to each point based on the number of each point in the point cloud for each frame of the image, and to generate a second number for each pixel base block in each frame of the image.

[0051] Optionally, the difference calculation module includes:

[0052] An identification unit is used to determine the sequence identification information of the second exposure time of the pixel blocks in each frame image based on the second number of each pixel block in each frame image and the image frame number of the multi-frame image; the image frame number is determined based on the time order of the continuous exposure of the multi-frame image;

[0053] The difference calculation unit is used to determine the difference between the first exposure time of the point and the second exposure time of the corresponding pixel block in each frame image based on the sequence identification information.

[0054] Optionally, the image reconstruction apparatus further includes an image frame number determination module, used to determine the number of frames of the multi-frame image based on the ratio of the second frame rate to the first frame rate.

[0055] On the other hand, an electronic device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the steps of the above method.

[0056] On the other hand, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by a processor to implement the steps of the above method.

[0057] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium, wherein a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the above-described method.

[0058] By adopting the above technical solution, the present invention has the following beneficial effects:

[0059] For point clouds and multi-frame images of continuous exposures of the target scene, the target pixel blocks with the smallest exposure time difference with the point cloud are obtained from the multi-frame images. Each target pixel block is reconstructed into a new image to obtain an image that is closer to the actual physical situation and minimizes the time error between the point cloud acquired by the lidar and the image acquired by the camera.

[0060] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and the same reference numerals usually represent the same parts. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart illustrating an image reconstruction method provided in an embodiment of the present invention.

[0063] Figure 2 This is a schematic diagram of an optional method for implementing the image reconstruction method provided in an embodiment of the present invention;

[0064] Figure 3 This is a schematic diagram of the structure of the image reconstruction device provided in an embodiment of the present invention;

[0065] Figure 4 A schematic diagram of the hardware structure of a terminal running an image reconstruction method provided in an embodiment of the present invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0067] The term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the invention. In the description of the invention, it should be understood that the terms "upper," "lower," "top," "bottom," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature. Moreover, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein.

[0068] The application scenarios of this invention can include sensors used in autonomous driving, such as LiDAR and cameras. Furthermore, it can also be applied to other fields.

[0069] During autonomous driving, the system collects real-time scene data through sensors such as LiDAR and cameras. If the time of the messages received by the computing unit from each sensor is inconsistent, problems such as inaccurate obstacle recognition can occur. The time difference in sensor data is mainly due to the inconsistent sampling frequencies of each sensor; for example, LiDAR is typically 10Hz, while cameras are typically 25Hz or 30Hz. There is also a certain delay in data transmission between different sensors. While the nearest neighbor frame is found by searching for adjacent timestamps, if the two timestamps differ significantly, and the sensor or obstacle is moving, a large synchronization error will result. Existing technologies use hardware synchronization triggering to mitigate the error caused by timestamp searching: after unifying the time sources of LiDAR and cameras to GPS time, the PPS signal is used as the trigger signal, ensuring that both LiDAR and camera collect data at the rising edge of the PPS signal and add their respective clock timestamps.

[0070] To ensure that the lidar acquires point cloud data at the rising edge of the PPS signal, the phase-locking angle of the lidar is set to the center of the camera's field of view. When the coordinate systems of the camera's field of view and the lidar are aligned, the phase-locking angle should be 0 degrees. Whenever the lidar's laser beam rotates to the 0-degree position, which is exactly the center of the camera's field of view, the rising edge of the PPS signal arrives, triggering the camera. This achieves synchronous data acquisition by the lidar and the camera.

[0071] However, due to the different data acquisition methods of LiDAR and cameras, the LiDAR laser beam rotates continuously 360 degrees. Assuming a frame rate of 10Hz, the earliest and latest points in a single frame of point cloud data will differ by 100ms. In contrast, cameras expose instantaneously, and the acquisition times of all pixels in the image can be considered identical to a certain degree. This means that only approximate synchronization between LiDAR and camera acquisition can be guaranteed. The point cloud data in the center of the camera's field of view is acquired at the same time as the image acquisition, but the point cloud data at the edge of the camera's field of view has a time deviation from the image acquisition. Depending on the size of the camera's field of view and the point cloud acquisition frame rate, this time deviation may range from 5ms to 20ms. If the point cloud time is used as a reference value (the true value) and the image time is aligned with the point cloud time, the inconsistent exposure times result in an inaccurate image exposure time value, leading to a significant time error between the point cloud acquired by the LiDAR and the image acquired by the camera.

[0072] See Figure 1 The diagram illustrates a flowchart of an image reconstruction method according to an embodiment of the present invention. This specification provides the operational steps of the method as described in the embodiments or flowcharts, but based on conventional or uninventive methods, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system devices or products, the method can be executed sequentially according to the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). The image reconstruction method provided by the embodiments of the present invention includes:

[0073] S101, acquire point cloud and multi-frame images of continuous exposure for the target scene; the point cloud is acquired by the lidar at a first frame rate per unit time; the multi-frame images are acquired by the camera at a second frame rate per unit time.

[0074] In one possible implementation, the number of frames in the multi-frame image is determined based on the ratio of the second frame rate to the first frame rate.

[0075] Specifically, because the frame rates of LiDAR and cameras differ (typically, the LiDAR frame rate is lower than the camera frame rate), the time it takes for the LiDAR to acquire one frame of point cloud data is longer than the time it takes for the camera to acquire one frame of image data per unit time. To ensure that a single frame of point cloud data corresponds to the image data, it is necessary to analyze multiple consecutively exposed frames. For example, in one embodiment, the LiDAR frame rate is 10Hz, and acquiring one frame of point cloud data takes 100ms; the camera frame rate is 30Hz, and acquiring one frame of image data takes 33.3ms. Therefore, three consecutively exposed frames are needed to correspond with the acquired point cloud data for analysis. In other embodiments, the number of image frames can be determined based on the frame rates of the LiDAR and the camera.

[0076] S102, based on the first exposure time of each point in the point cloud, obtain the target pixel base block corresponding to the point from the multi-frame image; the difference between the second exposure time of the target pixel base block and the first exposure time is less than the difference between the exposure time of other pixel base blocks in the multi-frame image and the first exposure time;

[0077] In one possible implementation, step S102 includes:

[0078] S201, based on the number of points in the point cloud, each frame image is divided to obtain pixel base blocks corresponding to each point; in one possible implementation, step S201 includes:

[0079] (1) Determine the first number of each point in the point cloud according to a preset order;

[0080] (2) Divide the pixel blocks in each frame image equally based on the number of points in the point cloud, wherein the number of pixel blocks is the same as the number of points in the point cloud;

[0081] (3) For each frame of image, the pixel base block corresponding to each point is determined based on the number of each point in the point cloud, and a second number of each pixel base block in each frame of image is generated.

[0082] Specifically, the point cloud contains m horizontal points arranged in n rows. Each point in the point cloud is numbered 1, 2, 3, ..., m*n according to a preset order, such as from left to right or from top to bottom. For each frame of the image, the image is divided into m*n pixel blocks, which is the same as the number of points in the point cloud. The pixel blocks in each frame of the image are numbered according to the numbering method of the point cloud, and the second number of each pixel block in each frame of the image is determined as 1, 2, 3, ..., m*n.

[0083] S202, determining the second exposure time for each pixel block in each frame image; in one possible implementation, step S202 includes:

[0084] (1) Determine the number of pixel rows contained in each pixel base block in each frame image;

[0085] (2) Mark the exposure time of each row of pixels in each pixel block;

[0086] (3) Determine the second exposure time of each pixel block in each frame image based on the number of pixel rows contained in each pixel block and the exposure time of each row of pixels. Specifically, for each pixel block, the exposure times of each row of pixels are summed and then averaged based on the number of pixel rows to obtain the average exposure time of the pixels in each pixel block; the average exposure time is used as the second exposure time of each pixel block.

[0087] Specifically, each pixel block contains k rows of pixels, and the exposure time of each row of pixels is t'(1), t'(2), ..., t'(k). The average exposure time of these k rows of pixels is obtained as the second exposure time of the pixel block.

[0088] S203, for each point in the point cloud, determine the difference between the first exposure time of the point and the second exposure time of the corresponding pixel block in each frame image; in one possible implementation, step S203 includes:

[0089] (1) Based on the second number of each pixel block in each frame image and the image frame number of the multi-frame image, determine the sequence identification information of the second exposure time of the pixel block in each frame image; the image frame number is determined based on the time order of the continuous exposure of the multi-frame image;

[0090] (2) Determine the difference between the first exposure time of the point and the second exposure time of the corresponding pixel block in each frame image based on the sequence identification information.

[0091] Specifically, based on the first number of the point cloud, the first exposure time of each point in the point cloud is sequentially marked as tl(1), tl(2), tl(3), ..., tl(m*n), that is, the first exposure time of the point cloud point is tl(i), i∈{1,2,3,……,m*n}; the number of frames in the multi-frame image is x, and based on the second number of each pixel block in each frame image, the second exposure time of the pixel block in each frame image is sequentially marked as t(1)(1), t(1)(2), ..., t(1)(m*n), ..., t(x)(1), t(x)(2), ..., t(x)(m*n). That is, the second exposure time of the pixel block is t(x)(i), where x represents the image frame number, i represents the second number of the pixel block, and i∈{1,2,3,……,m*n}.

[0092] For each point in the point cloud, the difference between the first exposure time of the point and the second exposure time of the corresponding pixel block in each frame image is calculated, i.e., δt(x)(i)=tl(i)-t(x)(i).

[0093] S204, determine the pixel base block corresponding to the minimum difference, and obtain the target pixel base block corresponding to the point.

[0094] In one possible implementation, when the minimum difference is taken, the target sequence identification information corresponding to the second exposure time is obtained. The pixel block indicated by the second target number and the target image frame number in the target sequence identification information is taken as the target pixel block corresponding to the point. Specifically, for example, for point 1 in the point cloud, the difference between its first exposure time and the second exposure time of pixel block 1 in each frame image is taken. The difference with the second exposure time of pixel block 1 in the second frame image is the smallest. Then, pixel block 1 in the second frame image is determined as the target pixel block corresponding to point 1. For point 2 in the point cloud, the difference between its first exposure time and the second exposure time of pixel block 2 in each frame image is taken. The difference with the second exposure time of pixel block 2 in the third frame image is the smallest. Then, pixel block 2 in the second frame image is determined as the target pixel block corresponding to point 2. Other points are determined in the same way. The target pixel block corresponding to each point is determined and marked for image reconstruction described later.

[0095] S103, reconstruct the target image based on the target pixel block corresponding to each point in the point cloud.

[0096] To facilitate understanding of the technical solution of the present invention, an embodiment is described in detail below. The quantities listed are small for ease of explanation, and the actual situation shall prevail in the specific implementation:

[0097] The lidar acquires one frame of point cloud at a frame rate of 10Hz, and the camera acquires three frames of images corresponding to the point cloud at a frame rate of 30Hz. The point cloud has 5 points horizontally, with a total of 8 rows. The first exposure time of each point in the point cloud is marked as tl(1), tl(2), tl(3), ..., tl(40) in sequence. Each frame of the image is divided into 40 pixel blocks, and each pixel block contains 10 rows of pixels. The second exposure time of the pixel block is determined based on the average exposure time of the 10 rows of pixels. The second exposure time of the pixel block in each frame of the image is marked as t(1)(1), t(1)(2), ..., t(1)(40), t(2)(1) in sequence. …, t(2)(40), t(3)(1), …, t(3)(40); For point 1 in the point cloud, according to δt(x)(i)=tl(i)-t(x)(i), take the difference between tl(1) and t(1)(1), t(2)(1), t(3)(1), and obtain δt(2)(1) as the minimum value. That is, pixel block 1 in the second frame image is the target pixel block, and it is marked for image reconstruction. Other points are similarly deduced. Finally, the target image is reconstructed based on each pixel block. It should be noted that the value of δt(x)(i) should be the absolute value, and the value with the smallest absolute value is finally determined.

[0098] Corresponding to the above image reconstruction methods, this embodiment of the invention also provides an image reconstruction apparatus. Since the image reconstruction apparatus provided by this embodiment of the invention corresponds to the image reconstruction methods provided by the above embodiments, the implementation methods of the aforementioned image reconstruction methods are also applicable to the image reconstruction apparatus provided in this embodiment, and will not be described again in this embodiment of the invention.

[0099] refer to Figure 3 The diagram shows a schematic of an image reconstruction device according to an embodiment of the present invention. This device has the function of implementing the image reconstruction method described in the above-described method embodiments. This function can be implemented in hardware or by hardware executing corresponding software. The device may include:

[0100] The acquisition module 310 acquires point clouds and multi-frame images of continuous exposure for the target scene; the point clouds are acquired by the lidar at a first frame rate within a unit time; the multi-frame images are acquired by the camera at a second frame rate within a unit time.

[0101] The calculation module 320 is used to obtain the target pixel base block corresponding to the point from the multi-frame image based on the first exposure time of each point in the point cloud; the difference between the second exposure time of the target pixel base block and the first exposure time is less than the difference between the exposure time of other pixel base blocks in the multi-frame image and the first exposure time.

[0102] The image reconstruction module 330 is used to reconstruct a target image based on the target pixel block corresponding to each point in the point cloud.

[0103] Optionally, the computing module 320 includes:

[0104] The image processing module is used to divide each frame of the image based on the number of points in the point cloud to obtain pixel base blocks corresponding to each point;

[0105] The second exposure time determination module is used to determine the second exposure time of each pixel block in each frame image;

[0106] The difference calculation module is used to determine the difference between the first exposure time of each point in the point cloud and the second exposure time of the corresponding pixel block in each frame image for each point.

[0107] The target pixel base block determination module is used to determine the pixel base block corresponding to the minimum difference, and obtain the target pixel base block corresponding to the point.

[0108] Optionally, the second exposure time determination module includes:

[0109] A pixel partitioning unit is used to determine the number of pixel rows contained in each pixel base block in each frame image;

[0110] A pixel exposure time determination unit is used to mark the exposure time of each row of pixels in each pixel block;

[0111] The second exposure time calculation unit is used to determine the second exposure time of each pixel block in each frame image based on the number of pixel rows contained in each pixel block and the exposure time of each row of pixels.

[0112] Optionally, the second exposure time calculation unit includes:

[0113] The average exposure time calculation unit is used to sum the exposure times of each row of pixels for each pixel block and then average them based on the number of pixel rows to obtain the average exposure time of the pixels in each pixel block.

[0114] The second exposure time determination unit is used to use the average exposure time as the second exposure time for each pixel block.

[0115] Optionally, the image processing module includes:

[0116] A point cloud numbering unit is used to determine the first number of each point in the point cloud according to a preset order.

[0117] A pixel block partitioning unit is used to divide the pixel blocks in each frame image equally based on the number of points in the point cloud, wherein the number of pixel blocks is the same as the number of points in the point cloud.

[0118] The pixel base block numbering unit is used to determine the pixel base block corresponding to each point based on the number of each point in the point cloud for each frame of the image, and to generate a second number for each pixel base block in each frame of the image.

[0119] Optionally, the difference calculation module includes:

[0120] An identification unit is used to determine the sequence identification information of the second exposure time of the pixel blocks in each frame image based on the second number of each pixel block in each frame image and the image frame number of the multi-frame image; the image frame number is determined based on the time order of the continuous exposure of the multi-frame image;

[0121] The difference calculation unit is used to determine the difference between the first exposure time of the point and the second exposure time of the corresponding pixel block in each frame image based on the sequence identification information.

[0122] Optionally, the image reconstruction apparatus further includes an image frame number determination module, used to determine the number of frames of the multi-frame image based on the ratio of the second frame rate to the first frame rate.

[0123] This invention also provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the steps of the image reconstruction method described above.

[0124] Memory can be used to store software programs and modules. The processor executes various functional applications by running the software programs and modules stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for functions, etc.; the data storage area can store data created based on the use of the device, etc. Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory. The processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0125] The methods and embodiments provided in this invention can be executed on a computer terminal, server, or similar computing device. Taking running on a terminal as an example, refer to... Figure 4 The diagram shown is a hardware structure diagram of a terminal that runs an image reconstruction method according to an embodiment of the present invention.

[0126] Specifically, the terminal may include an RF (Radio Frequency) circuit 410, a memory 420 including one or more computer-readable storage media, an input unit 430, a display unit 440, a sensor 450, an audio circuit 460, a WiFi (Wireless Fidelity) module 470, a processor 480 including one or more processing cores, and a power supply 490, among other components. Those skilled in the art will understand that... Figure 4 The terminal structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0127] The RF circuit 410 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and hands it over to one or more processors 480 for processing; in addition, it transmits uplink data to the base station.

[0128] The memory 420 can be used to store software programs and modules. The processor 480 executes various functional applications and data processing by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for the functions, etc.; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 420 may also include a memory controller to provide access to the memory 420 for the processor 480 and the input unit 430.

[0129] The input unit 430 can be used to receive input numeric or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, the input unit 430 may include a touch-sensitive surface 431 and other input devices 432.

[0130] The display unit 440 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the terminal. These graphical user interfaces can be composed of graphics, text, icons, videos, and any combination thereof. The display unit 440 may include a display panel 441, which may optionally be configured as an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or similar display. Further, a touch-sensitive surface 431 may cover the display panel 441. When the touch-sensitive surface 431 detects a touch operation on or near it, it transmits the information to the processor 480 to determine the type of touch event. Subsequently, the processor 480 provides corresponding visual output on the display panel 441 according to the type of touch event. The touch-sensitive surface 431 and the display panel 441 can be two independent components to implement input and output functions; however, in some embodiments, the touch-sensitive surface 431 and the display panel 441 can be integrated to achieve input and output functions.

[0131] The terminal may also include at least one sensor 450, such as a light sensor, motion sensor, and other sensors. An audio circuit 460, a speaker 461, and a microphone 462 provide an audio interface between the user and the terminal. The audio circuit 460 converts received audio data into electrical signals and transmits them to the speaker 461, where the speaker 461 converts them into sound signals for output. Conversely, the microphone 462 converts collected sound signals into electrical signals, which are then received by the audio circuit 460, converted into audio data, and then processed by the processor 480 before being transmitted via the RF circuit 410 to, for example, another terminal, or output to the memory 420 for further processing. The audio circuit 460 may also include an earphone jack to provide communication between a peripheral headset and the terminal.

[0132] WiFi is a short-range wireless transmission technology. The terminal, through the WiFi module 470, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 4 WiFi module 470 is shown, but it is understood that it is not an essential component of the terminal and can be omitted as needed without changing the nature of the invention.

[0133] The processor 480 is the control center of the terminal, connecting various parts of the terminal via various interfaces and lines. It executes software programs and / or modules stored in the memory 420, and calls data stored in the memory 420, to perform various functions and process data, thereby providing overall monitoring of the terminal. Optionally, the processor 480 may include one or more processing cores; preferably, the processor 480 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 480.

[0134] The terminal also includes a power supply 490 (such as a battery) to power various components. Preferably, the power supply can be logically connected to the processor 480 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 490 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0135] Although not shown, the terminal may also include a camera, Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. The aforementioned one or more programs include steps for performing the image reconstruction method provided in the above method embodiments.

[0136] This invention also provides a computer-readable storage medium storing at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the steps of the image reconstruction method described above. In this invention, the computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0137] This invention also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the method described above.

[0138] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image reconstruction method, characterized in that, include: Acquire point cloud and multi-frame images with continuous exposure for the target scene; The point cloud is collected by the lidar at the first frame rate per unit time; The multi-frame images are captured by the camera at the second frame rate within a unit of time; Based on the first exposure time of each point in the point cloud, the target pixel base block corresponding to the point is obtained from the multi-frame image; The difference between the second exposure time and the first exposure time of the target pixel block is less than the difference between the exposure time and the first exposure time of other pixel blocks in the multi-frame image; The target image is reconstructed based on the target pixel block corresponding to each point in the point cloud.

2. The image reconstruction method according to claim 1, characterized in that, The step of obtaining the target pixel base block corresponding to each point from the multi-frame images based on the first exposure time of each point in the point cloud includes: Based on the number of points in the point cloud, each frame of the image is divided to obtain pixel blocks corresponding to each point; Determine the second exposure time for each pixel block in each frame of the image; For each point in the point cloud, determine the difference between the first exposure time of the point and the second exposure time of the corresponding pixel block in each frame image; The pixel base block corresponding to the minimum difference is determined to obtain the target pixel base block corresponding to the point.

3. The image reconstruction method according to claim 2, characterized in that, Determining the second exposure time for each pixel block in each frame image includes: Determine the number of pixel rows contained in each pixel block in each frame image; Mark the exposure time of each row of pixels in each pixel block; The second exposure time of each pixel block in each frame image is determined based on the number of pixel rows contained in each pixel block and the exposure time of each row of pixels.

4. The image reconstruction method according to claim 3, characterized in that, The determination of the second exposure time for each pixel block in each frame image based on the number of pixel rows contained in each pixel block and the exposure time of each row of pixels includes: For each pixel block, the exposure times of the pixels in each row are summed and then averaged based on the number of pixel rows to obtain the average exposure time of the pixels in each pixel block; The average exposure time is used as the second exposure time for each pixel block.

5. The image reconstruction method according to claim 2, characterized in that, The step of dividing each frame of the image based on the number of points in the point cloud to obtain pixel blocks corresponding to each point includes: The first number of each point in the point cloud is determined according to a preset order; The pixel blocks in each frame image are equally divided based on the number of points in the point cloud, and the number of pixel blocks is the same as the number of points in the point cloud. For each frame of image, the pixel base block corresponding to each point is determined based on the first number of each point in the point cloud, and a second number of each pixel base block in each frame of image is generated.

6. The image reconstruction method according to claim 5, characterized in that, For each point in the point cloud, determining the difference between the first exposure time of that point and the second exposure time of the corresponding pixel block in each frame image includes: The sequence identification information of the second exposure time corresponding to the pixel block in each frame image is determined based on the second number of each pixel block in each frame image and the image frame number of the multi-frame image; the image frame number is determined based on the time order of the continuous exposure of the multi-frame image. The difference between the first exposure time of the point and the second exposure time of the corresponding pixel block in each frame image is determined based on the sequence identification information; The step of determining the pixel base block corresponding to the minimum difference to obtain the target pixel base block corresponding to the point includes: When the minimum difference is taken, the target sequence identification information corresponding to the second exposure time is obtained, and the pixel base block indicated by the second target number and the target image frame number in the target sequence identification information is taken as the target pixel base block corresponding to the point.

7. The image reconstruction method according to claim 1, characterized in that, The number of frames in the multi-frame image is determined based on the ratio of the second frame rate to the first frame rate.

8. An image reconstruction apparatus, characterized in that, include: The acquisition module acquires point clouds and multi-frame images of continuous exposure for the target scene; The point cloud is acquired by the lidar at a first frame rate within a unit of time; the multi-frame images are acquired by the camera at a second frame rate within a unit of time. The calculation module is used to obtain the target pixel base block corresponding to each point from the multi-frame image based on the first exposure time of each point in the point cloud; The difference between the second exposure time and the first exposure time of the target pixel block is less than the difference between the exposure time and the first exposure time of other pixel blocks in the multi-frame image; The image reconstruction module is used to reconstruct the target image based on the target pixel block corresponding to each point in the point cloud.

9. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the steps of the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the steps of the method as claimed in any one of claims 1 to 7.

Images