Exposure time prediction method and related equipment

Abstract

The invention discloses an exposure time prediction method and related equipment, and the method comprises the steps: obtaining a second moment position of a camera according to a first moment position and speed of the camera, and calculating an average light measurement value of a laser point cloud subjected to color assignment processing in a target range of the second moment position, and obtaining a first predicted exposure time corresponding to the second moment position based on the average photometric value and the photometric parameter. And then obtaining a first adjustment value according to the laser point cloud which is not subjected to color assignment processing in the Bayer image, and determining the exposure time of a second moment position according to the exposure time corresponding to the first moment position, the first adjustment value, the first predicted exposure time and the color assignment processing proportion of the laser point cloud. According to the method, on one hand, more accurate exposure time prediction is carried out on laser point clouds subjected to color assignment processing and laser point clouds not subjected to color assignment processing, and the system construction quality is improved; on the other hand, the position and speed information is combined, and the prediction precision is improved by more accurately utilizing the environment information of the predicted exposure position.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an exposure time prediction method and an exposure time prediction device. Background Art

[0002] With the advancement of computer vision and image processing technologies, real-time reconstruction and rendering of three-dimensional scenes have become important research areas in computer graphics, augmented reality (AR), and virtual reality (VR). For example, it is used in simultaneous localization and mapping (SLAM) systems to achieve environmental perception and mapping, playing a vital role in scenarios such as mobile robots, drones, and autonomous driving.

[0003] SLAM systems typically require the collaboration of multiple sensors. For example, they utilize high-resolution 3D information provided by light detection and ranging (LiDAR) sensors, combined with 2D image data from cameras and other image sensors to construct a 3D model of the surrounding environment. SLAM systems can also include inertial measurement units (IMUs) for even more precise positioning.

[0004] However, related technologies have difficulty balancing the resource allocation relationship between multiple sensors and computing devices, such as the balance between the camera's exposure algorithm and frame rate, resulting in reduced construction quality of the SLAM system in certain scenarios, especially high dynamic range (HDR) scenarios. Summary of the Invention

[0005] In view of this, the present application provides an exposure time prediction method and an exposure time prediction device, which are used to solve the problem of reduced construction quality of SLAM systems in high dynamic range scenes.

[0006] In a first aspect, the present application provides an exposure time prediction method, the method comprising:

[0007] Obtaining a second moment position of the camera according to the first moment position and the first moment velocity of the camera;

[0008] Calculate the average photometric value of the colored laser point cloud within the target range at the second moment;

[0009] Obtaining a first predicted exposure time corresponding to the second moment position based on the average light metering value and the light metering parameter;

[0010] Obtaining a first adjustment value based on a Bayer Raw Image captured by the camera, wherein the first adjustment value is used to predict an exposure time corresponding to the position at the second moment, wherein the laser point cloud in the Bayer Raw Image is not colorized;

[0011] The second predicted exposure time corresponding to the second moment position is determined according to the exposure time corresponding to the first moment position, the first adjustment value, the first predicted exposure time and the coloring parameter, wherein the coloring parameter is used to indicate the coloring processing ratio of the laser point cloud within the target range.

[0012] In some possible implementations, determining the second predicted exposure time corresponding to the second moment position according to the exposure time corresponding to the first moment position, the first adjustment value, the first predicted exposure time, and the coloring parameter includes:

[0013] determining a second adjustment value according to the first predicted exposure time and the exposure time corresponding to the first moment position;

[0014] A second predicted exposure time corresponding to the second moment position is determined according to the exposure time corresponding to the first moment position, the coloring parameter, the first adjustment value, and the second adjustment value.

[0015] In some possible implementations, obtaining the first predicted exposure time corresponding to the second moment position based on the average light measurement value and the light measurement parameter includes:

[0016] Obtaining a theoretical exposure time according to the average light measurement value and the light measurement parameters;

[0017] If the theoretical exposure time is less than the exposure time threshold, determining the theoretical exposure time as the first predicted exposure time corresponding to the second moment position;

[0018] If the theoretical exposure time is greater than or equal to the exposure time threshold, the exposure time threshold is determined to be the first predicted exposure time corresponding to the second moment position.

[0019] In some possible implementations, the photometric parameter includes a median brightness value , black level value and sensor response coefficient , the intermediate brightness value is related to the Bayer image, the black level value is the response value of the sensor when there is no light, and the theoretical exposure time Obtained according to the following formula:

[0020]

[0021] In some possible implementations, the method further includes:

[0022] Projecting an unobstructed laser point cloud in the global laser point cloud to the Bayer image based on the camera pose and camera parameters;

[0023] Performing color component interpolation on the unobstructed laser point cloud according to the Bayer image to obtain a colored laser point cloud, wherein the color components include a red component, a green component, and a blue component;

[0024] Calculating the normalized photometric value corresponding to each color component of the laser point in the color-imparting laser point cloud according to the photometric parameters;

[0025] Determining the type of the laser point according to the normalized photometric value corresponding to each color component of the laser point in the colored laser point cloud;

[0026] Determining a photometric weight of the target laser point according to an orientation parameter of the target laser point in the global laser point cloud and a type of a corresponding laser point in the coloring laser point cloud;

[0027] According to the type of the corresponding laser point, the photometric weight and the normalized photometric value corresponding to each color component of the corresponding laser point, the colored laser point cloud is fused with the global laser point cloud to obtain the colored laser point cloud.

[0028] In some possible implementations, determining the type of the laser point according to the normalized photometric value corresponding to each color component of the laser point in the color-assigned laser point cloud includes:

[0029] If the normalized photometric values of all color components of the laser point in the color-imparting laser point cloud are not less than a first photometric threshold and not greater than a second photometric threshold, determining that the laser point in the color-imparting laser point cloud is a valid point, and the first photometric threshold is less than the second photometric threshold;

[0030] If the normalized photometric value of any color component of the laser point in the color-imparting laser point cloud is less than the first photometric threshold, determine that the laser point in the color-imparting laser point cloud is an underexposed point;

[0031] If the normalized photometric value of any color component of the laser point in the color-imparting laser point cloud is greater than the second photometric threshold, it is determined that the laser point in the color-imparting laser point cloud is an overexposed point.

[0032] In some possible implementations, determining the photometric weight of the target laser point according to the orientation parameter of the target laser point in the global laser point cloud and the type of the corresponding laser point in the coloring laser point cloud includes:

[0033] If the corresponding laser point is a valid point, determining the N+1th fusion photometry weight of the target laser point according to the orientation parameter and the Nth fusion photometry weight of the target laser point, where N is a natural number. When N=0, the initial photometry weight of the target laser point is 0. The orientation parameters include a normal vector of the target laser point, a direction vector between the target laser point and the viewpoint, a distance between the target laser point and the viewpoint, an observation viewing angle parameter, and an observation distance parameter.

[0034] If the corresponding laser point is an underexposed point or an overexposed point, the photometric weight of the target laser point is determined to be 0.

[0035] In some possible implementations, the fusing the colored laser point cloud with the global laser point cloud according to the type of the corresponding laser point, the photometric weight, and the normalized photometric value corresponding to each color component of the corresponding laser point includes:

[0036] When the corresponding laser point is a valid point, obtaining a normalized photometric value corresponding to each color component of the target laser point after the N+1th fusion according to the Nth fusion photometric weight, the normalized photometric value corresponding to each color component of the corresponding laser point after the Nth fusion, the N+1th fusion photometric weight, and the normalized photometric value corresponding to each color component of the corresponding laser point after the N+1th fusion;

[0037] When the corresponding laser point is an underexposed point, determining a minimum value of the normalized photometric values corresponding to each color component of the corresponding laser point in the N+1 fusion, as the normalized photometric value corresponding to each color component of the target laser point in the N+1 fusion;

[0038] When the corresponding laser point is an overexposed point, the maximum value of the normalized photometric value corresponding to each color component of the corresponding laser point in the N+1 fusion is determined as the normalized photometric value corresponding to each color component of the target laser point in the N+1 fusion.

[0039] In some possible implementations, the photometric parameters further include a reference exposure time, a reference sensitivity, a radius attenuation function, a Bayer image sensitivity, and a Bayer image exposure time.

[0040] In a second aspect, the present application provides an exposure time prediction device, the device comprising:

[0041] A position prediction module, configured to obtain a second moment position of the camera based on the first moment position and first moment velocity of the camera;

[0042] A photometric value calculation module, configured to calculate an average photometric value of the colored laser point cloud within the target range at the second moment;

[0043] a time prediction module, configured to obtain a first predicted exposure time corresponding to the position at the second moment based on the average light metering value and the light metering parameters; and obtain a first adjustment value based on the Bayer image captured by the camera, the first adjustment value being used to predict the exposure time corresponding to the position at the second moment, wherein the laser point cloud in the Bayer image is not colorized;

[0044] The time prediction module is further used to determine the second predicted exposure time corresponding to the second moment position based on the exposure time corresponding to the first moment position, the first adjustment value, the first predicted exposure time and the coloring parameter, and the coloring parameter is used to indicate the coloring processing ratio of the laser point cloud within the target range.

[0045] The device can also be used to execute the information processing method as described in any implementation manner of the first aspect.

[0046] In a third aspect, the present application provides a simultaneous positioning and mapping system, which includes the exposure time prediction device described in the second aspect and can implement the method described in the first aspect of the present application or any possible implementation method of the first aspect.

[0047] In a fourth aspect, the present application provides a controller. The controller includes a processor and a memory. The memory is configured to store computer instructions; the processor is configured to execute the method described in the first aspect of the present application or any possible implementation of the first aspect according to the computer instructions.

[0048] In a fifth aspect, the present application provides a computer-readable medium, in which instructions are stored. When the computer-readable storage medium is run on a computer device, the computer device executes the method described in the first aspect of the present application or any possible implementation method of the first aspect.

[0049] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect of the present application or any possible implementation of the first aspect.

[0050] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods.

[0051] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0052] An embodiment of the present application provides an exposure time prediction method. On the one hand, this method predicts adjustment values for laser point clouds that have been color-processed and laser point clouds that have not been color-processed, thereby more accurately predicting the exposure time and improving the system construction quality; on the other hand, it combines the position and speed information provided by the inertial measurement unit to achieve more accurate positioning of the exposure position, thereby more accurately obtaining laser point cloud data for prediction, and improving the prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of an exposure time prediction method disclosed in an embodiment of the present application;

[0054] Figure 2 This is a flow chart of a laser point cloud coloring processing method disclosed in an embodiment of the present application;

[0055] Figure 3 A flowchart of a SLAM system working method disclosed in an embodiment of the present application;

[0056] Figure 4 This is a structural diagram of an exposure time prediction device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0058] The terms used in the following embodiments are for the purpose of describing specific embodiments only and are not intended to limit the present application. The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood to indicate or imply relative importance, the temporal order of operation, or implicitly indicate the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of such features.

[0059] First, some technical terms involved in the embodiments of this application are introduced.

[0060] Simultaneous Localization and Mapping (SLAM) is a technology that simultaneously localizes itself and constructs a map of the surrounding environment in an unknown environment. It is widely used in scenarios such as mobile robots, drones, autonomous driving, augmented reality (AR), and virtual reality (VR). SLAM systems typically operate collaboratively based on information from multiple environmental perception sensors collected in different dimensions, achieving real-time positioning and scene construction. For example, SLAM systems can include LiDAR (Light Detection and Ranging), cameras, and inertial measurement units (IMUs).

[0061] LiDAR, with its high resolution and support for multi-target tracking, can collect depth information in real time and construct a 3D model of the surrounding environment. This 3D model not only supports scene modeling and dynamic obstacle tracking, but can also be combined with ground truth detection technology. After projection (for example, onto camera images), it generates large datasets of annotated images for training target detection models, thereby improving the performance of target detection technology.

[0062] Cameras can capture two-dimensional image information, including color and lighting information, to more realistically reflect the characteristics of real-world scenes. There are many types of cameras, such as monocular cameras, binocular cameras, 360-degree surround view cameras, and surround-view fisheye cameras.

[0063] An inertial measurement unit (IMU) is a component that uses built-in sensors such as accelerometers, gyroscopes, and magnetometers to provide real-time motion data of a vehicle. Because IMUs enable precise positioning, navigation, and attitude control, they are widely used in applications such as autonomous driving, robotic navigation, and drone control.

[0064] SLAM systems typically require the collaboration of multiple sensors. For example, they use lidar to provide high-resolution 3D information, combined with 2D image data from cameras and other image sensors, and real-time motion data from IMUs to construct a 3D model of the surrounding environment in real time.

[0065] However, related technologies struggle to balance resource allocation across multiple sensors and computing devices. For example, the balance between a camera's exposure algorithm and frame rate is negatively correlated with the camera's exposure time. Longer exposure times mean longer acquisition cycles for single frames, corresponding to lower frame rates. High frame rates in SLAM systems can overload the system's communication and computational load, reducing the quality and efficiency of scene reconstruction. However, low frame rates slow the convergence of existing automatic exposure algorithms, reducing the sensor's range and granularity of light / brightness perception. In particular, in high dynamic range (HDR) scenes, where the difference between the maximum and minimum brightness levels is significant, existing automatic exposure algorithms struggle to cover this range. This can easily lead to overexposure or underexposure, preventing the effective preservation of rich color information. This can cause distortion or loss of detail in rendered results. This can also result in low saturation after color fusion of laser point clouds, significantly deviating from the real scene.

[0066] In view of this, the present application provides an exposure time prediction method and related equipment to solve the problem of reduced construction quality of SLAM systems in high dynamic range scenes.

[0067] Specifically, the method obtains the camera's second position at a second moment based on the camera's first position and speed, calculates the average photometric value of the color-processed laser point cloud within the target range at the second position, and obtains a first predicted exposure time corresponding to the second position based on the average photometric value and the photometric parameters. A first adjustment value is then obtained based on the uncolor-processed laser point cloud in the Bayer image, and the exposure time at the second position is determined based on the exposure time corresponding to the first position, the first adjustment value, the first predicted exposure time, and the color-processed ratio of the laser point cloud.

[0068] In this way, on the one hand, this method predicts the adjustment values for the laser point cloud that has been colored and the laser point cloud that has not been colored, thereby more accurately predicting the exposure time and improving the system construction quality; on the other hand, it combines the position and speed information provided by the inertial measurement unit to achieve more accurate positioning of the exposure position, thereby more accurately obtaining the laser point cloud data used for prediction, and improving the prediction accuracy.

[0069] The method provided in this application is introduced below with reference to specific embodiments.

[0070] Figure 1 This is an exposure time prediction method disclosed in an embodiment of the present application. The method specifically includes:

[0071] S102: The controller obtains the second position of the camera at a first moment according to the first position and first speed of the camera at a first moment.

[0072] SLAM systems support real-time construction of the surrounding environment while the vehicle is moving. This means that sensors such as cameras and lidar may be in different positions at different times, and the images captured by the cameras will reflect the characteristics of different locations at different times. Due to the complex lighting changes and details in nature, lighting conditions in different locations may also vary. Therefore, the exposure time in the camera exposure algorithm must be determined based on the actual lighting conditions to balance the efficiency and quality of the SLAM system's scene construction.

[0073] Specifically, the controller can obtain the first moment position and first moment velocity of the camera based on the inertial measurement unit, and obtain the second moment position of the camera based on the above data, for example, by combining a piecewise uniform acceleration motion model to predict the displacement of the camera between the first moment and the second moment. In one example, taking into account the characteristics of the collaborative lidar data acquisition, the first moment of the camera can select the earliest timestamp in the data set acquired by the lidar, for example, the moment corresponding to the first valid data in the acquisition cycle or unit time period. Valid data refers to data in the laser point cloud with a timestamp, after excluding invalid data segments (such as data segments displaying "NA") caused by unstable acquisition, and performing format conversion under preset rules (such as unified units, unified timestamp length, etc.).

[0074] At the same time, it is understood that the SLAM system in the embodiments of the present application supports the collaborative operation of sensors such as cameras and lidar. For the laser point cloud data collected by the lidar, the controller can project and convert it onto a two-dimensional Bayer image captured by the camera based on information such as the camera's internal parameters. The Bayer image refers to the original image captured by the camera in Bayer format. That is, the "laser point cloud on the image" mentioned below refers to the information after the laser point cloud is converted to the image, such as coordinate information, and will not be further described below.

[0075] It should be noted that in an embodiment of the present application, the controller may periodically send a request to the inertial measurement unit, camera, lidar, etc. to collect data, or the above modules may autonomously collect data and send it to the controller.

[0076] S104: The controller calculates the average photometric value of the colored laser point cloud within the target range at the second moment.

[0077] To achieve a more realistic construction effect, the SLAM system can combine color information from camera-captured images and add color information to the laser point cloud, which also includes depth information, through colorization processing. Colorization refers to the process of fusing the color information from the camera with the spatial coordinates of the laser point cloud using algorithms such as interpolation and image fusion to produce a laser point cloud with color information. The color of objects in the surrounding environment is determined by the wavelength of visible light they reflect. Furthermore, light intensity affects the brightness / brightness of the color. Higher intensity reflects or transmits more light from the object's surface, resulting in higher brightness; lower intensity results in lower brightness. In other words, the addition or change of color information corresponds to different lighting conditions. Therefore, after colorization processing, the lighting conditions at each laser point in the laser point cloud change compared to the lighting conditions at the corresponding location in the uncolored laser point cloud or the Bayer image captured by the camera, necessitating corresponding adjustments when estimating exposure time.

[0078] Among them, the photometric value is an indicator that measures the light intensity of a scene or an object surface, and represents different lighting conditions. Therefore, the controller needs to obtain the photometric value information of the camera near the position at the second moment to represent the lighting conditions at the position at the second moment, so as to predict the appropriate exposure time of the camera at the second position based on the actual lighting conditions. Specifically, since the measurement range of the laser radar is relatively wide, there is a high probability that the area near the position of the camera at the second moment has been covered before the second moment. The controller can measure the lighting conditions of the position based on the average photometric value of the laser point cloud that has been color-processed within the target range of the camera's position at the second moment. It should be noted that the embodiments of the present application do not make specific limitations on how the controller obtains the photometric value of the laser point in the laser point cloud.

[0079] As an example, the photometric value of each laser point in the laser point cloud can be calculated as follows:

[0080]

[0081] in, is the light measurement value of the laser point. is the black level value, which is the response value of the sensor (such as lidar) when there is no light. is the response coefficient of the sensor, The sensitivity of the camera at the time of shooting. is the exposure time of the camera shooting the moment, is the radius attenuation function, In this way, the normalized photometric value of each laser point can be obtained, which is used to characterize the lighting conditions at each laser point at the same granularity.

[0082] Furthermore, after obtaining the photometric values at each laser point, the controller can calculate the average photometric value of the laser point cloud within the target range based on the individual photometric values. For example, the controller can perform an averaging operation based on all photometric values, or based on a portion of the photometric values filtered by a threshold. The averaging operation can be performed directly based on the total photometric value and the number of data points, or it can be performed using a weighted average operation based on a set weight, which is not limited in this embodiment of the present application.

[0083] S106: The controller obtains a first predicted exposure time corresponding to the second moment position based on the average light metering value and the light metering parameters.

[0084] After obtaining the average light measurement value, the controller can predict the first predicted exposure time corresponding to the second moment position based on the average light measurement value and the light measurement parameters as the exposure time predicted based on the laser point cloud after color assignment processing. Specifically, the exposure time can be obtained according to the following formula:

[0085]

[0086] in, This is the average metered exposure value. is the response coefficient of the sensor, is the black level value, is the middle brightness value, which is generally , The number of bytes in the image. For example, in an 8-bit image, the median brightness value is =128, the image brightness range is 0 to 255. In this way, by comparing the actual brightness with the intermediate brightness, the exposure parameters can be dynamically adjusted so that the brightness output signal adapts to the dynamic range of the sensor.

[0087] In some possible implementations, the controller can determine the first predicted exposure time based on the color-processed laser point cloud that the controller ultimately predicts based on the exposure time threshold and the theoretical exposure time obtained by the above formula. If the theoretical exposure time is less than the exposure time threshold, the theoretical exposure time is determined to be the first predicted exposure time based on the color-processed laser point cloud; conversely, if the theoretical exposure time is greater than or equal to the exposure time threshold, the exposure time threshold is determined to be the first predicted exposure time. That is, the controller can set the maximum value of the theoretical exposure time or the first predicted exposure time. In this way, large deviations in the exposure time predicted based on the color-processed laser point cloud can be prevented, thereby improving the accuracy and stability of the exposure time prediction method in various possible scenarios.

[0088] The controller can set the exposure time threshold based on the camera's frame rate. For example, when the camera's frame rate is 30 Hz, the exposure time threshold can be set to 30 milliseconds. Alternatively, relevant technicians can determine the exposure time threshold through experiments or experience based on image clarity requirements. For example, when the camera moves, the angle value corresponding to the maximum displacement value of the pixel movement in the image divided by the angular velocity of the camera movement is used as the exposure time threshold. This application does not impose strict restrictions on this.

[0089] S108: The controller obtains a first adjustment value according to the Bayer image captured by the camera, where the laser point cloud in the Bayer image is not colorized.

[0090] In some cases, the LiDAR may not have fully covered the entire range near the camera's position at the second moment before the second moment, or the controller may not have performed color processing on all laser point clouds when predicting the exposure time. Therefore, the controller can also predict the exposure time based on the laser point cloud in the Bayer image captured by the camera without color processing.

[0091] Specifically, the controller can predict the exposure time based on methods such as the brightness histogram of the Bayer image, gradient and edge feature algorithms, and information entropy maximization to obtain a first adjustment value. The first adjustment value refers to the exposure time at the second time position predicted based on the uncolored point cloud, compared to the exposure time at the first time position (e.g., the difference). For example, the controller can identify the pixel distribution density in key areas by analyzing the brightness histogram of the Bayer image and dynamically adjust the exposure time to shift the main peak of the histogram toward a preset target brightness range (e.g., near mid-gray values). For another example, the controller can use image gradient (edge intensity) as an exposure quality evaluation metric, prioritizing the preservation of details (e.g., texture and contours) in high-contrast areas. If the gradient is low, the exposure time is increased to enhance image details. The first adjustment value in the embodiment of the present application is obtained based on the uncolored laser point cloud of the Bayer image. The actual method used by the controller to predict the first adjustment value is not limited, and those skilled in the art can adopt an appropriate prediction method based on actual circumstances.

[0092] It should be noted that there is no particular order for step S106 and step S108. The controller may first execute step S106 to obtain the first predicted exposure time, or may first execute step S108 to obtain the first adjustment value.

[0093] S110: The controller determines a second predicted exposure time corresponding to a second moment position according to the exposure time corresponding to the first moment position, the first adjustment value, the first predicted exposure time, and the coloring parameter.

[0094] In some possible implementations, the controller can first determine the second adjustment value based on the first predicted exposure time and the exposure time corresponding to the first moment position, and then determine the second predicted exposure time corresponding to the second moment position based on the second adjustment value, the first adjustment value, the color assignment parameter and the exposure time corresponding to the first moment position.

[0095] Specifically, the second adjustment value refers to the adjustment value (e.g., the difference) between the exposure time corresponding to the second moment position (i.e., the first predicted exposure time) predicted based on the laser point cloud that has undergone color processing and the exposure time corresponding to the first moment position. Thus, the first adjustment value is predicted based on the uncolored laser point cloud, and the second adjustment value is predicted based on the colored laser point cloud. The difference between the two can be weighted by the ratio of the colored and uncolored laser point clouds, and the final predicted value is obtained in combination with the exposure time corresponding to the first moment position. This method allows for the flexible use of the two prediction methods to more accurately determine the final exposure time, taking into account actual conditions.

[0096] In a possible example, the exposure time corresponding to the second moment position finally predicted by the controller (ie, the second predicted exposure time) can be expressed by the following formula:

[0097] (1- )( )

[0098] in, This is the second predicted exposure time, is the exposure time corresponding to the first moment position, It is a coloring parameter, which is used to indicate the proportion of the laser point cloud that has not been colored within the target position at the second moment. is the first adjustment value, is the second adjustment value, It is the first predicted exposure time of the laser point cloud based on color processing.

[0099] It should be noted that the above is only an example to show how to determine the final exposure time prediction value based on the second adjustment value, the first adjustment value, the color assignment parameter and the exposure time corresponding to the first moment position. In actual application, relevant technical personnel in this field can flexibly change the weighting form of the above adjustment values according to actual conditions, such as multiplying or raising the color assignment parameter to a power, or deforming the weights of the first adjustment value and the second adjustment value in other ways. The embodiments of the present application do not impose any limitations on this.

[0100] Based on the above description, an embodiment of the present application provides an exposure time prediction method. On the one hand, this method predicts adjustment values for the laser point cloud that has been color-processed and the laser point cloud that has not been color-processed, thereby more accurately predicting the exposure time and improving the system construction quality; on the other hand, it combines the position and speed information provided by the inertial measurement unit to achieve more accurate positioning of the exposure position, thereby more accurately obtaining the laser point cloud data used for prediction, and improving the prediction accuracy.

[0101] Figure 1 The corresponding embodiment details how the controller combines the laser point cloud that has not been colorized and the laser point cloud that has been colorized to more accurately predict the exposure time. Next, the colorization of the laser point cloud will be described in detail.

[0102] See also Figure 2 The flowchart of a laser point cloud coloring processing method shown in FIG. 1 includes:

[0103] S202: The controller projects the unobstructed laser point cloud in the global laser point cloud into a Bayer image based on the camera pose and camera parameters.

[0104] Since the image with color information captured by the camera is a two-dimensional image, the controller can project the laser point cloud available in the global laser point cloud in the three-dimensional information onto the Bayer image captured by the camera so that the laser point cloud obtains color information.

[0105] Specifically, the controller can perform coordinate conversion based on the camera pose, camera parameters, and projection model. The camera pose includes the positioning information of the image, and based on this, the positional relationship between the coordinates of the laser point in the laser point cloud and the image captured by the camera is determined, for example, to obtain a rotation and / or translation matrix; the camera parameters, especially the internal parameters of the camera, are used to achieve conversion between coordinate systems; the projection model is determined according to the imaging principle of the camera to reduce distortion and obtain more accurate images and coordinates. For example, the imaging methods of fisheye cameras and ordinary monocular cameras are different, and the selected projection models are also different.

[0106] S204: The controller performs color component interpolation on the unobstructed laser point cloud according to the Bayer image to obtain a colored laser point cloud.

[0107] Color components include red, green, and blue. Due to the imaging characteristics of the human eye, to better reflect reality, the green component is given a larger proportion in image interpolation, with the ratio of red, green, and blue being approximately 1:2:1. This principle allows the color components in a Bayer image to be interpolated, resulting in coordinate information for each color component. For example, the horizontal and vertical coordinate values of the red component are all even numbers, the horizontal and vertical coordinate values of the blue component are all odd numbers, and the remaining positions are green components.

[0108] Specifically, assuming that the coordinate value of the laser point is p(x, y), the color interpolation coordinate information obtained according to the above method is as follows:

[0109] For the red component, there is an expression:

[0110]

[0111] in, Represents the coordinate value of the red component, Represents rounding down x. and These are two parameters that change with the coordinates, and their specific values are as follows:

[0112]

[0113] For the blue component, there is an expression:

[0114]

[0115] in, Represents the coordinate value of the blue component, Represents rounding down x. and These are two parameters that change with the coordinates, and their specific values are as follows:

[0116]

[0117] For the green component, there is an expression:

[0118]

[0119] in, Represents the coordinate value of the green component, Indicates that x is rounded down. The green component has two cases: the horizontal coordinate is an even number and the vertical coordinate is an odd number; or the horizontal coordinate is an odd number and the vertical coordinate is an even number. The corresponding parameter calculation method is as follows:

[0120]

[0121]

[0122] In this way, the method can directly interpolate the color components from the Bayer image without performing a demosaic operation, thereby improving processing efficiency.

[0123] After the above interpolation processing, color information can be added to the laser point cloud projected onto the Bayer image to obtain a colored laser point cloud.

[0124] S206: The controller calculates the normalized photometric value corresponding to each color component of the laser point in the colored laser point cloud according to the photometric parameters.

[0125] As mentioned above, the color of an object is correlated with lighting conditions, and lighting or exposure conditions will affect the saturation of the assigned color. Therefore, the controller can calculate the normalized metering value corresponding to each color component based on the metering parameters to more accurately determine the exposure of the image and perform color processing accordingly.

[0126] Specifically, taking the red component as an example, the normalized photometric value of the red component of a laser point can be:

[0127]

[0128] in, That is the photometric value of the red component after interpolation of the laser point, is the black level value, which is the response value of the sensor (such as lidar) when there is no light. is the response coefficient of the sensor, The light sensitivity of the camera at the time of shooting. is the exposure time of the camera shooting the moment, is the radius attenuation function, The above-mentioned light measurement parameters may be known or obtained through external input, and the present embodiment does not impose any limitation on this.

[0129] The normalized photometric values of the blue component and the green component of the laser point can also be obtained in this way.

[0130] In this way, the controller can obtain the normalized photometric value of each color component of each laser point, which is used to characterize the lighting conditions corresponding to each color component of each laser point at the same granularity, thereby determining whether the laser point will produce overexposure or underexposure, and then perform processing.

[0131] S208: The controller determines the type of the laser point according to the normalized photometric value corresponding to each color component of the laser point in the colored laser point cloud.

[0132] If the collected light signal exceeds the recording capacity of a sensor (such as a camera), the image will be overexposed or underexposed. Specifically, overexposure occurs when the brightness of highlight areas in an image exceeds the sensor's recording capacity, resulting in a loss of detail and appearing as pure white or whitish blocks. Underexposure occurs when the brightness of dark areas falls below the sensor's recording capacity, causing details to be submerged in black (such as shadows and dark objects). Therefore, whether the laser spot is overexposed or underexposed will affect the retention of color information, thereby affecting the smooth color rendering process, so this needs to be determined.

[0133] Thus, the controller can set photometric thresholds to determine the type of laser spot. For example, a first photometric threshold and a second photometric threshold can be set. The first photometric threshold is less than the second photometric threshold, and the range between the first and second photometric thresholds (including both thresholds) is the valid photometric range. Laser spots whose photometric values for each color component fall within this range are considered valid spots. The magnitudes of the first and second photometric thresholds are related to the measured black and white levels. For example, by shooting in a completely dark environment at different ISO settings, the black and white levels for different ISO settings can be obtained. The first photometric threshold can then be set to the minimum representable value + a first buffer margin, and the second photometric threshold can be set to the maximum representable value - a second buffer margin. Both the first and second buffer margins are greater than zero. For example, based on a histogram, the first buffer margin can be an interpolation of the lowest 1% luminance value and the black level value, while the second buffer margin can be an interpolation of the highest 1% luminance value and the white level value.

[0134] Specifically, if the normalized photometric values of all color components of a laser point in a color-assigned laser point cloud are no less than a first photometric threshold and no greater than a second photometric threshold, the laser point in the color-assigned laser point cloud is determined to be a valid point, and the first photometric threshold is less than the second photometric threshold. If the normalized photometric value of any color component of a laser point in the color-assigned laser point cloud is less than the first photometric threshold, the laser point in the color-assigned laser point cloud is determined to be an underexposed point. If the normalized photometric value of any color component of a laser point in the color-assigned laser point cloud is greater than the second photometric threshold, the laser point in the color-assigned laser point cloud is determined to be an overexposed point. This allows for a more comprehensive and accurate determination of the type of laser points in a color-assigned laser point cloud.

[0135] In some possible implementations, when the laser point is determined to be an underexposed point, the light metering value of the point can be unified as a first light metering threshold; when the laser point is determined to be an overexposed point, the light metering value of the point can be unified as a second light metering threshold, thereby performing unified correction and simplifying calculations.

[0136] S210: The controller determines a photometric weight of the target laser point according to the orientation parameter of the target laser point in the global laser point cloud and the type of the corresponding laser point in the coloring laser point cloud.

[0137] Since the target laser point in the global laser point cloud may be surrounded by multiple laser points in the color point cloud, in order to more comprehensively and accurately obtain the color and lighting information of the target laser point, the target laser point can be fused according to the corresponding laser points in the multiple color point clouds. The fusion needs to be performed according to a certain ratio or weight. In particular, each newly added corresponding laser point may cause the photometric weight of the target laser point in the global laser point cloud to change.

[0138] Specifically, the process of determining the photometric weight can be performed as follows:

[0139] In some possible implementations, if the corresponding laser point is a valid point, the controller may determine the N+1th fusion photometric weight of the target laser point based on the orientation parameter and the Nth fusion photometric weight of the target laser point, where N is a natural number. When N=0, the initial photometric weight of the target laser point is 0. The orientation parameters include the normal vector of the target laser point, the direction vector between the target laser point and the viewpoint, the distance between the target laser point and the viewpoint, the observation angle parameter, and the observation distance parameter. That is, the adjustment value of the photometric weight of the target laser point during the N+1th fusion with the Nth fusion is determined based on the orientation parameter, as shown in the following formula:

[0140]

[0141] in, This is the adjustment value of the metering weight. is the normal vector of the target laser point, is the direction vector between the target laser point and the viewpoint, is the distance between the target laser point and the viewpoint, and These are the preset viewing angle parameters and observation distance parameters, respectively. In this way, a more accurate weight can be determined based on the azimuth relationship between the viewpoint and the target laser point.

[0142] After obtaining the adjusted value of the photometric weight, the photometric weight of the N+1th fusion can be obtained as follows:

[0143]

[0144] in, That is the metering weight of the N+1th fusion, is the metering weight of the Nth fusion, The adjustment value for the metering weight.

[0145] In some possible implementations, if the corresponding laser point is an underexposed point or an overexposed point, the controller may determine that the photometric weight of the target laser point is 0, thereby reducing the influence of the abnormal point on the color information fusion.

[0146] S212: The controller fuses the colored laser point cloud with the global laser point cloud according to the type of the corresponding laser point, the photometric weight, and the normalized photometric value corresponding to each color component of the corresponding laser point to obtain a colored laser point cloud.

[0147] The controller can update the metering weight and metering value based on the result of the previous fusion and the adjustment value obtained in the next fusion. The following describes the method for updating the metering value.

[0148] Specifically, when the corresponding laser point is a valid point, the controller can obtain the normalized photometric value corresponding to each color component of the N+1 fused target laser point based on the Nth fusion photometric weight, the normalized photometric value corresponding to each color component of the Nth fusion corresponding laser point, the N+1th fusion photometric weight, and the normalized photometric value corresponding to each color component of the N+1th fusion corresponding laser point. This method can be represented by the following formula:

[0149]

[0150] in, That is, the normalized photometric value corresponding to each color component of the N+1th fusion target laser point, is the metering weight of the Nth fusion, is the normalized photometric value corresponding to each color component of the corresponding laser point for the Nth fusion, The N+1th fusion metering weight.

[0151] In some possible implementations, when the corresponding laser point is an underexposed point, the controller can determine that the minimum value of the normalized photometric value corresponding to each color component of the corresponding laser point in the N+1 fusion is the normalized photometric value corresponding to each color component of the target laser point in the N+1 fusion.

[0152] In some possible implementations, when the corresponding laser point is an overexposed point, the controller can determine the maximum value of the normalized photometric value corresponding to each color component of the corresponding laser point in the N+1 fusions, which is the normalized photometric value corresponding to each color component of the target laser point in the N+1 fusion.

[0153] After obtaining the normalized photometric value corresponding to each color component of the N+1th fusion target laser point, the controller can substitute the coordinates of each color component into the global laser point according to the corresponding relationship between the photometric values of each color, so as to fuse the colored laser point cloud with the global laser point cloud and obtain a colored laser point cloud.

[0154] Based on the above description, an embodiment of the present application provides a color processing method for laser point clouds. On the one hand, this method directly uses the Bayer image for interpolation, which simplifies the coloring operation; on the other hand, the types of laser points in the laser point cloud are classified according to the normalized photometric values of each color component, and corresponding fusion strategies are adopted according to the categories, thereby retaining color details and improving the quality of color processing.

[0155] Based on the aforementioned exposure time prediction method and laser point cloud color processing method, the present application also provides a specific scenario embodiment combining the above two methods.

[0156] The two methods provided in this application can be applied in a simultaneous localization and mapping SLAM system. The SLAM system includes multiple sensors working together, such as lidar, camera, and inertial measurement unit, to achieve real-time positioning and construction of the surrounding environment, and is widely used in scenarios such as autonomous driving of vehicles. Figure 3 The following is a flow chart of a SLAM system working method disclosed in an embodiment of the present application, which includes:

[0157] S302: The SLAM system performs data acquisition to obtain the Bayer image captured by the camera, the global laser point cloud collected by the lidar, and the positioning information obtained by the inertial measurement unit.

[0158] S304: The SLAM system performs data preprocessing operations.

[0159] In some possible implementations, the SLAM system can combine the Bayer image captured by the camera and the laser point cloud captured by the lidar with the positioning information obtained by the inertial measurement unit to obtain an image including pose information and a stitched point cloud.

[0160] Specifically, the SLAM system can first run the Lidar-Inertial Odometry (LIO) system (such as fastlio, lio-sam, etc.) to output the radar pose and stitched point cloud based on the IMU messages and laser point cloud messages subscribed to by the LIO thread. Then, the SLAM system can subscribe to the IMU messages, raw images, and radar pose messages output by the Visual-Inertial Odometry (VIO) system (such as VINS, SVO2.0, etc.) based on the VIO thread to obtain an image containing pose information. For example, the SLAM system can first convert a 10-bit Bayer image into an 8-bit grayscale image, and then run the VIO system (such as VINS, SVO2.0, etc.) to output the image.

[0161] The process of converting a Bayer image to a grayscale image can be as follows: the SLAM system first extracts information about the green channel / component in the Bayer image and fills in missing pixels, then generates an initial grayscale image through bilinear downsampling. The SLAM system then calculates the histogram, calculates the cumulative histogram, equalizes the histogram, and calculates the mapping function to generate a balanced grayscale image.

[0162] S306: The SLAM system performs color processing on the global laser point cloud based on the Bayer image and pose information.

[0163] The SLAM system can perform color processing on the global laser point cloud based on the processed data obtained in S304. The related methods are similar to Figure 2 The corresponding laser point cloud coloring processing method is similar and will not be repeated here. It is worth noting that when calculating the normalized photometric value of each color in the embodiment of the present application, the exposure time at the moment of Bayer image capture can be obtained by the exposure time prediction method provided by the SLAM system based on the previous capture.

[0164] S308: The SLAM system predicts the exposure time based on the Bayer image and the positioning information.

[0165] The SLAM system can be based on the Figure 1 The exposure time prediction method shown predicts the exposure time for the next shot. The related method is similar to the previous one and will not be described in detail here. It should be noted that when calculating the average metering value in S308, the calculation can be performed based on the data calculated in S306.

[0166] S310: The SLAM system performs adaptive local tone mapping on the global laser point cloud after color processing.

[0167] In some possible implementations, the SLAM system may further adjust the colored laser point cloud obtained in S306 .

[0168] Specifically, the SLAM system can divide the global point cloud into equally spaced blocks according to space, for example, the spacing is Dx, Dy and Dz respectively, and the subscript of each color block is (i, j, k). Then, the SLAM system can statistically analyze the histogram distribution of the color blocks and calculate the maximum and minimum values. Next, the SLAM system can use the threshold (CLAHE) method on the histogram distribution of the color blocks to calculate the cumulative distribution histogram before and after truncation and the matching calculation mapping relationship. Finally, the SLAM system can bilinearly interpolate the mapping values of the current block and the neighborhood block for all points in the color blocks to obtain the adjusted color blocks and point clouds. In this way, the color information can be more accurately fused according to the local contrast and detail information to obtain a laser point cloud that is closer to the real situation.

[0169] Based on the above description, this scenario embodiment can be combined with the exposure time prediction method and laser point cloud color processing method provided above to improve the effect of the SLAM system in constructing the surrounding environment scene in practical applications.

[0170] Based on the above content, the present application further provides an exposure time prediction device. The device of the present application is described in detail below with reference to the accompanying drawings.

[0171] See also Figure 4 The structure diagram of an exposure time prediction device 400 is shown, and the device may include:

[0172] A position prediction module 402 is configured to obtain a second moment position of the camera based on the first moment position and the first moment velocity of the camera;

[0173] The photometric value calculation module 404 is used to calculate the average photometric value of the laser point cloud that has been color-processed within the target range at the second moment;

[0174] The time prediction module 406 is used to obtain a first predicted exposure time corresponding to the position at the second moment based on the average metering value and the metering parameters; and to obtain a first adjustment value based on the Bayer image captured by the camera, the first adjustment value being used to predict the exposure time corresponding to the position at the second moment, and the laser point cloud in the Bayer image is not color-processed; and to determine a second predicted exposure time corresponding to the position at the second moment based on the exposure time corresponding to the position at the first moment, the first adjustment value, the first predicted exposure time and the color-processing parameter, the color-processing parameter being used to indicate the color-processing ratio of the laser point cloud within the target range.

[0175] In some possible implementations, when the time prediction module 406 determines the second predicted exposure time corresponding to the second moment position based on the exposure time corresponding to the first moment position, the first adjustment value, the first predicted exposure time, and the coloring parameter, it is specifically configured to:

[0176] The second adjustment value is determined according to the first predicted exposure time and the exposure time corresponding to the first moment position, and then the second predicted exposure time corresponding to the second moment position is determined according to the exposure time corresponding to the first moment position, the color assignment parameter, the first adjustment value and the second adjustment value.

[0177] In some possible implementations, when the time prediction module 406 obtains the first predicted exposure time corresponding to the second moment position based on the average light measurement value and the light measurement parameter, it is specifically configured to:

[0178] The theoretical exposure time is obtained based on the average light metering value and the light metering parameters; if the theoretical exposure time is less than the exposure time threshold, the theoretical exposure time is determined to be the first predicted exposure time corresponding to the second moment position; if the theoretical exposure time is greater than or equal to the exposure time threshold, the exposure time threshold is determined to be the first predicted exposure time corresponding to the second moment position.

[0179] In some possible implementations, when the time prediction module 406 obtains the theoretical exposure time according to the average light measurement value and the light measurement parameters, it is determined specifically according to the following formula:

[0180]

[0181] in, is the middle brightness value, is the black level value and is the sensor response coefficient, This is the theoretical exposure time.

[0182] In some possible implementations, the device further includes a coloring processing module 408, configured to:

[0183] Based on the camera pose and camera parameters, the unobstructed laser point cloud in the global laser point cloud is projected onto a Bayer image. The color components of the unobstructed laser point cloud are interpolated based on the Bayer image to obtain a colored laser point cloud. The color components include red, green, and blue. Then, the normalized photometric value corresponding to each color component of the laser point in the colored laser point cloud is calculated based on the photometric parameters. The type of the laser point is determined based on the normalized photometric value corresponding to each color component of the laser point in the colored laser point cloud. The photometric weight of the target laser point is then determined based on the orientation parameters of the target laser point in the global laser point cloud and the type of the corresponding laser point in the colored laser point cloud. Finally, the colored laser point cloud is fused with the global laser point cloud based on the type of the corresponding laser point, the photometric weight, and the normalized photometric value corresponding to each color component of the corresponding laser point to obtain a colored laser point cloud.

[0184] In some possible implementations, when the coloring processing module 408 determines the type of the laser point according to the normalized photometric value corresponding to each color component of the laser point in the colored laser point cloud, it is specifically configured to:

[0185] If the normalized photometric values of all color components of the laser point in the color laser point cloud are not less than the first photometric threshold and not greater than the second photometric threshold, the laser point in the color laser point cloud is determined to be a valid point, and the first photometric threshold is less than the second photometric threshold; if the normalized photometric value of any color component of the laser point in the color laser point cloud is less than the first photometric threshold, the laser point in the color laser point cloud is determined to be an underexposed point; if the normalized photometric value of any color component of the laser point in the color laser point cloud is greater than the second photometric threshold, the laser point in the color laser point cloud is determined to be an overexposed point.

[0186] In some possible implementations, the coloring processing module 408 determines the photometric weight of the target laser point based on the orientation parameters of the target laser point in the global laser point cloud and the type of the corresponding laser point in the coloring laser point cloud, specifically for:

[0187] If the corresponding laser point is a valid point, the N+1th fusion photometry weight of the target laser point is determined according to the azimuth parameter and the Nth fusion photometry weight of the target laser point. N is a natural number. When N=0, the initial photometry weight of the target laser point is 0. The azimuth parameters include the normal vector of the target laser point, the direction vector between the target laser point and the viewpoint, the distance between the target laser point and the viewpoint, the observation angle parameter, and the observation distance parameter. If the corresponding laser point is an underexposed point or an overexposed point, the photometry weight of the target laser point is determined to be 0.

[0188] In some possible implementations, the color processing module 408 is specifically configured to:

[0189] When the corresponding laser point is a valid point, the normalized photometric value corresponding to each color component of the N+1th fused target laser point is obtained according to the Nth fusion photometric weight, the normalized photometric value corresponding to each color component of the Nth fusion corresponding laser point, the N+1th fusion photometric weight, and the normalized photometric value corresponding to each color component of the N+1th fusion corresponding laser point; when the corresponding laser point is an underexposed point, the minimum value of the normalized photometric value corresponding to each color component of the corresponding laser point in the N+1th fusion is determined as the normalized photometric value corresponding to each color component of the N+1th fusion target laser point; when the corresponding laser point is an overexposed point, the maximum value of the normalized photometric value corresponding to each color component of the corresponding laser point in the N+1th fusion is determined as the normalized photometric value corresponding to each color component of the N+1th fusion target laser point.

[0190] Based on the aforementioned methods and apparatus, this application also provides a controller. This controller may be, for example, an electronic control unit (ECU). The controller includes a processor and a memory. The memory stores computer-readable instructions, and the processor is configured to execute the computer-readable instructions to perform the aforementioned methods. In some examples, the controller is configured to implement the functions of the aforementioned apparatus.

[0191] The present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium. When the computer-readable storage medium is executed on a computer device, the computer device executes the above method.

[0192] The present application provides a computer program product comprising instructions, which, when executed on a computer device, enables the computer device to perform the above method.

[0193] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0194] Through the above description of the embodiments, those skilled in the art will clearly understand that the present application can be implemented using software plus necessary general-purpose hardware. Of course, it can also be implemented using dedicated hardware, including application-specific integrated circuits, dedicated CPUs, dedicated memories, and dedicated components. Generally speaking, any function performed by a computer program can be easily implemented using corresponding hardware. Moreover, the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, software program implementation is often the preferred implementation method. Based on this understanding, the technical solution of the present application, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored on a readable storage medium, such as a computer floppy disk, USB flash drive, removable hard disk, ROM, RAM, magnetic disk, or optical disk, and includes a number of instructions for causing a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in the various embodiments of the present application.

[0195] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0196] The computer program product includes one or more computer instructions. When loaded and executed on a computer, the computer program instructions fully or partially perform the processes or functions described in the embodiments of the present application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium capable of computer storage, or a data storage device such as a training device or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, hard disk, or magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0197] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for predicting exposure time, characterized in that: The method comprises: Obtaining a second moment position of the camera according to the first moment position and the first moment velocity of the camera; Calculate the average photometric value of the colored laser point cloud within the target range at the second moment; Obtaining a first predicted exposure time corresponding to the second moment position based on the average light metering value and the light metering parameter; Obtaining a first adjustment value based on a Bayer Raw Image captured by the camera, wherein the first adjustment value is used to predict an exposure time corresponding to the position at the second moment, wherein the laser point cloud in the Bayer Raw Image is not colorized; The second predicted exposure time corresponding to the second moment position is determined according to the exposure time corresponding to the first moment position, the first adjustment value, the first predicted exposure time and the coloring parameter, wherein the coloring parameter is used to indicate the coloring processing ratio of the laser point cloud within the target range.

2. The method according to claim 1, characterized in that The determining, according to the exposure time corresponding to the first moment position, the first adjustment value, the first predicted exposure time, and the coloring parameter, of the second predicted exposure time corresponding to the second moment position includes: determining a second adjustment value according to the first predicted exposure time and the exposure time corresponding to the first moment position; A second predicted exposure time corresponding to the second moment position is determined according to the exposure time corresponding to the first moment position, the coloring parameter, the first adjustment value, and the second adjustment value.

3. The method according to claim 1, characterized in that The obtaining, based on the average light metering value and the light metering parameter, a first predicted exposure time corresponding to the second moment position includes: Obtaining a theoretical exposure time according to the average light measurement value and the light measurement parameters; If the theoretical exposure time is less than the exposure time threshold, determining the theoretical exposure time as the first predicted exposure time corresponding to the second moment position; If the theoretical exposure time is greater than or equal to the exposure time threshold, the exposure time threshold is determined to be the first predicted exposure time corresponding to the second moment position.

4. The method according to claim 3, characterized in that The photometric parameters include the median brightness value , black level value and sensor response coefficient , the intermediate brightness value is related to the Bayer image, the black level value is the response value of the sensor when there is no light, and the theoretical exposure time Obtained according to the following formula: 。 5. The method according to claim 1, wherein The method further comprises: Projecting an unobstructed laser point cloud in the global laser point cloud to the Bayer image based on the camera pose and camera parameters; Performing color component interpolation on the unobstructed laser point cloud according to the Bayer image to obtain a colored laser point cloud, wherein the color components include a red component, a green component, and a blue component; Calculating the normalized photometric value corresponding to each color component of the laser point in the color-imparting laser point cloud according to the photometric parameters; Determining the type of the laser point according to the normalized photometric value corresponding to each color component of the laser point in the colored laser point cloud; Determining a photometric weight of the target laser point according to an orientation parameter of the target laser point in the global laser point cloud and a type of a corresponding laser point in the coloring laser point cloud; According to the type of the corresponding laser point, the photometric weight and the normalized photometric value corresponding to each color component of the corresponding laser point, the colored laser point cloud is fused with the global laser point cloud to obtain the colored laser point cloud.

6. The method according to claim 5, characterized in that The determining the type of the laser point according to the normalized photometric value corresponding to each color component of the laser point in the color-assigned laser point cloud includes: If the normalized photometric values of all color components of the laser point in the color-imparting laser point cloud are not less than a first photometric threshold and not greater than a second photometric threshold, determining that the laser point in the color-imparting laser point cloud is a valid point, and the first photometric threshold is less than the second photometric threshold; If the normalized photometric value of any color component of the laser point in the color-imparting laser point cloud is less than the first photometric threshold, determine that the laser point in the color-imparting laser point cloud is an underexposed point; If the normalized photometric value of any color component of the laser point in the color-imparting laser point cloud is greater than the second photometric threshold, it is determined that the laser point in the color-imparting laser point cloud is an overexposed point.

7. The method according to claim 6, characterized in that The determining the photometric weight of the target laser point according to the orientation parameter of the target laser point in the global laser point cloud and the type of the corresponding laser point in the coloring laser point cloud includes: If the corresponding laser point is a valid point, determining the N+1th fusion photometry weight of the target laser point according to the orientation parameter and the Nth fusion photometry weight of the target laser point, where N is a natural number. When N=0, the initial photometry weight of the target laser point is 0. The orientation parameters include a normal vector of the target laser point, a direction vector between the target laser point and the viewpoint, a distance between the target laser point and the viewpoint, an observation viewing angle parameter, and an observation distance parameter. If the corresponding laser point is an underexposed point or an overexposed point, the photometric weight of the target laser point is determined to be 0.

8. The method according to claim 7, characterized in that The fusing the colored laser point cloud with the global laser point cloud according to the type of the corresponding laser point, the photometric weight, and the normalized photometric value corresponding to each color component of the corresponding laser point includes: When the corresponding laser point is a valid point, obtaining a normalized photometric value corresponding to each color component of the target laser point after the N+1th fusion according to the Nth fusion photometric weight, the normalized photometric value corresponding to each color component of the corresponding laser point after the Nth fusion, the N+1th fusion photometric weight, and the normalized photometric value corresponding to each color component of the corresponding laser point after the N+1th fusion; When the corresponding laser point is an underexposed point, determining a minimum value of the normalized photometric values corresponding to each color component of the corresponding laser point in the N+1 fusion, as the normalized photometric value corresponding to each color component of the target laser point in the N+1 fusion; When the corresponding laser point is an overexposed point, the maximum value of the normalized photometric value corresponding to each color component of the corresponding laser point in the N+1 fusion is determined as the normalized photometric value corresponding to each color component of the target laser point in the N+1 fusion.

9. The method according to any one of claims 1 to 8, characterized in that The photometric parameters further include a reference exposure time, a reference sensitivity, a radius attenuation function, a Bayer image sensitivity, and a Bayer image exposure time.

10. An exposure time prediction device, characterized in that: include: A position prediction module, configured to obtain a second moment position of the camera based on the first moment position and first moment velocity of the camera; A photometric value calculation module, configured to calculate an average photometric value of the colored laser point cloud within the target range at the second moment; a time prediction module, configured to obtain a first predicted exposure time corresponding to the second moment position based on the average light metering value and the light metering parameters; and obtaining a first adjustment value based on the Bayer image captured by the camera, wherein the first adjustment value is used to predict the exposure time corresponding to the position at the second moment, wherein the laser point cloud in the Bayer image is not colorized; The time prediction module is further used to determine the second predicted exposure time corresponding to the second moment position based on the exposure time corresponding to the first moment position, the first adjustment value, the first predicted exposure time and the coloring parameter, and the coloring parameter is used to indicate the coloring processing ratio of the laser point cloud within the target range.

Images