Image processing method and device, storage medium and electronic equipment

Abstract

Embodiments of the present disclosure disclose an image processing method and device, a storage medium and an electronic device. The image processing method comprises: controlling an image acquisition sensor to alternately acquire original images with consistent brightness based on at least two exposure strategies in a time division multiplexing manner to obtain an original image sequence; and performing interval frame extraction processing on the original image sequence based on the number of exposure strategies to obtain at least two data streams, so as to accurately detect images / data streams acquired by different exposure strategies for different types of targets in a smart driving scene and provide accurate basis for downstream tasks.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent assisted driving technology, and in particular to an image processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] In intelligent driving systems, image acquisition sensors are the core perception sensors. By collecting environmental data through image acquisition sensors, the state of surrounding targets (static and dynamic targets) can be detected, providing accurate data for downstream tasks such as target behavior prediction, target trajectory prediction, and vehicle driving path planning.

[0003] In related technologies, during vehicle operation, a longer exposure time is usually used to obtain sufficient light, but this can lead to motion blur in the images acquired by the image acquisition sensor, affecting the accuracy of detection and recognition of dynamic targets such as vehicles and pedestrians. On the other hand, using a shorter exposure time to suppress motion blur can easily cause traffic lights (LED lights) to flicker, affecting the accuracy of traffic light status recognition and causing safety hazards for intelligent driving. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides an image processing method, apparatus, storage medium, and electronic device to solve the issues of low robustness and accuracy in vehicle trajectory prediction during intelligent driving.

[0005] A first aspect of this disclosure provides an image processing method, comprising: The image acquisition sensor is controlled to acquire original images with consistent brightness alternately based on at least two exposure strategies in a time-division multiplexing manner to obtain an original image sequence. Based on the number of exposure strategies, the original image sequence is subjected to interval frame-by-frame processing to obtain at least two data streams.

[0006] In some alternative implementations, it also includes: At least two of the data streams are sent to different task processing models, so that the different task processing models can process the corresponding tasks based on the received data streams.

[0007] In some alternative implementations, at least two of the data streams include a long-exposure data stream generated from the original image acquired based on a long-exposure strategy, and a short-exposure data stream generated from the original image acquired based on a short-exposure strategy. The long-exposure data stream is used for traffic light perception processing, and the short-exposure data stream is used for dynamic target perception processing.

[0008] In some alternative implementations, it also includes: The original images from at least two of the data streams are interleaved and merged in the order of acquisition to obtain the target video.

[0009] In some optional implementations, the step of interleaving and merging the original images from at least two data streams in the order of acquisition to obtain the target video includes: Use the original images acquired based on the long exposure strategy from at least two of the data streams as guide images; Based on the guide image, the original image to be adjusted, which is adjacent to the acquisition time of the guide image but located after the acquisition time, is subjected to image quality enhancement processing to obtain the adjusted original image; The guiding image and the adjusted original image are interleaved and merged according to the acquisition order to obtain the target video.

[0010] In some optional implementations, the step of performing image quality enhancement processing on the original image to be adjusted, which is adjacent to the acquisition time of the guide image but located after the acquisition time, based on the guide image, includes: Based on the guiding image and the original image to be adjusted, determine the joint bilateral filter parameters; Based on the joint bilateral filtering algorithm and the joint bilateral filtering parameters, the original image to be adjusted is subjected to image quality enhancement processing to obtain the adjusted original image.

[0011] In some optional implementations, the step of performing image quality enhancement processing on the original image to be adjusted, which is adjacent to the acquisition time of the guide image but located after the acquisition time, based on the guide image, includes: The target image parameters are obtained by performing a linear transformation on the guided image using a guided filtering algorithm. Based on the target image parameters, the original image to be adjusted is subjected to image quality enhancement processing to obtain the adjusted original image.

[0012] In some optional implementations, the step of performing image quality enhancement processing on the original image to be adjusted, which is adjacent to and after the acquisition time of the guide image, based on the guide image to obtain the adjusted original image, includes: Determine the brightness difference of each pixel in the guide image and the original image to be adjusted; Based on the brightness difference and the brightness histogram of the guide image, the average brightness difference of pixels in each brightness range is determined; Based on the average brightness difference of pixels within each brightness range, the brightness adjustment amount of each pixel in the original image to be adjusted is determined; Based on the brightness adjustment of each pixel in the original image to be adjusted, the original image to be adjusted is subjected to image quality enhancement processing to obtain the adjusted original image.

[0013] In some optional implementations, the controlled image acquisition sensor, using a time-division multiplexing method, alternately acquires original images of consistent brightness based on at least two exposure strategies to obtain an original image sequence, including: Based on the alternation period of at least two exposure strategies and the exposure time and image gain when acquiring the previous original image, the brightness balance algorithm is used to determine the exposure time and image gain required to acquire the next original image. The next original image is acquired based on the exposure time and image gain required to acquire the next original image.

[0014] A second aspect of this disclosure provides an image processing apparatus, comprising: The image acquisition module is used to control the image acquisition sensor to acquire original images with consistent brightness alternately based on at least two exposure strategies in a time-division multiplexing manner to obtain an original image sequence. The data stream generation module is used to perform interval frame-stripping processing on the original image sequence based on the number of exposure strategies to obtain at least two data streams.

[0015] In some alternative embodiments, the image processing apparatus may further include: The task processing module is used to send at least two data streams to different task processing models, so that the different task processing models can process the corresponding tasks based on the received data streams.

[0016] In some implementations, at least two data streams include a long-exposure data stream generated from the original image acquired based on a long-exposure strategy, and a short-exposure data stream generated from the original image acquired based on a short-exposure strategy. Long-exposure data streams were used for traffic light perception processing, while short-exposure data streams were used for dynamic target perception processing.

[0017] In some embodiments, the image processing apparatus may further include: The image fusion module is used to interleave and merge the original images from at least two data streams in the order of acquisition to obtain the target video.

[0018] In some implementations, the image fusion module may include: The image determination submodule is used to take the original images acquired based on the long exposure strategy from at least two data streams as guide images; The image quality enhancement submodule is used to perform image quality enhancement processing on the original image to be adjusted, which is adjacent to the acquisition time of the guide image but located after the acquisition time, based on the guide image, to obtain the adjusted original image; The merging submodule is used to interleave and merge the guide image with the adjusted original image according to the acquisition order to obtain the target video.

[0019] In some implementations, the image enhancement submodule is specifically used for: Determine the joint bilateral filter parameters based on the guide image and the original image to be adjusted; Based on the joint bilateral filtering algorithm and the joint bilateral filtering parameters, image quality enhancement processing is performed on the original image to be adjusted to obtain the adjusted original image.

[0020] In some implementations, the image enhancement submodule is specifically used for: The target image parameters are obtained by performing a linear transformation on the guiding image using a guided filtering algorithm. Based on the target image parameters, image quality enhancement processing is performed on the original image to be adjusted to obtain the adjusted original image.

[0021] In some implementations, the image enhancement submodule is specifically used for: Determine the brightness differences of each pixel in the guide image and the original image to be adjusted; Based on the brightness difference and the brightness histogram of the guide image, the average brightness difference of pixels in each brightness range is determined. Based on the average brightness difference of pixels in each brightness range, determine the brightness adjustment amount of each pixel in the original image to be adjusted; Based on the brightness adjustment of each pixel in the original image to be adjusted, image quality enhancement processing is performed on the original image to be adjusted to obtain the adjusted original image.

[0022] In some implementations, the image acquisition module includes: The parameter determination submodule is used to determine the required exposure time and image gain for acquiring the next original image based on the alternation period of at least two exposure strategies and the exposure time and image gain when acquiring the previous original image, using a brightness balance algorithm. The acquisition submodule is used to acquire the next raw image based on the exposure time and image gain required to acquire the next raw image.

[0023] A third aspect of this disclosure provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the image processing method described above.

[0024] A fourth aspect of this disclosure provides an electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image processing method described above.

[0025] A fifth aspect of this disclosure provides a computer program product including computer program instructions, characterized in that, when the computer program instructions are executed by a processor, they implement the above-described image processing method.

[0026] Based on embodiments of this disclosure, when task perception processing based on image data is required in an intelligent driving scenario, the image acquisition sensor can be controlled to acquire original images of consistent brightness alternately using at least two exposure strategies in a time-division multiplexing manner to obtain an original image sequence. Then, based on the number of exposure strategies, the original image sequence is subjected to interval frame extraction processing to obtain at least two data streams. Thus, this disclosure acquires and generates data streams with different exposure parameters using different exposure strategies, and performs different task perception processing based on these data streams with different exposure parameters. This enables accurate state detection of images / data streams acquired using different exposure strategies for different types of targets in intelligent driving scenarios, providing accurate data for downstream tasks and improving the safety of intelligent driving. Attached Figure Description

[0027] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0028] Figure 1 This is an application scenario diagram provided by an exemplary embodiment of this disclosure; Figure 2 This is a schematic flowchart of an image processing method provided in an exemplary embodiment of this disclosure; Figure 3 This is a flowchart illustrating step 101 provided in an exemplary embodiment of this disclosure; Figure 4 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this disclosure; Figure 5 This is a flowchart illustrating step 403 provided in an exemplary embodiment of this disclosure; Figure 6This is a flowchart illustrating step 432 provided in an exemplary embodiment of this disclosure; Figure 7 This is a flowchart illustrating step 432 provided in another exemplary embodiment of this disclosure; Figure 8 This is a flowchart illustrating step 432 provided in another exemplary embodiment of this disclosure; Figure 9 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of the present disclosure; Figure 10 This is a schematic diagram of the structure of an image processing apparatus provided in another exemplary embodiment of the present disclosure; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0029] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0030] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0031] This disclosure outlines In realizing this disclosure, the inventors discovered through research that in the field of intelligent driving (including assisted driving and autonomous driving), it is usually necessary to acquire images of the vehicle's driving environment based on image acquisition sensors configured on the vehicle, and to detect the state of dynamic and static targets in the environment based on the acquired image data. This allows for accurate data collection to be used for the driving control of intelligent driving vehicles based on the detected target states, thereby improving the driving safety of autonomous vehicles. However, in order to clearly present the state of different targets in the environment in the image, different exposure parameters are required for image acquisition. For example, during vehicle movement, a longer exposure time is used to obtain sufficient light, but this can lead to motion blur of moving targets in the image. Therefore, a shorter exposure time is needed to clearly present the state of dynamic targets in the image. Traffic lights generally use LED light sources, which flash at specific frequencies (such as 50Hz, 60Hz, 100Hz, 120Hz). If the exposure time is too short and not synchronized with the LED flashing frequency, the traffic lights in the image will appear uneven in brightness or even completely extinguished. Therefore, a longer exposure time is needed to clearly present the state of traffic lights in the image.

[0032] In related technologies, existing automatic exposure strategies typically tend to set shorter exposure times to prioritize the suppression of motion blur, but this exacerbates LED flickering, affects the accuracy of traffic light status recognition, and poses safety hazards for intelligent driving.

[0033] In this embodiment of the present disclosure, during vehicle operation, the image acquisition sensor can be controlled to acquire original images with consistent brightness in a time-division multiplexing manner based on at least two exposure strategies to obtain an original image sequence. Based on the number of exposure strategies, the original image sequence is subjected to interval frame extraction processing to obtain at least two data streams. This enables the state detection and perception processing of different types of targets to be performed through the at least two data streams, providing accurate information for downstream tasks and improving the safety of intelligent driving.

[0034] Exemplary Applications The embodiments disclosed herein can be used for image acquisition and task perception processing of autonomous mobile devices such as vehicles, robots, and drones in intelligent driving scenarios.

[0035] Figure 1 This is an application scenario diagram provided by an exemplary embodiment of this disclosure, such as... Figure 1 As shown, a computing platform 120 and multiple image acquisition sensors 130 can be deployed on the vehicle 110 as needed. The image acquisition sensors 130 can acquire raw images of the vehicle's driving environment during vehicle operation.

[0036] The image acquisition sensor 130 can be deployed at different positions and / or orientations on the vehicle 110 according to perception needs, in order to perceive the corresponding areas in the external environment of the vehicle 110 (e.g., the left front area, the front area, the right front area, the left rear area, the rear area, the right rear area, the left side area of ​​the vehicle body, the rear side area of ​​the vehicle body, etc.) and obtain image data of the surrounding environment. The image data may include targets such as dynamic targets and traffic lights.

[0037] The computing platform 120 includes a processor and can transmit data with the image acquisition sensor 130 via a serial data bus or a controller area network (CAN) bus.

[0038] During the driving process, the vehicle 110 can control the image acquisition sensor to acquire original images with consistent brightness in a time-division multiplexing manner based on at least two exposure strategies to obtain an original image sequence. Based on the number of exposure strategies, the original image sequence is subjected to interval frame extraction processing to obtain at least two data streams.

[0039] Figure 1This is merely an exemplary application scenario implementation of the present disclosure. Those skilled in the art, based on the description of the present disclosure, will understand that the present disclosure can also adopt any other feasible implementation. For example, the computing platform can be deployed entirely or partially on a cloud server or terminal device (such as a mobile terminal, tablet computer, PC, etc.), and receive sensor data collected by multiple sensors 130 through a communication connection with the vehicle driving control system. The data is then processed using the image processing device provided in the present disclosure, and the at least two data streams obtained are used to accurately detect the state of dynamic targets such as vehicles and pedestrians, as well as the state of traffic lights.

[0040] Exemplary methods Figure 2 This is a schematic flowchart of an exemplary embodiment of the image processing method provided in this disclosure. The embodiments of this disclosure can be applied to... Figure 1 The computing platform shown. Figure 2 As shown, the image processing method of this disclosure includes: Step 201: Control the image acquisition sensor to acquire original images with consistent brightness alternately based on at least two exposure strategies in a time-division multiplexing manner to obtain an original image sequence.

[0041] In this embodiment of the disclosure, the image acquisition time is divided into multiple time slices by setting the acquisition frame rate, such as 60fps acquisition. Different exposure strategies are executed alternately in different time slices to achieve the acquisition of original images based on at least two exposure strategies in a time-division multiplexing manner, sharing the same sensor array or readout channel.

[0042] For example, the image acquisition sensor is controlled to acquire the original image using two exposure strategies: a long exposure strategy and a short exposure strategy. After acquiring the original image using the short exposure strategy in time slice 1, the original image is acquired using the long exposure strategy in the next time slice (time slice 2), and the original image is acquired using the short exposure strategy in the next time slice (time slice 3). In this way, the different exposure strategies are used alternately periodically to acquire the original image.

[0043] Long exposure strategy refers to imaging methods that use a longer exposure time, such as 5-10 milliseconds, to construct high dynamic range (HDR) images. Short exposure strategy refers to imaging methods that use a shorter exposure time, such as 50-200 microseconds, to construct images.

[0044] In this embodiment of the disclosure, other exposure strategies that differ from long exposure strategies and short exposure strategies can also be set, for example, the exposure duration is between long exposure strategies and short exposure strategies.

[0045] In this embodiment of the disclosure, in order to ensure that the brightness of the original images acquired alternately by different exposure strategies is consistent, the sensor gain (ISO) can be adjusted according to the exposure time. For short exposure strategies, the gain can be turned on to increase the brightness, while for long exposure strategies, the gain can be turned off to avoid amplifying the dark current noise generated by the sensor due to heat.

[0046] In practice, an FPGA (Field Programmable Gate Array) or an embedded controller can be used to write the exposure parameters of different exposure strategies into a SequencerSet (a register group inside the sensor used to store the preset exposure parameter sequence) before starting to acquire each frame of raw image, so as to realize the alternating switching of exposure strategies.

[0047] Step 202: Based on the number of exposure strategies, perform interval frame-stripping on the original image sequence to obtain at least two data streams.

[0048] In this embodiment of the disclosure, the exposure strategy corresponding to the original image in the same data stream is the same. If n exposure strategies are used when acquiring the original image, the original image sequence can be divided into n data streams.

[0049] For example, if the number of exposure strategies is 2, after acquiring the original image sequence through the image acquisition sensor at a frame rate of 60fps, two interleaved and independent 30fps data streams can be obtained by intermittent frame extraction.

[0050] Based on embodiments of this disclosure, when task perception processing based on image data is required in an intelligent driving scenario, the image acquisition sensor can be controlled to acquire original images of consistent brightness alternately using at least two exposure strategies in a time-division multiplexing manner to obtain an original image sequence. Then, according to the number of exposure strategies, the original image sequence is subjected to interval frame-by-frame processing to obtain at least two data streams. Thus, this disclosure acquires and generates data streams with different exposure parameters through different exposure strategies, and performs different task perception processing based on these data streams with different exposure parameters. This enables accurate state detection of different types of targets in intelligent driving scenarios, providing accurate data for downstream tasks and improving the safety of intelligent driving.

[0051] In some implementations, at least two of the data streams include a long-exposure data stream generated from raw images acquired based on a long-exposure strategy, and a short-exposure data stream generated from raw images acquired based on a short-exposure strategy; the long-exposure data stream is used for traffic light perception processing, and the short-exposure data stream is used for dynamic target perception processing.

[0052] In this embodiment, using a long-exposure data stream with a longer exposure time and lower noise for traffic light (traffic signal) perception can effectively avoid LED flickering and improve the accuracy and effectiveness of traffic light perception. On the other hand, using a short-exposure data stream with a shorter exposure time for general target perception can ensure higher accuracy in detecting dynamic targets because the short-exposure image is clear and has no ghosting.

[0053] In this embodiment of the disclosure, at least two data streams are described as a long exposure data stream and a short exposure data stream, but it is not limited to only two data streams.

[0054] Figure 3 This is a flowchart illustrating step 201 provided in an exemplary embodiment of this disclosure. Figure 3 As shown above, in the above Figure 2 Based on the illustrated embodiment, the process of acquiring the original image sequence through step 201 may include the following steps: Step 211: Based on the alternation period of at least two exposure strategies and the exposure time and image gain when acquiring the previous original image, use the brightness balance algorithm to determine the exposure time and image gain required to acquire the next original image.

[0055] The alternation period of the exposure strategies refers to the time interval required for all exposure strategies to complete one full switch. For example, if there are only two strategies, long exposure and short exposure, and the image acquisition sensor has a frame rate of 60fps (one frame is acquired every 16.67ms), the alternation period is 16.67*2=33.34ms. If there are three exposure strategies and the image acquisition sensor has a frame rate of 60fps, the alternation period is 16.67*3=50.01ms.

[0056] In this embodiment of the disclosure, since the image is acquired alternately according to the exposure strategy, one frame of image will be acquired according to each exposure strategy in each alternation period. For example, if there are two exposure strategies, one long exposure image and one short exposure image will be acquired in one alternation period.

[0057] In some implementations, to ensure that images acquired using different exposure strategies have consistent brightness, the sensor gain can be adjusted according to the exposure time. The image brightness L is proportional to the product of the exposure time S and the gain G.

[0058] In practice, a brightness balancing algorithm can be used to control the brightness of images acquired using different exposure strategies to be the same, such as the target brightness.

[0059] The brightness balance algorithm can be found in equation (1).

[0060] S_s* G_s= S_f* G_f=k*L_target Formula (1) In equation (1), S_s is the exposure time under the short exposure strategy, G_s is the gain under the short exposure strategy, S_f is the exposure time under the long exposure strategy, G_f is the gain under the long exposure strategy, k is the camera response parameter, and L_target is the target brightness.

[0061] Combining equation (1), if the previous original image was acquired under the long exposure strategy, the exposure duration and gain of the short exposure strategy can be determined based on S_s, G_s, and the relationship between the exposure durations of the short exposure strategy and the long exposure strategy.

[0062] The exposure durations of short-exposure and long-exposure strategies can be related by a multiple; for example, the exposure duration of the long-exposure strategy is equal to the exposure duration of the short-exposure strategy. p If the exposure time of the long exposure strategy is multiplied by the multiple, the exposure time of the short exposure strategy can be determined based on the exposure time of the long exposure strategy and the multiple relationship. Then, based on the brightness balance algorithm of the above formula (1), the exposure time of the short exposure strategy, and the exposure time and gain of the long exposure strategy, the gain of the short exposure strategy can be determined.

[0063] In other implementations, when determining the exposure duration and gain of the exposure strategy for the next original image, the exposure adjustment amount can be determined based on the statistical brightness information Y and the target brightness L_target of the previous original image. Then, the exposure duration and gain of the previous original image acquired under the exposure strategy can be adjusted based on the exposure adjustment amount to determine the exposure duration and image gain required for acquiring the next original image.

[0064] Step 212: Based on the exposure time and image gain required to acquire the next raw image, acquire the next raw image.

[0065] After determining the exposure time and image gain required to acquire the next raw image through step 211, the exposure time and image gain can be written into the corresponding SequencerSet register. Then, the image acquisition sensor can access the register to acquire the next raw image based on the exposure time and image gain required for the next raw image.

[0066] Based on the embodiments of this disclosure, it is possible to maintain consistent brightness in original images acquired alternately using different exposure strategies, ensuring that two data streams with different characteristics can be generated without increasing hardware costs, and that different perception tasks can be performed on the data streams with different characteristics, which helps to meet the perception requirements of highly reliable intelligent driving.

[0067] Figure 4This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this disclosure. For example... Figure 4 As shown, the process includes steps 401 to 404. Each step is explained below.

[0068] Step 401: Control the image acquisition sensor to acquire original images with consistent brightness alternately based on at least two exposure strategies in a time-division multiplexing manner to obtain an original image sequence.

[0069] Step 402: Based on the number of exposure strategies, perform interval frame-stripping on the original image sequence to obtain at least two data streams.

[0070] For details on the implementation of steps 401 and 402, please refer to [link to relevant documentation]. Figure 2 The illustrated embodiment will not be described further here.

[0071] Furthermore, after generating at least two data streams, the images in the at least two data streams can be interleaved and merged according to the acquisition order in step 403 to obtain the target video; or the at least two data streams can be sent to different task processing models in step 404.

[0072] Step 403: Interleave and merge the original images from at least two data streams in the order of acquisition to obtain the target video.

[0073] The target video can be used as a video stream in recording mode (such as a video stream captured by a dashcam). Because the target video is fused from a data stream that has had motion blur resolved and a data stream that has had its traffic light flickering issue resolved, the visual effect of the target video is better.

[0074] In this embodiment, by fusing long exposure data streams and short exposure data streams, it helps to improve image quality, enhance detail, and optimize visual perception in dynamic scenes, making the video closer to what the human eye sees in high-contrast scenes.

[0075] In this embodiment of the disclosure, due to vehicle movement, there is a slight parallax between adjacent long-exposure strategy images and short-exposure strategy images. Image feature points can be extracted using feature point extraction algorithms, such as Scale-invariant feature transform (SIFT) or ORB (Oriented FAST and Rotated BRIEF) algorithms. Then, the image coordinates of adjacent long-exposure images and short-exposure images are aligned by homography matrix calculation.

[0076] As an example, let the homography matrix be a 3*3 matrix. Align the long exposure image I-long to the coordinate system of the adjacent short exposure image I-short. Then the aligned long exposure image I-long-warped is as shown in equation (2).

[0077] ‌I-long-warped=warpPerspective(I-long,H) Formula (2) In equation (2), I-long-warped is the long exposure image after alignment, I-long is the long exposure image before alignment, and H is the homography matrix.

[0078] In other examples, the short exposure image can also be aligned to the coordinate system of the long exposure image.

[0079] After aligning the coordinate systems of the long-exposure and short-exposure images, the brightness difference between them can be calculated. Then, the original images from at least two data streams are interleaved and merged according to the acquisition order. For details, please refer to [link to documentation]. Figure 5 The embodiments shown are not described in detail here.

[0080] Step 404: Send at least two data streams to different task processing models so that the different task processing models can process the corresponding tasks based on the received data streams.

[0081] In this embodiment of the disclosure, a low-noise data stream with a longer exposure time in the exposure strategy can be input into a task processing model for traffic light (traffic signal) perception; and a non-ambiguous data stream with a shorter exposure time in the exposure strategy can be input into a task processing model for general target detection, such as the YOLO model, the SSD (Single Shot MultiBox Detector) model, etc., for detecting vehicles, pedestrians, lane lines, etc.

[0082] In some alternative implementations, if at least two data streams include, in addition to long exposure data streams and short exposure data streams, a data stream with an exposure time between the long exposure data stream and the short exposure data stream, then the data stream can be used for sensing and processing complex dynamic environments, such as urban intersections where stationary signs, slow-moving vehicles, pedestrians, and strong light changes coexist.

[0083] In this embodiment of the disclosure, images can be acquired by cross-collecting two or more exposure strategies according to the actual scene perception requirements of intelligent driving, and used for different perception tasks.

[0084] Based on the embodiments of this disclosure, the process of using data streams corresponding to different exposure strategies for traffic light perception and general target detection perception is realized. Using a data stream with a longer exposure time and low noise for traffic light perception can effectively avoid LED flickering problems, while using a short exposure data stream for general target perception can ensure higher target detection accuracy because the short exposure image is clear and has no ghosting.

[0085] Figure 5 This is a flowchart illustrating step 403 provided in an exemplary embodiment of this disclosure. Figure 5 As shown above, in the above Figure 4 Based on the illustrated embodiment, the process of interleaving and merging the original images from at least two data streams according to the acquisition order includes steps 431 to 433. Each step is explained below.

[0086] Step 431: Use the original images acquired from at least two data streams based on the long exposure strategy as guide images.

[0087] Since the noise level and tone of the original image acquired using the long exposure strategy are usually better than those of the short exposure image, this embodiment can use the long exposure image in the long exposure data stream as a guide image to perform image quality enhancement processing on the short exposure image when fusing the short exposure data stream and the long exposure data stream.

[0088] Step 432: Based on the guide image, perform image quality enhancement processing on the original image to be adjusted that is adjacent to the acquisition time of the guide image but located after the acquisition time, to obtain the adjusted original image.

[0089] Specifically, guided filtering can be used to perform linear transformations on short-exposure images in a short-exposure data stream to achieve image quality enhancement, such as... Figure 7 The illustrated embodiment; image quality enhancement processing of short-exposure images in a short-exposure data stream can also be performed using a combined bilateral filtering method, such as... Figure 6 The illustrated embodiment; image quality enhancement processing can also be performed on short-exposure images in the short-exposure data stream using brightness mapping, such as... Figure 8 As shown, details will not be elaborated here.

[0090] In this embodiment of the disclosure, the image quality consistency between short-exposure images and long-exposure images is enhanced through image quality enhancement processing.

[0091] Step 433: The guide image and the adjusted original image are interleaved and merged according to the acquisition order to obtain the target video.

[0092] Based on the embodiments of this disclosure, a method is disclosed in which, when fusing short-exposure images and long-exposure images, the image quality of the short-exposure image is first enhanced, and then the images are interleaved and merged according to the acquisition order to obtain the target video. This method helps to enhance the brightness consistency of the fused target video, ensures that the target video has a unified brightness style in the global scope, and the effect is more stable.

[0093] Figure 6 This is a flowchart illustrating step 432 provided in an exemplary embodiment of this disclosure. Figure 6 As shown above, in the above Figure 5 Based on the illustrated embodiment, the process of enhancing the image quality of the original image to be adjusted includes steps 4321 to 4322. Each step is explained below.

[0094] Step 4321: Determine the joint bilateral filter parameters based on the guide image and the original image to be adjusted.

[0095] Joint Bilateral Filter is a nonlinear filtering method for image processing that uses a guide image to guide the filtering process, achieving a smoothing effect while preserving edge information.

[0096] In this embodiment of the disclosure, the joint bilateral filtering parameters include a spatial Gaussian kernel, a range Gaussian kernel, a joint Gaussian kernel, and a neighborhood diameter. The spatial Gaussian kernel measures the spatial distance between pixels in the original image to be adjusted, the range Gaussian kernel measures the grayscale difference between pixels in the guide image, the joint Gaussian kernel measures the structural difference between pixels in the guide image, and the neighborhood diameter defines the size of the local window considered when processing each pixel.

[0097] Step 4322: Based on the joint bilateral filtering algorithm and the joint bilateral filtering parameters, perform image quality enhancement processing on the original image to be adjusted to obtain the adjusted original image.

[0098] In this embodiment of the disclosure, after determining the above-mentioned joint bilateral filtering parameters based on the pixel information in the guiding image and the image to be adjusted, the joint bilateral filtering algorithm can be used to filter each pixel in the original image to be adjusted and calculate the weighted average value of the pixels in its neighborhood. The weights are jointly determined by the spatial Gaussian kernel, the range Gaussian kernel and the joint Gaussian kernel.

[0099] Based on the embodiments of this disclosure, a method for image quality enhancement processing of the original image to be adjusted, such as a short-exposure image, is disclosed by means of joint bilateral filtering. The joint Gaussian kernel in the joint bilateral filtering can better preserve the edge and texture information in the original image while removing noise, which helps to further improve the picture effect of the target video obtained by fusion.

[0100] Figure 7 This is a flowchart illustrating step 432 provided in another exemplary embodiment of this disclosure. Figure 7 As shown above, in the above Figure 5 Based on the illustrated embodiment, the process of enhancing the image quality of the original image to be adjusted using the guided filtering algorithm includes steps 4323 to 4324. Each step is explained below.

[0101] Step 4323: Using the guided filtering algorithm, perform a linear transformation on the guided image to obtain the target image parameters.

[0102] In this embodiment of the disclosure, the guided filtering algorithm guides the filtering process of the original image to be adjusted by using a guide image, so that the adjusted original image achieves smoothing or noise reduction while preserving the edge structure.

[0103] In practice, within a local window, the pixels of the adjusted original image and the pixels of the guide image are linearly related. See equation (3) for the linear relationship.

[0104] Q_i=a_k*I_long_i+b_k Formula (3) In equation (3), a_k and b_k are linear coefficients, I_long_i is the pixel parameter of the guide image, and Q_i is the pixel parameter of the adjusted original image.

[0105] Among them, a_k and b_k can be obtained by minimizing the cost function of the following equation (4).

[0106] Equation (4) In equation (4), a_k and b_k are linear coefficients, and I_long_i is the pixel parameter of the guiding image. It is a regularization parameter to prevent Too large.

[0107] Step 4324: Based on the target image parameters, perform image quality enhancement processing on the original image to be adjusted to obtain the adjusted original image.

[0108] In this embodiment of the disclosure, based on the target image parameter Q_i determined above, image quality enhancement processing can be performed on the original image to be adjusted to obtain the adjusted original image.

[0109] Based on the embodiments of this disclosure, a method for image quality enhancement processing of the original image to be adjusted, such as a short-exposure image, is disclosed through guided filtering. Guided filtering can better preserve the edge and texture information in the original image, and also helps to suppress noise and enhance details, further improving the image quality of the obtained target video.

[0110] Figure 8 This is a flowchart illustrating step 432 provided in another exemplary embodiment of this disclosure. Figure 8 As shown above, in the above Figure 5 Based on the illustrated embodiment, the process of enhancing the image quality of the original image to be adjusted by adjusting the brightness includes steps 4325 to 4328. Each step is explained below.

[0111] Step 4325: Determine the brightness difference of each pixel in the guide image and the original image to be adjusted.

[0112] In this embodiment of the disclosure, the brightness difference of each pixel in the guide image and the original image to be adjusted can be determined pixel by pixel.

[0113] In some implementations, the brightness difference of each pixel in the guide image and the original image to be adjusted can be determined by equation (5).

[0114] Diff=I_short-I_long_warped Formula (5) In equation (5), I_short is the brightness value of the pixel in the original image to be adjusted, and I_long_warped is the brightness value of the pixel in the aligned long exposure image (guide image).

[0115] In other implementations, the brightness difference of each pixel in the guide image and the original image to be adjusted can also be determined by equation (6).

[0116] Equation (6) In equation (6), I_short is the brightness value of a pixel in the original image to be adjusted, and I_long_warped is the brightness value of a pixel in the aligned long-exposure image (guide image). It is the regularization parameter.

[0117] Step 4326: Based on the brightness difference and the brightness histogram of the guide image, determine the average brightness difference of pixels in each brightness range.

[0118] In this embodiment of the disclosure, a brightness histogram of the guide image can be obtained based on the brightness of the guide image. For each brightness interval in the brightness histogram and the brightness difference corresponding to the pixel points in each brightness interval, all brightness differences Diff in each brightness interval can be obtained, and then the average brightness difference in each brightness interval can be obtained.

[0119] Step 4327: Based on the average brightness difference of pixels in each brightness range, determine the brightness adjustment amount of each pixel in the original image to be adjusted.

[0120] In this embodiment of the disclosure, the average brightness difference within each brightness range can be used as the brightness adjustment amount of the pixels within that brightness range.

[0121] Furthermore, the brightness adjustment amount of each pixel can be fitted using curve fitting to obtain a brightness adjustment curve, which is used to indicate the adjustment amount made to each pixel.

[0122] In other implementations, a mapping table (LUT) can be established to record the mapping relationship between each pixel and the brightness adjustment amount.

[0123] Step 4328: Based on the brightness adjustment amount of each pixel in the original image to be adjusted, perform image quality enhancement processing on the original image to be adjusted to obtain the adjusted original image.

[0124] Based on the embodiments of this disclosure, a method is disclosed for adjusting the brightness of the original image to be adjusted, such as a short-exposure image, by means of brightness mapping, which helps to obtain a target video with extremely high brightness consistency and stable brightness effect.

[0125] Exemplary device Figure 9 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of this disclosure. Figure 9 As shown, the image processing apparatus includes: Image acquisition module 91 is used to control the image acquisition sensor to acquire original images with consistent brightness in a time-division multiplexing manner based on at least two exposure strategies to obtain an original image sequence. The data stream generation module 92 is used to perform interval frame-stripping processing on the original image sequence based on the number of exposure strategies to obtain at least two data streams.

[0126] In this embodiment, the image acquisition module 91 can be an image acquisition sensor, such as a camera or other device capable of acquiring image signals.

[0127] The data stream generation module 92 contains software or algorithms capable of performing interval frame extraction processing on the raw image sequence acquired by the image acquisition sensor.

[0128] Figure 10 This is a schematic diagram of the structure of an image processing apparatus provided in another exemplary embodiment of this disclosure. For example... Figure 10 As shown, in Figure 9 Based on the illustrated embodiments, in some implementations, the image processing apparatus may further include: The task processing module 93 is used to send at least two data streams to different task processing models, so that the different task processing models can process the corresponding tasks based on the received data streams.

[0129] In some implementations, at least two data streams include a long-exposure data stream generated from the original image acquired based on a long-exposure strategy, and a short-exposure data stream generated from the original image acquired based on a short-exposure strategy. Long-exposure data streams were used for traffic light perception processing, while short-exposure data streams were used for dynamic target perception processing.

[0130] In some embodiments, the image processing apparatus may further include: The image fusion module 94 is used to interleave and merge the original images from at least two data streams in the order of acquisition to obtain the target video.

[0131] In some implementations, the image fusion module 94 may include: Image determination submodule 941 is used to take raw images acquired based on a long exposure strategy from at least two data streams as guide images; The image quality enhancement submodule 942 is used to perform image quality enhancement processing on the original image to be adjusted, which is adjacent to the acquisition time of the guide image and located after the acquisition time, based on the guide image, to obtain the adjusted original image; The merging submodule 943 is used to interleave and merge the guide image and the adjusted original image according to the acquisition order to obtain the target video.

[0132] In some implementations, the image enhancement submodule 942 is specifically used for: Determine the joint bilateral filter parameters based on the guide image and the original image to be adjusted; Based on the joint bilateral filtering algorithm and the joint bilateral filtering parameters, image quality enhancement processing is performed on the original image to be adjusted to obtain the adjusted original image.

[0133] In some implementations, the image enhancement submodule 942 is specifically used for: The target image parameters are obtained by performing a linear transformation on the guiding image using a guided filtering algorithm. Based on the target image parameters, image quality enhancement processing is performed on the original image to be adjusted to obtain the adjusted original image.

[0134] In some implementations, the image enhancement submodule 942 is specifically used for: Determine the brightness differences of each pixel in the guide image and the original image to be adjusted; Based on the brightness difference and the brightness histogram of the guide image, the average brightness difference of pixels in each brightness range is determined. Based on the average brightness difference of pixels in each brightness range, determine the brightness adjustment amount of each pixel in the original image to be adjusted; Based on the brightness adjustment of each pixel in the original image to be adjusted, image quality enhancement processing is performed on the original image to be adjusted to obtain the adjusted original image.

[0135] In some implementations, the image acquisition module 91 includes: The parameter determination submodule 911 is used to determine the exposure time and image gain required to acquire the next original image based on the alternation period of at least two exposure strategies and the exposure time and image gain when acquiring the previous original image, using a brightness balance algorithm. The acquisition submodule 912 is used to acquire the next raw image based on the exposure time and image gain required to acquire the next raw image.

[0136] It should be noted that the specific implementation of the image processing apparatus in this disclosure is similar to the specific implementation of the image processing method in this disclosure. For details, please refer to the image processing method section. To reduce redundancy, further details will not be provided.

[0137] Exemplary electronic devices Figure 11 This is a structural diagram of an electronic device provided in an exemplary embodiment of the present disclosure, including at least one processor 1101 and a memory 1002.

[0138] The processor 1101 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 11 to perform desired functions.

[0139] The memory 1102 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1101 may execute one or more computer program instructions to implement the image processing methods and / or other desired functions of the various embodiments of this disclosure described above.

[0140] In one example, the electronic device may also include an input device 1103 and an output device 1104, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0141] The input device 1103 may also include, for example, a keyboard, mouse, touch screen, sound pickup device (such as a microphone array), etc.

[0142] The output device 1104 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0143] Of course, for the sake of simplicity, Figure 11 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0144] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0145] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0146] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0147] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0148] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0149] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0150] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0151] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above, unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0152] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0153] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0154] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. An image processing method, comprising: The image acquisition sensor is controlled to acquire original images with consistent brightness alternately based on at least two exposure strategies in a time-division multiplexing manner to obtain an original image sequence. Based on the number of exposure strategies, the original image sequence is subjected to interval frame-by-frame processing to obtain at least two data streams.

2. The method according to claim 1, further comprising: At least two of the data streams are sent to different task processing models, so that the different task processing models can process the corresponding tasks based on the received data streams.

3. The method according to claim 2, wherein, At least two of the data streams include a long-exposure data stream generated from the original image acquired based on a long-exposure strategy, and a short-exposure data stream generated from the original image acquired based on a short-exposure strategy; The long-exposure data stream is used for traffic light perception processing, and the short-exposure data stream is used for dynamic target perception processing.

4. The method according to any one of claims 1-3, further comprising: The original images from at least two of the data streams are interleaved and merged in the order of acquisition to obtain the target video.

5. The method according to claim 4, wherein, The step of interleaving and merging the original images from at least two data streams according to the acquisition order to obtain the target video includes: Use the original images acquired based on the long exposure strategy from at least two of the data streams as guide images; Based on the guide image, the original image to be adjusted, which is adjacent to the acquisition time of the guide image but located after the acquisition time, is subjected to image quality enhancement processing to obtain the adjusted original image; The guiding image and the adjusted original image are interleaved and merged according to the acquisition order to obtain the target video.

6. The method according to claim 5, wherein, The step of performing image quality enhancement processing on the original image to be adjusted, which is adjacent to the acquisition time of the guide image but located after the acquisition time, based on the guide image, includes: Based on the guiding image and the original image to be adjusted, determine the joint bilateral filter parameters; Based on the joint bilateral filtering algorithm and the joint bilateral filtering parameters, the original image to be adjusted is subjected to image quality enhancement processing to obtain the adjusted original image.

7. The method according to claim 5, wherein, The step of performing image quality enhancement processing on the original image to be adjusted, which is adjacent to the acquisition time of the guide image but located after the acquisition time, based on the guide image, includes: The target image parameters are obtained by performing a linear transformation on the guided image using a guided filtering algorithm. Based on the target image parameters, the original image to be adjusted is subjected to image quality enhancement processing to obtain the adjusted original image.

8. The method according to claim 5, wherein, The step of performing image quality enhancement processing on the original image to be adjusted, which is adjacent to the acquisition time of the guide image but located after the acquisition time, based on the guide image, to obtain the adjusted original image includes: Determine the brightness difference of each pixel in the guide image and the original image to be adjusted; Based on the brightness difference and the brightness histogram of the guide image, the average brightness difference of pixels in each brightness range is determined; Based on the average brightness difference of pixels within each brightness range, the brightness adjustment amount of each pixel in the original image to be adjusted is determined; Based on the brightness adjustment of each pixel in the original image to be adjusted, the original image to be adjusted is subjected to image quality enhancement processing to obtain the adjusted original image.

9. The method according to any one of claims 1, wherein, The controlled image acquisition sensor, using a time-division multiplexing method, alternately acquires original images of consistent brightness based on at least two exposure strategies to obtain an original image sequence, including: Based on the alternation period of at least two exposure strategies and the exposure time and image gain when acquiring the previous original image, the brightness balance algorithm is used to determine the exposure time and image gain required to acquire the next original image. The next original image is acquired based on the exposure time and image gain required to acquire the next original image.

10. An image processing apparatus, comprising: The image acquisition module is used to control the image acquisition sensor to acquire original images with consistent brightness alternately based on at least two exposure strategies in a time-division multiplexing manner to obtain an original image sequence. The data stream generation module is used to perform interval frame-stripping processing on the original image sequence based on the number of exposure strategies to obtain at least two data streams.

11. A computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the method described in any one of claims 1-9.

12. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-9.

13. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-9.

Images