Image processing method and apparatus, vehicle, storage medium, and program product

Abstract

The present application relates to the technical field of image processing, and discloses an image processing method and device, a vehicle, a storage medium and a program product, wherein the present application determines the scene label of a target driving scene by acquiring multi-modal environment perception data. Then, the joint parameter mapping table is queried based on the scene label to match the target joint parameter corresponding to the target driving scene, the target joint parameter including a pair of first parameter group and second parameter group, and the synergistic constraint relationship existing between the first parameter group and the second parameter group can effectively inhibit the gain conflict caused by the opposite image gain trend of the two groups of parameters. The first parameter group and the second parameter group are used to control the first image processor and the second image processor to perform image processing and optimization in turn to obtain a target image, and the algorithm synergistic processing of the image acquisition end and the image display end is established, so that the picture quality degradation caused by algorithm conflict is avoided while adapting to various driving scenes, and the display effect of the vehicle-mounted image is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to image processing methods, apparatus, vehicles, storage media, and program products. Background Technology

[0002] With the development of smart cockpit technology, the integration of in-vehicle system-on-chips (SOCs) is constantly improving. Traditional independent image signal processors (ISPs) are gradually being integrated into the SOC, and the algorithm capabilities of picture quality processors (PQs) are also continuously being enhanced to meet the picture quality and scenario-based requirements of in-vehicle displays.

[0003] In existing automotive image processing architectures, the ISP module operates at the image acquisition end, converting sensor output data into visual images, while the image quality optimization processor (PQ) operates at the image display end, responsible for optimizing the image quality to be displayed. As the two ends of the image data stream, the ISP and PQ modules require image data to be transferred through memory, resulting in a serious problem of algorithm silos, and their parameters cannot coordinate and work together.

[0004] For example, in low-light scenarios such as tunnels and night driving, insufficient ambient light can increase sensor gain and significantly increase noise. The ISP module needs to perform noise reduction to suppress noise, resulting in blurred image edges. Meanwhile, the PQ module performs sharpening enhancement to improve details, which can amplify residual noise and create a ringing effect. The noise reduction of the ISP module and the sharpening of the PQ module have direct algorithmic conflicts, affecting the final display effect of the image. Summary of the Invention

[0005] This invention provides an image processing method, apparatus, vehicle, storage medium, and program product to solve the problem of image processing algorithm conflicts between the image acquisition end and the image display end in the prior art, which affect the image display effect.

[0006] In a first aspect, the present invention provides an image processing method applied to an in-vehicle display system, the in-vehicle display system including a first image processor deployed at an image acquisition end and a second image processor deployed at an image display end; the method includes: Acquire multimodal environmental perception data and raw image signals output from the image acquisition terminal, and determine the scene label of the target driving scene based on the multimodal environmental perception data; Based on the pre-built joint parameter mapping table, the target joint parameters corresponding to the scene label are obtained by querying. The target joint parameters include a pair of first parameter groups and second parameter groups. There is a cooperative constraint relationship between the first parameter groups and the second parameter groups. The cooperative constraint relationship is used to suppress the gain conflict between the first parameter groups and the second parameter groups. The gain conflict is used to characterize the opposite image gain trends of the first parameter groups and the second parameter groups to the image. The first image processor is controlled to process the original image signal using the first parameter group to obtain the image to be displayed, and then forwards the image to be displayed to the second image processor; The second image processor is controlled to optimize the image to be displayed using the second parameter group to obtain the target image, and the image display terminal is controlled to display the target image.

[0007] This invention determines the scene label of the target driving scenario by acquiring multimodal environmental perception data. Then, based on the scene label, a joint parameter mapping table is queried to match the target joint parameters corresponding to the target driving scenario. These target joint parameters include a pair of first and second parameter groups. The cooperative constraint relationship between the first and second parameter groups effectively suppresses gain conflicts caused by opposite image gain trends between the two groups. Using the first and second parameter groups, the first and second image processors are controlled to sequentially perform image processing and optimization to obtain the target image. This establishes a collaborative processing algorithm between the image acquisition end and the image display end, adaptively matching various driving scenarios while avoiding image quality degradation caused by algorithm conflicts, significantly improving the display effect of in-vehicle images.

[0008] In some optional implementations, the image to be displayed is optimized using a second set of parameters to obtain a target image, including: Identify safe pixels and non-safe pixels in the image to be displayed; where safe pixels are pixels related to driving safety information, and non-safe pixels are pixels in the image other than safe pixels. The safe pixels are enhanced, and the non-safe pixels are optimized using the second parameter group to obtain the target image.

[0009] This invention identifies safe pixels related to driving safety information and the remaining unsafe pixels in an image. It then performs targeted enhancement processing on the safe pixels to highlight key driving safety information, while simultaneously using a second parameter group to uniformly optimize the image quality of the unsafe pixels. This achieves differentiated processing of image regions, effectively enhancing the visual recognition of driving safety information and improving driving safety.

[0010] In some alternative implementations, the security pixel is enhanced, including: Increase the brightness of safety pixels and enhance the color of safety colors within the safety pixels; whereby safety colors are used to characterize the color features that convey driving safety information. After optimizing the image quality of unsafe pixels using the second parameter set, the method also includes reducing the saturation of unsafe pixels.

[0011] This invention significantly enhances the visual recognition of driving safety information by increasing the brightness of safety pixels and enhancing the safety colors within them. Simultaneously, by optimizing the image quality of non-safety pixels and reducing their saturation, it effectively weakens color interference in the background area, preventing irrelevant areas from excessively attracting the driver's attention. While ensuring overall image quality optimization, it highlights driving safety information and safeguards driving safety.

[0012] In some alternative implementations, the process of constructing the joint parameter mapping table includes: A first parameter group corresponding to each scene label is determined, and the original parameter group of the second image processor is obtained; wherein, the first parameter group includes at least some of the following items: noise reduction intensity, exposure gain and high dynamic range, and the original parameter group includes at least some of the following items: sharpening intensity, first saturation of the image center region and second saturation of the image edge region; Based on the image gain corresponding to the first parameter group and the original parameter group, the original parameter group is adjusted to obtain the second parameter group; Construct a joint parameter mapping table based on the first and second parameter groups corresponding to each scene label.

[0013] This invention determines a first parameter set corresponding to each scene label and an original parameter set of a second image processor. Based on the corresponding image gains, the original parameter set is adaptively adjusted to obtain a second parameter set, thereby constructing a joint parameter mapping table. This establishes a cooperative constraint relationship between the processing parameters of the first and second image processors, preventing gain conflicts due to opposing image gain trends at the parameter construction level. It effectively prevents the second image processor's sharpening and saturation enhancement from amplifying noise and weakening the noise reduction effect of the first image processor, achieving coordination between noise suppression and image quality optimization, and improving scene adaptability.

[0014] In some optional implementations, based on the image gain corresponding to the first parameter set and the original parameter set, the original parameter set is adjusted to obtain the second parameter set, including: Determine the first image gain of the first parameter group and the second image gain of the original parameter group, and determine whether there is a gain conflict between the first image gain and the second image gain; If there is a gain conflict between the first image gain and the second image gain, the original parameter set is adjusted in the direction of suppressing the gain conflict to obtain the second parameter set.

[0015] This invention accurately determines whether there is a gain conflict between the first image gain corresponding to the first parameter group and the second image gain corresponding to the original parameter group, and adaptively adjusts the original parameter group in the direction of suppressing the gain conflict when a conflict exists, so as to obtain a cooperatively matched second parameter group. This effectively avoids the image gain of the second image processor from being mutually exclusive with that of the first image processor, and prevents the second parameter group from sharpening and enhancing noise by increasing saturation, thus avoiding the cancellation of the noise reduction effect of the first parameter group. While ensuring the image noise reduction effect, it also improves the rationality of image quality optimization, significantly suppresses image quality degradation phenomena such as noise amplification and edge distortion, and improves the final display effect of the image.

[0016] In some optional implementations, the multimodal environment perception data includes navigation information and map information, and the target driving scenario includes the driving area located in front of the vehicle; after determining the scene label of the target driving scenario based on the multimodal environment perception data, the method further includes: Based on navigation and map information, determine the vehicle's current location and the target area of ​​the target driving scenario; When it is detected that the vehicle's current position is outside the target area and the distance between the vehicle's current position and the target area is less than a preset distance, the step of querying the target joint parameters corresponding to the scene label based on the pre-built joint parameter mapping table is executed. When the vehicle's current position is detected to be within the target area, the first image processor is executed to process the original image signal using the first parameter group to obtain the image to be displayed.

[0017] This invention determines the vehicle's current location and the target area of ​​the target driving scenario using navigation and map information. It determines the timing of parameter query and image processing based on the positional relationship between the vehicle and the target driving scenario. The target joint parameters are queried in advance when the vehicle approaches the target driving scenario to avoid parameter switching delays. Image processing is performed immediately after the vehicle enters the target driving scenario, improving the real-time performance and scenario adaptability of in-vehicle image processing, and effectively enhancing driving safety and user experience.

[0018] In some optional implementations, the multimodal environment perception data also includes driver eye images; the method further includes, before displaying the target image at the image display terminal: Based on the driver's eye images, determine the driver's pupil diameter and the driver's gaze direction; If the driver's pupil diameter is detected to be larger than the preset pupil diameter, the driver's gaze area in the target image is determined based on the driver's gaze direction, and the saturation of the gaze area is increased.

[0019] This invention acquires the pupil diameter and gaze direction in real time through driver's eye images. When the pupil diameter is larger than a preset pupil diameter, it locates the driver's gaze area and directionally increases the saturation, adaptively enhancing the visual recognition of the driver's gaze area and improving the perception efficiency of key visual information.

[0020] In some optional implementations, the multimodal environment perception data further includes accelerometer vibration frequencies; before displaying the target image at the controlled image display end, the method further includes: If the accelerometer vibration frequency is detected to be greater than the preset vibration frequency, jitter compensation is performed on the target image.

[0021] This invention compensates for jitter in the target image when the accelerometer vibration frequency is high, thereby offsetting the image jitter, offset and blur caused by severe vehicle vibration, ensuring the stability and clarity of the displayed image, avoiding the impact of image jitter on the driver's recognition of key safety information, and improving driving safety.

[0022] In a second aspect, the present invention provides an image processing apparatus for use in an in-vehicle display system, the in-vehicle display system comprising a first image processor deployed at an image acquisition end and a second image processor deployed at an image display end; the apparatus includes: The first processing module is used to acquire multimodal environmental perception data and raw image signals output by the image acquisition terminal, and determine the scene label of the target driving scene based on the multimodal environmental perception data. The second processing module is used to query the target joint parameters corresponding to the scene label based on the pre-built joint parameter mapping table. The target joint parameters include a pair of first parameter groups and second parameter groups. There is a cooperative constraint relationship between the first parameter groups and the second parameter groups. The cooperative constraint relationship is used to suppress the gain conflict between the first parameter groups and the second parameter groups. The gain conflict is used to characterize the opposite image gain trends of the first parameter groups and the second parameter groups to the image. The third processing module is used to control the first image processor to process the original image signal using the first parameter group to obtain the image to be displayed, and to forward the image to be displayed to the second image processor; The fourth processing module is used to control the second image processor to optimize the image to be displayed using the second parameter group to obtain the target image, and to control the image display terminal to display the target image.

[0023] Thirdly, the present invention provides a vehicle, the vehicle including an in-vehicle display system, the in-vehicle display system including a controller, a first image processor deployed at an image acquisition end and a second image processor deployed at an image display end; the controller includes a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the image processing method of the first aspect or any corresponding embodiment described above.

[0024] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the image processing method described in the first aspect or any corresponding embodiment thereof.

[0025] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the image processing method described in the first aspect or any corresponding embodiment thereof.

[0026] The beneficial effects of this invention are as follows: This invention determines the scene label of the target driving scenario by acquiring multimodal environmental perception data. Then, based on the scene label, a joint parameter mapping table is queried to match the target joint parameters corresponding to the target driving scenario. These target joint parameters include a pair of first and second parameter groups. The cooperative constraint relationship between the first and second parameter groups effectively suppresses gain conflicts caused by opposite image gain trends between the two groups. Using the first and second parameter groups, the first and second image processors are controlled to sequentially perform image processing and optimization to obtain the target image. This establishes a collaborative processing algorithm between the image acquisition end and the image display end, adaptively matching various driving scenarios while avoiding image quality degradation caused by algorithm conflicts, significantly improving the display effect of in-vehicle images. Attached Figure Description

[0027] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0028] Figure 1 This is a structural block diagram of an in-vehicle display system according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the first type of image processing method according to an embodiment of the present invention; Figure 3This is a schematic diagram of a second process of an image processing method according to an embodiment of the present invention; Figure 4 It is a region mask map according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the third process of the image processing method according to an embodiment of the present invention; Figure 6 This is a schematic diagram of an image processing architecture according to an embodiment of the present invention; Figure 7 This is a structural block diagram of an image processing apparatus according to an embodiment of the present invention; Figure 8 This is a structural block diagram of a vehicle according to an embodiment of the present invention; Figure 9 This is a schematic diagram of the hardware structure of the controller according to an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0031] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0032] As the capabilities of System-on-Chips (SoCs) in smart cockpits continue to improve, ISP modules have been gradually integrated into the SoC, replacing the independent ISP modules in traditional image acquisition systems. This not only reduces chip costs but also fully utilizes the SoC's hardware computing power to handle more complex and diverse scenarios. Meanwhile, the image quality optimization capabilities of the PQ module are also continuously being upgraded to meet user demands for image quality and the scenario adaptation requirements of in-vehicle display systems. However, as the two ends of the image data stream (image acquisition end and image display end), image data must be transferred through DDR memory. The parameters of the ISP and PQ modules cannot coordinate effectively, resulting in a severe algorithmic silo phenomenon.

[0033] In certain scenarios, the algorithms in the ISP and PQ modules conflict. For example, in low-light conditions such as tunnels or night driving, due to insufficient light intake from the camera's haptic sensor, the system automatically increases the gain, amplifying a large amount of noise. To suppress noise, the ISP module increases the noise reduction intensity, easily causing image edge blurring. Meanwhile, the PQ module sharpens the image to enhance details, which amplifies noise and creates a ringing effect. The two algorithms directly conflict, severely affecting the image display quality and weakening safety information such as lane lines and signs, reducing the recognition accuracy of Advanced Driver Assistance Systems (ADAS) and human eyes, thus impacting driving safety.

[0034] To address the aforementioned problems, this invention provides an image processing method. By determining the scene label of a target driving scene, a joint parameter mapping table is queried based on the scene label to match the target joint parameters corresponding to the target driving scene. These target joint parameters include a pair of first and second parameter groups. The cooperative constraint relationship between the first and second parameter groups effectively suppresses gain conflicts caused by opposite image gain trends between the two groups. Using the first and second parameter groups, the first and second image processors are controlled to sequentially perform image processing and optimization to obtain the target image. This establishes algorithmic collaborative processing between the image acquisition end and the image display end, adaptively matching various driving scenes while avoiding image quality degradation caused by algorithmic conflicts, significantly improving the display effect of in-vehicle images.

[0035] According to embodiments of the present invention, an in-vehicle display system is provided, such as... Figure 1 As shown, the in-vehicle display system 100 includes a controller 101, a first image processor 103 deployed at the image acquisition end 102, and a second image processor 105 deployed at the image display end 104. Specifically, the controller 101 is used for: The system acquires multimodal environmental perception data and raw image signals output by the image acquisition terminal 102, and determines the scene label of the target driving scene based on the multimodal environmental perception data. Based on the pre-built joint parameter mapping table, the target joint parameters corresponding to the scene label are obtained by querying. The target joint parameters include a pair of first parameter groups and second parameter groups. There is a cooperative constraint relationship between the first parameter groups and the second parameter groups. The cooperative constraint relationship is used to suppress the gain conflict between the first parameter groups and the second parameter groups. The gain conflict is used to characterize the opposite image gain trends of the first parameter groups and the second parameter groups to the image. The first image processor 103 processes the original image signal using the first parameter group to obtain the image to be displayed, and then forwards the image to be displayed to the second image processor 105; The second image processor 105 is controlled to optimize the image to be displayed using the second parameter group to obtain the target image, and the image display terminal 104 is controlled to display the target image.

[0036] In this embodiment, the image acquisition end 102 may include an image sensor, such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, to acquire raw images of the road in front of or around the vehicle and output raw image signals in RAW format. These raw image signals are then transmitted to the first image processor 103 to provide a data source for subsequent image processing. The first image processor 103 may be an ISP module, used to receive the raw image signals, perform basic preprocessing such as RAW de-mosaicing, black level calibration, and lens correction, and execute operations such as photosensitive gain adjustment, noise reduction, and high dynamic range (HDR) synthesis according to the first parameter set issued by the controller 101, before outputting the image to be displayed.

[0037] Furthermore, the controller 101 stores the image to be displayed in the memory DDR, and forwards the image to be displayed to the second image processor 105 through the memory DDR. The second image processor 105 may be a PQ module, which is used to perform sharpening, saturation adjustment and other operations according to the second parameter group issued by the controller 101, output a target image adapted to the target driving scene, and transmit it to the image display terminal 104, where the image display terminal 104 displays the target image.

[0038] The vehicle display system provided in this embodiment coordinates the first image processor 103 (e.g., ISP module) of the image acquisition end 102 and the second image processor 105 (e.g., PQ module) of the image display end 104 through the controller 101. Based on the joint parameter mapping table, the system configures target joint parameters for the first image processor 103 and the second image processor 105 that have cooperative constraints and match the target driving scene. This solves the problem of algorithm silos between the traditional image acquisition end and the image display end. Through the target joint parameters, the system directly suppresses the image processing algorithm conflicts between the first image processor 103 and the second image processor 105, effectively improving the display quality and visual effect of the vehicle images.

[0039] According to an embodiment of the present invention, an image processing method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] This embodiment provides an image processing method that can be used for, for example Figure 1 The in-vehicle display systems shown include, for example, in-vehicle display terminals and in-vehicle imaging systems. Figure 2 This is a flowchart of an image processing method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Acquire multimodal environmental perception data and raw image signals output by the image acquisition terminal, and determine the scene label of the target driving scene based on the multimodal environmental perception data.

[0041] Specifically, the system communicates with the environmental perception module in real time via the vehicle-mounted CAN FD bus, and acquires multimodal environmental perception data from the environmental perception module at regular intervals. The environmental perception module includes, but is not limited to, a light sensor, a GPS system, an accelerometer, an in-cabin camera, and millimeter-wave radar. The multimodal environmental perception data includes, but is not limited to, the ambient light intensity output by the light sensor, navigation information (such as vehicle positioning information) and map information (such as road matching information) output by the GPS system, the vibration frequency output by the accelerometer, and driver eye images captured by the in-cabin camera.

[0042] In this embodiment, the image acquisition end can be a sensor such as a CMOS image sensor, which is used to convert optical signals from the external environment into electrical signals and output raw image signals in RAW format. These raw image signals contain raw photosensitive information that has not undergone image preprocessing or optimization. Therefore, changes in ambient light intensity significantly affect the sensor gain. When the ambient light intensity is low, the sensor gain is significantly increased to achieve the brightness required for normal display of the output image, resulting in a significant increase in noise in the output signal.

[0043] In this embodiment, the target driving scene can be the driving area located in front of the vehicle. The scene label of the target driving scene can be determined by predicting the ambient light intensity of the target driving scene. The scene label can be divided into dark light scene, normal light scene and strong light scene according to the intensity of ambient light. The specific division can be adjusted according to the actual scene requirements.

[0044] In some embodiments, navigation information and map information can be combined to identify the road type of the target driving scenario located at a preset distance (e.g., 500 m, which can be adjusted according to actual needs) in front of the vehicle, and the weather conditions and time information of the target driving scenario can be obtained from the cloud. Then, by combining the road type, weather conditions, and time information of the target driving scenario, the scenario label of the target driving scenario is determined.

[0045] For example, if the detected road type is a preset road type, the scene label for the target driving scenario is "low-light scene," where the preset road type includes, but is not limited to, roads with low ambient light intensity such as tunnels and underground parking lots. If the detected road type is not a preset road type, the time information and weather conditions of the target driving scenario are further detected, specifically including: If the target driving scenario is detected during nighttime (e.g., 8 PM to 4 AM the next day), the scene label is "Dark Light Scene." If the target driving scenario is detected during midday (e.g., 10 AM to 2 PM) and the weather condition is the first weather condition (e.g., sunny), the scene label is "High Light Scene." If the target driving scenario is detected during midday (e.g., 10 AM to 2 PM) and the weather condition is the second weather condition (weather condition other than sunny, such as cloudy, rainy, foggy, snowy, etc.), the scene label is "Normal Light Scene." If the target driving scenario is detected during a time other than midday or nighttime, the scene label is "Normal Light Scene." It should be noted that the division between midday and nighttime can be further refined by considering time zone information and seasonal conditions.

[0046] The above embodiment performs a preliminary prediction of the scene label of the target driving scene before the vehicle enters the target driving scene, so as to load relevant joint parameters in advance and process the raw image signal in a timely manner. However, after the vehicle enters the target driving scene, the accuracy of the preliminary predicted scene label can be determined by combining the ambient light intensity collected in real time by the light sensor. For example, light intensity threshold intervals are divided for low light scenes, normal light scenes, and strong light scenes. If the ambient light intensity is detected to be within the corresponding light intensity threshold interval for a relatively long period of time (e.g., for 5 seconds to prevent data abrupt changes from causing judgment errors), then the scene corresponding to that light intensity threshold interval is determined as the scene label for secondary detection. If the preliminary predicted scene label is consistent with the scene label detected in the secondary detection after entering the target driving scene, it means that the preliminary predicted scene label is correct; if they are inconsistent, the scene label detected in the secondary detection after entering the target driving scene is taken as the standard, and the scene label of the target driving scene is corrected.

[0047] For example, after the vehicle enters the target driving scene, if the ambient light intensity is detected to be <100 Lux and the duration reaches 5 s, the scene is labeled as a dark light scene; if 100 Lux ≤ ambient light intensity ≤ 8000 Lux and the duration reaches 5 s, it is a normal light scene; if the ambient light intensity is >8000 Lux and the duration reaches 5 s, it is a strong light scene.

[0048] In some embodiments, scene labels can be further subdivided, for example, low-light scenes can be divided into global low-light scenes and glare scenes. Glare scenes refer to driving scenes where a localized extremely strong light source directly or indirectly shines into the image acquisition end, causing localized overexposure and a washed-out image. Examples include spotlights inside tunnels, strong light when a vehicle exits a tunnel, direct illumination from oncoming high beams in a night driving environment, and strong light from roadside reflective signs in a nighttime environment. After the vehicle enters the target driving scene, the ambient light intensity and the original image signal are combined in real time to determine whether it is a glare scene, including: If the ambient light intensity is detected to be within the light intensity threshold range corresponding to a dark scene, and the proportion of overexposed pixels (i.e., pixels whose brightness exceeds the preset brightness) in the brightness histogram of the original image signal exceeds the preset threshold (e.g., 20%), then the target driving scene is determined to be a glare scene with local strong light interference; otherwise, the target driving scene is determined to be a globally dark scene.

[0049] This embodiment uses multimodal environmental perception data to pre-predict scene labels for the target driving scene located in front of the vehicle, so that relevant joint parameters can be loaded before entering the target driving scene. Furthermore, after the vehicle enters the target driving scene, the pre-predicted scene labels are further detected and subdivided by combining real-time collected ambient light intensity and raw image signals, improving the accuracy of scene label determination and thus enhancing the scene adaptability of the joint parameters, ensuring optimal image display quality.

[0050] It should be noted that, for ease of understanding, the following embodiments are based on scene labels including low-light scenes (which can be divided into global low-light scenes and glare scenes), normal lighting scenes, and strong light scenes. However, in other embodiments, scene labels can be further subdivided, such as into tunnel scenes, underground parking garage scenes, glare scenes, night driving scenes, etc. The specific subcategories can be adjusted according to actual needs.

[0051] Step S202: Based on the pre-built joint parameter mapping table, the target joint parameters corresponding to the scene label are queried; wherein, the target joint parameters include a pair of first parameter groups and second parameter groups, and there is a cooperative constraint relationship between the first parameter groups and the second parameter groups. The cooperative constraint relationship is used to suppress the gain conflict between the first parameter groups and the second parameter groups. The gain conflict is used to characterize the opposite image gain trends of the first parameter groups and the second parameter groups on the image.

[0052] Specifically, the pre-built joint parameter mapping table uses scene labels as unique indexes and stores a pair of bound first parameter groups (i.e. parameter groups for the ISP module) and second parameter groups (i.e. parameter groups for the PQ module). There is a negatively correlated cooperative constraint relationship between the first parameter group and the second parameter group, which is used to suppress the gain conflict between the first parameter group and the second parameter group on the image.

[0053] It should be noted that the first parameter group is used to convert the original image signal into a high-quality visual image, and its overall image gain is positive. Gain conflict refers to the situation where, without coordinated constraints between the first and second parameter groups, the second parameter group will cancel or suppress the positive image gain brought about by image processing using the first parameter group, resulting in opposite image gain trends between the first and second parameter groups. For example, the ISP module reduces image noise by decreasing the edge sharpness gain, while the PQ module needs to increase the image sharpening gain to enhance details and improve image quality, which may lead to increased noise and blurring, resulting in increased image noise. Therefore, by configuring the coordinated constraint relationship between the first and second parameter groups, the image gain conflict between the two can be suppressed, preventing the second parameter group from exhibiting an image gain trend opposite to that of the first parameter group.

[0054] In this embodiment, before the vehicle enters the target driving scene, the corresponding target joint parameters are retrieved in advance from the joint parameter mapping table using the scene label of the target driving scene as an index. The target joint parameters include a first parameter group and a second parameter group. After the vehicle enters the target driving scene, a more accurate scene label can also be determined by combining the real-time collected ambient light intensity and the original image signal. If the scene label is consistent with the scene label initially predicted before the vehicle enters the target driving scene, there is no need to query the joint parameter mapping table; if they are inconsistent, the corresponding target joint parameters are retrieved again based on the new scene label.

[0055] Step S203: Control the first image processor to process the original image signal using the first parameter group to obtain the image to be displayed, and forward the image to be displayed to the second image processor.

[0056] Specifically, the first image processor performs basic preprocessing on the raw image signal (RAW format), including RAW depigmentation, black level calibration, and lens shading correction, to remove hardware noise and photosensitive distortion. Furthermore, the system sends a first set of parameters to the first image processor (i.e., the ISP module). Based on these parameters, the first image processor processes the raw image signal output from the image acquisition terminal to obtain the image to be displayed (YUV format image). Then, the image to be displayed is forwarded to the second image processor (PQ module).

[0057] In some embodiments, after the first image processor outputs the image to be displayed, the image to be displayed is stored in memory. Other modules can be used to update the image to be displayed, such as synthesizing entertainment information, warning prompts, AR navigation prompts, etc. in the image to be displayed, thereby increasing the richness of the content of the image to be displayed and meeting the diverse display needs of the vehicle display system.

[0058] Step S204: Control the second image processor to optimize the image to be displayed using the second parameter group to obtain the target image, and control the image display terminal to display the target image.

[0059] Specifically, a second set of parameters is sent to the second image processor (PQ module). Based on the parameters after collaborative constraints, the second image processor optimizes the image quality of the image to be displayed, avoiding suppressing or canceling the image gain generated by the first image processor. This solves the algorithmic silo problem caused by the independent control of the traditional ISP and PQ modules, suppresses algorithmic conflicts between the first and second image processors, and ultimately obtains a target image adapted to the target driving scenario. Then, the image display terminals such as the vehicle's instrument panel and central control screen are controlled to render and output the target image, ensuring image display effects under various driving scenarios.

[0060] The image processing method provided in this embodiment determines the scene label of the target driving scene by acquiring multimodal environmental perception data. Then, based on the scene label, it queries a joint parameter mapping table to match the target joint parameters corresponding to the target driving scene. The target joint parameters include a pair of first parameter groups and second parameter groups. The cooperative constraint relationship between the first parameter groups and the second parameter groups can effectively suppress gain conflicts caused by the opposite image gain trends of the two sets of parameters. Using the first parameter groups and the second parameter groups, the first image processor and the second image processor are controlled to perform image processing and optimization sequentially to obtain the target image. This establishes a collaborative processing algorithm between the image acquisition end and the image display end, which avoids image quality degradation caused by algorithm conflicts while adaptively matching various driving scenes, significantly improving the display effect of in-vehicle images.

[0061] This embodiment provides an image processing method that can be used for, for example Figure 1 The in-vehicle display systems shown include, for example, in-vehicle display terminals and in-vehicle imaging systems. Figure 3 This is a flowchart of an image processing method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps: Step S301: Acquire multimodal environment perception data and raw image signals output from the image acquisition terminal, and determine the scene label of the target driving scene based on the multimodal environment perception data. See details in [link to relevant documentation]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0062] Step S302: Based on the pre-built joint parameter mapping table, the target joint parameters corresponding to the scene label are queried; wherein, the target joint parameters include a pair of first parameter groups and second parameter groups, and there is a cooperative constraint relationship between the first parameter groups and the second parameter groups. The cooperative constraint relationship is used to suppress the gain conflict between the first parameter groups and the second parameter groups. The gain conflict is used to characterize that the image gain trends of the first parameter groups and the second parameter groups are opposite to those of the second parameter groups.

[0063] Specifically, the process of constructing the joint parameter mapping table includes: Step a1: Determine the first parameter group corresponding to each scene label and obtain the original parameter group of the second image processor; wherein, the first parameter group includes at least some of the following items: noise reduction intensity, exposure gain and high dynamic range, and the original parameter group includes at least some of the following items: sharpening intensity, first saturation of the image center region and second saturation of the image edge region.

[0064] In this embodiment, the scene label of the driving scene is first determined. For example, the scene label may include a global low light scene, a glare scene, a normal lighting scene, and a strong light scene. Then, for each scene label, a first parameter group (ISP parameters, i.e., the working parameters of the first image processor) is preset in combination with the scene characteristics. The first parameter group may select one or more of the following: noise reduction intensity, exposure gain, and high dynamic range.

[0065] For example, the first parameter set for a globally low-light scene is: noise reduction intensity 38 dB, exposure gain 1.8x, and HDR level Level 3 (highest level). Among these, the higher the noise reduction intensity, the lower the image noise; the higher the exposure gain, the higher the overall brightness of the image; and the higher the HDR level, the greater the stretching of the image's brightness and darkness. The first parameter set for a glare scene is: noise reduction intensity 32 dB, exposure gain 1.2x, and HDR level Level 3 (highest level). The first parameter set for a normal lighting scene is: noise reduction intensity 15 dB, exposure gain 1.0x, and HDR level Level 1 (basic level). The first parameter set for a bright light scene is: noise reduction intensity 10 dB, exposure gain 0.6x, and HDR level Level 2 (medium level).

[0066] In this embodiment, the original parameter set of the second image processor is its default optimized parameter set. This original parameter set does not consider its synergy with the first parameter set. The original parameter set can select one or more of the following: sharpening intensity, first saturation of the image center region, and second saturation of the image edge region. It should be noted that the image center region and the image edge region can be pre-divided according to the size of the image display device. Since the driver's attention is mainly focused on the image center region, the first saturation is greater than the second saturation in order to highlight the display content in the center region.

[0067] Step a2: Based on the image gain corresponding to the first parameter group and the original parameter group, adjust the original parameter group to obtain the second parameter group.

[0068] In some alternative implementations, step a2 above includes: Step a21: Determine the first image gain of the first parameter group and the second image gain of the original parameter group, and determine whether there is a gain conflict between the first image gain and the second image gain.

[0069] Specifically, image gain is used to characterize the overall gain trend of the parameter set on image noise. If image noise is suppressed, it indicates that the overall image gain is positive; if image noise is amplified, it indicates that the overall image gain is negative. The first image processor (ISP module) is used to convert the original image signal into a high-quality visual image based on the first parameter set. When setting the first parameter set corresponding to each scene label, it is necessary to ensure that the first image gain of the first parameter set is to suppress image noise.

[0070] In this embodiment, image noise can be divided into luminance noise, color noise, and motion blur noise. Luminance noise refers to noise caused by random and irregular fluctuations in pixel brightness values ​​in the image, manifested as black and white snowflake-like spots on the screen. Color noise refers to random color deviations in the RGB color channels of pixels in the image, manifested as scattered red, green, and blue noise blocks on the screen. Motion blur noise is a unique noise caused by excessive temporal domain noise reduction, manifested as blurred edges of moving objects in the image. The higher the exposure gain in the first parameter group, the greater the luminance and color noise; the higher the HDR level, the greater the luminance and color noise. The noise reduction intensity is used to suppress luminance and color noise, thereby suppressing image noise overall.

[0071] In this embodiment, sharpening intensity is used to enhance image edge gradients and high-frequency details. The higher the sharpening intensity, the easier it is to amplify brightness noise, color noise, and motion blur noise; the higher the saturation, the higher the color noise. Therefore, when optimizing image quality, it is necessary to suppress sharpening intensity and saturation to avoid amplifying image noise after suppressing it in the first parameter group, which would cause gain conflicts.

[0072] For example, if it is detected that the sharpening intensity in the original parameter set is greater than the sharpening intensity threshold, the first saturation in the image center region is greater than the first saturation threshold, and the second saturation in the image edge region is greater than the second saturation threshold, it indicates that the second image gain of the original parameter set amplifies image noise, that is, there is a gain conflict between the first image gain and the second image gain. The sharpening intensity threshold, the first saturation threshold, and the second saturation threshold can be adjusted in conjunction with specific scene labels.

[0073] Step a22: If there is a gain conflict between the first image gain and the second image gain, adjust the original parameter set in the direction of suppressing the gain conflict to obtain the second parameter set.

[0074] Specifically, if there is a gain conflict between the first image gain and the second image gain, at least one of the sharpening intensity, the first saturation, and the second saturation in the original parameter set is reduced, thereby adjusting the original parameter set in the direction of suppressing the gain conflict, so as to ensure that the obtained second parameter set forms a cooperative constraint with the first parameter set.

[0075] For example, the original parameter set is: sharpening intensity 1.2x, first saturation 1.2x, and second saturation 1.0x. For instance, if a gain conflict is detected in a globally low-light scene, the sharpening intensity is reduced from 1.2x to 0.85x (to avoid conflict with ISP noise reduction), the first saturation is increased from 1.2x to 1.3x (to adapt to the need for enhanced safe areas in low-light scenes), and the second saturation is reduced from 1.0x to 0.7x. The adjusted second parameter set is then: sharpening intensity 0.85x, first saturation 1.3x, and second saturation 0.7x. Furthermore, by combining scene labels, the original parameter set is adjusted to different degrees to obtain the second parameter set corresponding to each scene label.

[0076] This invention accurately determines whether there is a gain conflict between the first image gain corresponding to the first parameter group and the second image gain corresponding to the original parameter group, and adaptively adjusts the original parameter group in the direction of suppressing the gain conflict when a conflict exists, so as to obtain a cooperatively matched second parameter group. This effectively avoids the image gain of the second image processor from being mutually exclusive with that of the first image processor, and prevents the second parameter group from sharpening and enhancing noise by increasing saturation, thus avoiding the cancellation of the noise reduction effect of the first parameter group. While ensuring the image noise reduction effect, it also improves the rationality of image quality optimization, significantly suppresses image quality degradation phenomena such as noise amplification and edge distortion, and improves the final display effect of the image.

[0077] Step a3: Construct a joint parameter mapping table based on the first parameter group and the second parameter group corresponding to each scene label.

[0078] Specifically, the first parameter group corresponding to each scene label and the adjusted second parameter group are bound one by one to form a joint parameter mapping table. This joint parameter mapping table is stored in the controller's storage module with the scene label as the unique index, which facilitates quick query and retrieval in the future.

[0079] In some embodiments, after the joint parameter mapping table is constructed, the Over-the-Air (OTA) upgrade function can be used to continuously expand new scene labels according to the optimization needs of actual driving scenarios, and iteratively update the joint parameter configuration of each scene label to ensure the effectiveness of collaborative constraints and continuously suppress the gain conflict between the first image processor and the second image processor.

[0080] This invention determines a first parameter set corresponding to each scene label and an original parameter set of a second image processor. Based on the corresponding image gains, the original parameter set is adaptively adjusted to obtain a second parameter set, thereby constructing a joint parameter mapping table. This establishes a cooperative constraint relationship between the processing parameters of the first and second image processors, preventing gain conflicts due to opposing image gain trends at the parameter construction level. It effectively prevents the second image processor's sharpening and saturation enhancement from amplifying noise and weakening the noise reduction effect of the first image processor, achieving coordination between noise suppression and image quality optimization, and improving scene adaptability.

[0081] Step S303: Control the first image processor to process the original image signal using the first parameter group to obtain the image to be displayed, and then forward the image to be displayed to the second image processor. For details, please refer to [link to relevant documentation]. Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0082] Step S304: Control the second image processor to optimize the image to be displayed using the second parameter group to obtain the target image, and control the image display terminal to display the target image.

[0083] Specifically, step S304 includes: Step S3041: Identify safe pixels and non-safe pixels in the image to be displayed; wherein, safe pixels are pixels related to driving safety information, and non-safe pixels are pixels in the image other than safe pixels.

[0084] Specifically, after receiving the image to be displayed from the first image processor, the controller can divide the image into pixel-by-pixel regions using image semantic segmentation and object detection algorithms, distinguishing between safe pixels and unsafe pixels. Safe pixels refer to pixels within information regions in the image related to driving safety information, including but not limited to pixels corresponding to safety targets such as lane lines, road signs, traffic lights, vehicles ahead, pedestrians, non-motorized vehicles, road edges, and curbs. Unsafe pixels are the remaining pixels in the image after removing safe pixels.

[0085] In this embodiment, after pixel classification is completed, a region mask map with the same size as the image to be displayed is generated. This region mask map is used to mark the distribution locations of safe and unsafe pixels. Figure 4 As shown, the area occupied by the safe pixels associated with lane line A1 is marked as the safe mask area (gray area), and the area occupied by the remaining non-safe pixels is marked as the non-safe mask area R1 (white area).

[0086] In step S3042, the safe pixels are enhanced, and the image quality of the non-safe pixels is optimized using the second parameter group to obtain the target image, and the image display terminal is controlled to display the target image.

[0087] Specifically, the controller performs differentiated image processing based on the region mask map, performing targeted enhancement processing on safe pixels and conventional image quality optimization on non-safe pixels using a second parameter group: sharpening and color saturation are adjusted according to the sharpening intensity, first saturation, and second saturation in the second parameter group. Then, the enhanced safe pixel region is fused with the image quality optimized non-safe pixel region to obtain a target image that balances noise reduction, image quality optimization, and safety highlighting. The controller then controls the image display terminal to complete the display output, ensuring noise suppression and gain conflict while highlighting visual information related to driving safety.

[0088] This invention identifies safe pixels related to driving safety information and the remaining unsafe pixels in an image. It then performs targeted enhancement processing on the safe pixels to highlight key driving safety information, while simultaneously using a second parameter group to uniformly optimize the image quality of the unsafe pixels. This achieves differentiated processing of image regions, effectively enhancing the visual recognition of driving safety information and improving driving safety.

[0089] In some alternative implementations, the safety pixels are enhanced, including increasing the brightness of the safety pixels and enhancing the safety colors within the safety pixels, where the safety colors are used to characterize color features conveying driving safety information. For example, the brightness of the safety pixels can be increased by 15%, and the blue and red colors within the safety pixels can be enhanced to improve color purity and contrast, thus highlighting the safety information.

[0090] Furthermore, after optimizing the image quality of unsafe pixels using the second parameter group, in order to further highlight safe pixels, the saturation of unsafe pixels is reduced, the color performance of irrelevant backgrounds is weakened, redundant visual interference is reduced, and the accuracy of intelligent driving system in recognizing driving safety information and the driver's visual perception of driving safety information are enhanced.

[0091] This invention significantly enhances the visual recognition of driving safety information by increasing the brightness of safety pixels and enhancing the safety colors within them. Simultaneously, by optimizing the image quality of non-safety pixels and reducing their saturation, it effectively weakens color interference in the background area, preventing irrelevant areas from excessively attracting the driver's attention. While ensuring overall image quality optimization, it highlights driving safety information and safeguards driving safety.

[0092] It's important to understand that image noise also affects the rendering of driving safety information. For example, brightness noise such as snowflake-like spots can obscure the outlines of lane lines, obstacles, and other markings in the image. Color noise such as red and green patches can reduce the accuracy of intelligent driving systems in recognizing traffic lights, and motion blur noise can easily cause distance judgment errors in intelligent driving systems. Traditional in-vehicle systems use YUV to sRGB conversion for image processing. In this process, the red channel is compressed, and the PQ module adds a gamma curve on top of sRGB processing, which can easily lead to a decrease in the brightness of safety colors. Combined with factors such as rendering layer conflicts and ambient light interference, this can lead to a decrease in the accuracy of driving safety information recognition.

[0093] Therefore, this embodiment coordinates the image processing and optimization processes of the first and second image processors. After the first parameter group reduces image noise, the second parameter group avoids amplifying the image noise, thus preventing image noise from seriously affecting the display of driving safety information. Furthermore, by differentiating between safe and non-safe pixels, the recognition accuracy of driving safety information is further improved.

[0094] Taking the first image processor as the ISP module and the second image processor as the PQ module as an example, the ISP module and the PQ module dynamically match the joint parameters of the target in real time according to the scene label, and at the same time perform differential processing on safe pixels and non-safe pixels in the image. The above process can be represented by pseudocode as follows: struct SceneParamMap { uint8_t scene_id; / / Scene tag (0: global dark light, 1: glare, 2: strong light, etc.) struct { float denoise_db; / / Noise reduction level (dB) uint8_t hdr_level; / / HDR level (1-3) ... / / Other related parameters } isp_params; / / First parameter group struct { float sharpness_gain; / / Sharpening intensity gain float sat_center_gain; / / First saturation gain for the center region float sat_surround_gain; / / Second saturation gain for the edge region ... / / Other related parameters } pq_params; / / Second parameter group uint32_t security_mask[1080 / 8][1920 / 8]; / / Region mask image }; In the pseudocode of the above embodiment, scene_id is the scene label. The system selects the matching target joint parameters from the joint parameter mapping table SceneParamMap based on the scene label. The target joint parameters include a pair of first parameter groups isp_params and second parameter groups pq_params, which are used as processing parameters for the ISP module and PQ module, respectively. security_mask[1080 / 8][1920 / 8] is a region mask map. It marks the safe and unsafe pixels of the image, which facilitates the subsequent differential processing of safe and unsafe pixels. The differential processing flow is represented by the following pseudocode: def security_layer_remap(yuv_frame, mask): / / YUV space processing for y in range(height): for x in range(width): if mask[y][x] == SECURITY_AREA: / / Safe pixels Y = yuv_frame.Y[y][x] * 1.15 / / Brightness increased by 15% U = yuv_frame.U[y][x]* 1.2 / / Blue enhancement V = yuv_frame.V[y][x] * 1.3 / / Red enhancement else: / / non-safe pixels Y = yuv_frame.Y[y][x] U = yuv_frame.U[y][x] * 0.8 / / Saturation suppression V = yuv_frame.V[y][x] * 0.7 return YUVFrame(Y, U, V); In the pseudocode of the above embodiment, the brightness of the safety pixels is increased by 15%, and the blue and red components of the image are enhanced, thereby strengthening the safety pixels. Furthermore, to further highlight the safety pixels, the saturation of non-safe pixels is suppressed to ensure the accuracy of the intelligent driving system and the driver's recognition of driving safety information.

[0095] The image processing method provided in this embodiment determines scene labels through multimodal environment perception data, matches paired target joint parameters using a pre-constructed joint parameter mapping table, suppresses gain conflicts through the collaborative processing of the first parameter group and the second parameter group, avoids weakening the noise reduction effect by image quality optimization, and simultaneously distinguishes between safe pixels and unsafe pixels, increases the brightness and enhances the safe color of safe pixels, and reduces the saturation of unsafe pixels after image quality optimization using the second parameter group, thereby achieving a synergistic unity of noise suppression and image quality optimization, highlighting driving safety information, weakening background interference, and improving the display quality of in-vehicle images.

[0096] This embodiment provides an image processing method that can be used for, for example Figure 1 The in-vehicle display systems shown include, for example, in-vehicle display terminals and in-vehicle imaging systems. Figure 5 This is a flowchart of an image processing method according to an embodiment of the present invention, such as... Figure 5 As shown, the process includes the following steps: Step S501: Acquire multimodal environmental perception data and raw image signals output by the image acquisition terminal, and determine the scene label of the target driving scene based on the multimodal environmental perception data.

[0097] The image processing flow of this embodiment is explained below using a tunnel scene as an example. Before the vehicle enters the tunnel, the changes in values ​​of various sensors are monitored in real time, such as those of GPS, light sensor, cabin camera, and accelerometer. Based on navigation and map information, the controller detects that the driving area in front of the vehicle is a tunnel, and then determines the scene label as a global low-light scene.

[0098] Step S502: Determine the vehicle's current position and the target area of ​​the target driving scenario based on navigation information and map information; if the vehicle's current position is detected to be outside the target area and the distance between the vehicle's current position and the target area is less than a preset distance, proceed to step S503.

[0099] For example, the vehicle's current position is determined by combining navigation information, and the target area range of the tunnel is obtained based on map information. Before the vehicle enters the tunnel, if the controller detects that the distance between the vehicle's current position and the target area range is less than a preset distance (e.g., 500 m, adjustable), it triggers the preloading process of the target joint parameters and executes step S503.

[0100] Step S503: Based on the pre-built joint parameter mapping table, the target joint parameters corresponding to the scene label are queried; wherein, the target joint parameters include a pair of first parameter groups and second parameter groups, and there is a cooperative constraint relationship between the first parameter groups and the second parameter groups. The cooperative constraint relationship is used to suppress the gain conflict between the first parameter groups and the second parameter groups. The gain conflict is used to characterize the opposite image gain trends of the first parameter groups and the second parameter groups on the image.

[0101] As a vehicle enters a tunnel from outside, the ambient light intensity drops sharply, resulting in insufficient light intake for the CMOS image sensor. To compensate for this, the sensor amplifies the light intensity gain, which in turn amplifies image noise. The ISP module reduces this noise through noise reduction processing to output a high-quality image. However, the PQ module also amplifies noise during image quality optimization, further amplifying the noise reduced by the ISP module, creating a gain conflict. This can also lead to the loss of critical driving safety information, causing various safety hazards. For example, lane line recognition failure may cause the vehicle to deviate from its lane, obstacles may be misinterpreted as shadows, leading to delayed braking, and traffic signs may be misinterpreted, resulting in incorrect steering.

[0102] In this embodiment, a global low-light scene is used as the scene label. The joint parameter mapping table is queried, and the first parameter group is preloaded: noise reduction intensity 38 dB, HDR level Level 3 (highest level). The second parameter group is: sharpening intensity 0.85x, first saturation 1.3x, and second saturation 0.7x. Through the coordination of the first and second parameter groups, the gain conflict between them is reduced, and noise amplification is avoided.

[0103] Step S504: When the vehicle's current location is detected to be within the target area, step S505 is executed.

[0104] For example, when the ambient light intensity is detected to drop below 800 Lux and the rate of change of ambient light intensity is >500 Lux / ms, it indicates that the vehicle has entered the tunnel. At this time, the pre-loaded first parameter group and second parameter group are retrieved, and step S505 is executed to perform joint processing of the ISP module and PQ module.

[0105] In this embodiment, the first and second parameter groups are preloaded. When the vehicle enters the target driving scenario, the joint processing of the ISP module and PQ module begins, improving the parameter switching speed.

[0106] Step S505: Control the first image processor to process the original image signal using the first parameter group to obtain the image to be displayed, and forward the image to be displayed to the second image processor; control the second image processor to optimize the image to be displayed using the second parameter group to obtain the target image.

[0107] Exemplarily, the pseudo-code for the combined processing flow of the ISP module and the PQ module is as follows: def update_tunnel_params(lux, gps_zone): if lux<TUNNEL_LUX_THRES or gps_zone == GPS_TUNNEL: / / Determine the tunnel scenario current_map.scene_id = 0 / / The scene label is the global low-light scene # First parameter group current_map.isp_params.denoise_db = 38.0 / / Noise reduction intensity current_map.isp_params.hdr_level = 3 / / HDR level # Second parameter group current_map.pq_params.sharpness_gain = 0.85 / / Suppress sharpening current_map.pq_params.sat_center_gain = 1.3 / / Higher saturation in the center current_map.pq_params.sat_surround_gain = 0.7 / / Lower saturation at the edges # Region mask map set_security_mask(ROAD_CENTER_REGION); In this embodiment, after the vehicle enters the tunnel, the noise reduction intensity of the ISP module is adjusted to 38 dB, and the HDR level is adjusted to the highest level; in the PQ module, the sharpening suppression intensity parameter is used, and since the key information is mainly located in the middle, the first saturation in the central region is higher than the second saturation in the edge region. Refer again to Figure 4 , the central security mask area is marked as 1, and the surrounding non-security mask areas can be marked as 0, and then differential processing is performed on the security pixels and non-security pixels. Among them, the defined positions of the security mask area and the non-security mask area can be flexibly defined according to the scenario and the display terminal size.

[0108] In some embodiments, different hardware accelerations can also be performed on the ISP module and the PQ module. For example, when processes such as noise reduction, HDR level, and color matrix are involved, fixed hardware units can be used for acceleration, and the computing power of the ISP module and the PQ module can be improved through hardware acceleration. The execution core logic is as follows: void isp_pq_coupling(FrameBuffer *frame) { / / Read scene tags SceneTag tag = env_engine_get_scene(); / / Get parameters from the mapping table ISPPara isp_para = param_map[tag].isp; PQPara pq_para = param_map[tag].pq; / / Apply collaborative constraints if (isp_para.denoise_db>35.0f) { / / Stronger noise reduction results in weaker sharpening. pq_para.sharpness = constrain(pq_para.sharpness, 0.0f, 0.9f); } / / Hardware acceleration processing fixed_function_isp(frame, isp_para); / / Fixed-function ISP core fixed_function_pq(frame, pq_para); / / Fixed-function PQ core }; This invention determines the vehicle's current location and the target area of ​​the target driving scenario using navigation and map information. It determines the timing of parameter query and image processing based on the positional relationship between the vehicle and the target driving scenario. The target joint parameters are queried in advance when the vehicle approaches the target driving scenario to avoid parameter switching delays. Image processing is performed immediately after the vehicle enters the target driving scenario, improving the real-time performance and scenario adaptability of in-vehicle image processing, and effectively enhancing driving safety and user experience.

[0109] Step S506: Based on the driver's eye image, determine the driver's pupil diameter and the driver's gaze direction; if the driver's pupil diameter is detected to be larger than the preset pupil diameter, based on the driver's gaze direction, determine the driver's gaze area in the target image and increase the saturation of the gaze area.

[0110] Specifically, the controller acquires real-time images of the driver's eyes using an in-vehicle camera. These images are then processed for grayscale conversion, noise filtering, and pupil edge extraction to locate the pupil contour and calculate the pupil diameter. If the pupil diameter is larger than a preset diameter (e.g., 5.2 mm), it indicates increased driver attention. The controller then calculates the eye posture, determines the driver's gaze direction, and projects this gaze direction onto the target image plane on the display screen to obtain the driver's gaze area within the target image. For details regarding the driver's pupil diameter and gaze direction, please refer to relevant technical documentation.

[0111] In this embodiment, after obtaining the gaze region, the gaze region and the identified safe pixels are marked as safe mask regions to obtain a new region mask image, and image processing is performed based on the new region mask image. Since the saturation parameter of the safe mask region is high, by using the new region mask image for differential processing, the saturation of the gaze region can be improved, thereby enhancing the visual recognition of the gaze region. In other words, the position of the safe mask region in the region mask image can be dynamically generated based on environmental perception data.

[0112] In some embodiments, the region mask map can be dynamically updated by combining road curvature and eye-tracking center coordinates. Specifically, the road curvature of the current road is determined based on GPS positioning and map information, and the eye center coordinates are obtained by calculating the eye pose. Using the eye center coordinates as the starting point and the road curvature as the gaze direction, the driver's gaze area in the target image is determined, thereby dynamically updating the region mask map based on the gaze area. The specific calculation process for the gaze area can be found in the description of related technologies and will not be elaborated here. The following is example code: / / Dynamically update the region mask map based on multimodal environment perception data void update_security_mask() { / / GPS location of road curvature float road_curvature = gps.get_curvature(); / / Tracking the coordinates of the eye center Point gaze_center = camera.get_gaze_point(); / / Generate dynamic mask mask = calc_road_center_mask(road_curvature, gaze_center); }; This invention acquires the pupil diameter and gaze direction in real time through driver's eye images. When the pupil diameter is larger than a preset pupil diameter, it locates the driver's gaze area and directionally increases the saturation, adaptively enhancing the visual recognition of the driver's gaze area and improving the perception efficiency of key visual information.

[0113] Step S507: If the accelerometer vibration frequency is detected to be greater than the preset vibration frequency, jitter compensation is performed on the target image.

[0114] Specifically, the vibration frequency of the vehicle is collected using an accelerometer and compared with a preset vibration frequency (e.g., 15Hz). Jitter compensation is then performed using temporal denoising, and the temporal denoising weight is increased to enhance the compensation effect on the target image. The specific process of temporal denoising can be found in the descriptions of relevant technologies and will not be elaborated upon here.

[0115] This invention compensates for jitter in the target image when the accelerometer vibration frequency is high, thereby offsetting the image jitter, offset and blur caused by severe vehicle vibration, ensuring the stability and clarity of the displayed image, avoiding the impact of image jitter on the driver's recognition of key safety information, and improving driving safety.

[0116] Step S508: Control the image display terminal to display the target image.

[0117] The image processing method provided in this embodiment achieves scenario-based linkage between the ISP module and the PQ module algorithms through a joint parameter mapping table, effectively improving the parameter matching degree under different driving scenarios. The collaborative processing of the ISP module and the PQ module solves the problem of image processing algorithm conflicts between the image acquisition end and the image display end in traditional solutions. Furthermore, by differentiating between safe and unsafe pixels, it can highlight driving safety information in the image and ensure driving safety.

[0118] According to embodiments of the present invention, an image processing architecture is provided. For example... Figure 6 As shown, the image processing architecture adopts a three-stage pipeline architecture, namely the preprocessing stage, the joint processing stage, and the postprocessing stage.

[0119] In the preprocessing stage, preprocessing such as RAW depigmentation, black level calibration, and lens shading correction are performed to ensure image usability and remove the influence of image acquisition equipment on the image.

[0120] In the joint processing phase, the algorithm parameters of the ISP and PQ modules are matched by scene labels to solve the algorithm silo problem caused by the separation of algorithm parameters between the ISP and PQ modules. By dynamically constructing region mask maps, safe and non-safe pixels in the image are distinguished and differentiated, highlighting key driving safety information and addressing the problem of weakened safety information. Hardware acceleration significantly reduces software computing power.

[0121] In the post-processing stage, focusing on the image display end, the safety layer (containing driving safety information) and entertainment layer in the image are identified. The safety layer is given higher priority, and layered rendering of the safety and entertainment layers is adopted (suitable for multi-layer scenarios, such as when ADAS prompts, navigation, and entertainment screens need to be rendered simultaneously). The safety layer containing driving safety information is placed on top to maximize the recognizability of safety information, while hardware adds pass-through channels to reduce latency. For the entertainment layer, a compensation algorithm based on screen aging is added to address brightness reduction and color shift during screen use.

[0122] This embodiment also provides an image processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0123] This embodiment provides an image processing apparatus that can be applied to, for example... Figure 1 The in-vehicle display system shown is, for example Figure 7 As shown, the device includes: The first processing module 701 is used to acquire multimodal environmental perception data and raw image signals output by the image acquisition terminal, and determine the scene label of the target driving scene based on the multimodal environmental perception data. The second processing module 702 is used to query the target joint parameters corresponding to the scene label based on the pre-built joint parameter mapping table; wherein the target joint parameters include a pair of first parameter group and second parameter group, there is a cooperative constraint relationship between the first parameter group and the second parameter group, the cooperative constraint relationship is used to suppress the gain conflict between the first parameter group and the second parameter group, and the gain conflict is used to characterize the opposite image gain trend of the first parameter group and the second parameter group to the image. The third processing module 703 is used to control the first image processor to process the original image signal using the first parameter group to obtain the image to be displayed, and to forward the image to be displayed to the second image processor; The fourth processing module 704 is used to control the second image processor to optimize the image to be displayed using the second parameter group to obtain the target image, and to control the image display terminal to display the target image.

[0124] In some optional implementations, the first processing module 701 is further configured to: A first parameter group corresponding to each scene label is determined, and the original parameter group of the second image processor is obtained; wherein, the first parameter group includes at least some of the following items: noise reduction intensity, exposure gain and high dynamic range, and the original parameter group includes at least some of the following items: sharpening intensity, first saturation of the image center region and second saturation of the image edge region; Based on the image gain corresponding to the first parameter group and the original parameter group, the original parameter group is adjusted to obtain the second parameter group; Construct a joint parameter mapping table based on the first and second parameter groups corresponding to each scene label.

[0125] In some optional implementations, the first processing module 701 is further configured to: Determine the first image gain of the first parameter group and the second image gain of the original parameter group, and determine whether there is a gain conflict between the first image gain and the second image gain; If there is a gain conflict between the first image gain and the second image gain, the original parameter set is adjusted in the direction of suppressing the gain conflict to obtain the second parameter set.

[0126] In some alternative implementations, the fourth processing module 704 is further configured to: Identify safe pixels and non-safe pixels in the image to be displayed; where safe pixels are pixels related to driving safety information, and non-safe pixels are pixels in the image other than safe pixels. The safe pixels are enhanced, and the non-safe pixels are optimized using the second parameter group to obtain the target image.

[0127] In some alternative implementations, the fourth processing module 704 is further configured to: Increase the brightness of safety pixels and enhance the color of safety colors within the safety pixels; whereby safety colors are used to characterize the color features that convey driving safety information. After optimizing the image quality of the unsafe pixels using the second parameter group, the fourth processing module 704 is also used to reduce the saturation of the unsafe pixels.

[0128] In some optional implementations, the multimodal environment perception data includes navigation information and map information, and the target driving scenario includes the driving area located in front of the vehicle; after determining the scene label of the target driving scenario based on the multimodal environment perception data, the device is further used to: Based on navigation and map information, determine the vehicle's current location and the target area of ​​the target driving scenario; When it is detected that the vehicle's current position is outside the target area and the distance between the vehicle's current position and the target area is less than a preset distance, the step of querying the target joint parameters corresponding to the scene label based on the pre-built joint parameter mapping table is executed. When the vehicle's current position is detected to be within the target area, the first image processor is executed to process the original image signal using the first parameter group to obtain the image to be displayed.

[0129] In some optional implementations, the multimodal environment perception data also includes driver eye images; before displaying the target image at the control image display terminal, the fourth processing module 704 is further configured to: Based on the driver's eye images, determine the driver's pupil diameter and the driver's gaze direction; If the driver's pupil diameter is detected to be larger than the preset pupil diameter, the driver's gaze area in the target image is determined based on the driver's gaze direction, and the saturation of the gaze area is increased.

[0130] In some optional implementations, the multimodal environmental perception data also includes accelerometer vibration frequencies; before displaying the target image at the control image display terminal, the fourth processing module 704 is further configured to: If the accelerometer vibration frequency is detected to be greater than the preset vibration frequency, jitter compensation is performed on the target image.

[0131] This invention compensates for jitter in the target image when the accelerometer vibration frequency is high, thereby offsetting the image jitter, offset and blur caused by severe vehicle vibration, ensuring the stability and clarity of the displayed image, avoiding the impact of image jitter on the driver's recognition of key safety information, and improving driving safety.

[0132] The image processing apparatus provided in this embodiment of the invention can execute the image processing method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0133] According to embodiments of the present invention, a vehicle is provided, such as Figure 8 As shown, the vehicle includes, Figure 1 The vehicle display system 100 shown is shown.

[0134] Figure 9 This is a schematic diagram of the structure of a controller provided in an embodiment of the present invention.

[0135] The following is a detailed reference. Figure 9The diagram illustrates a structural schematic suitable for implementing a controller in an embodiment of the present invention. The controller may include a processor (e.g., a central processing unit, graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 902 or a program loaded from memory 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for controller operation. The processor 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0136] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows the controller to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 A controller with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and may alternatively implement or have more or fewer devices.

[0137] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a memory 908, or installed from a ROM 902. When the computer program is executed by the processor 901, it performs the functions defined in the image processing method of the embodiments of the present invention.

[0138] Figure 9 The controller shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0139] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the image processing method shown in the above embodiments is implemented.

[0140] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0141] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. An image processing method, characterized in that, The method is applied to an in-vehicle display system, the in-vehicle display system including a first image processor deployed at an image acquisition end and a second image processor deployed at an image display end; the method includes: Acquire multimodal environmental perception data and raw image signals output from the image acquisition terminal, and determine the scene label of the target driving scene based on the multimodal environmental perception data; Based on a pre-built joint parameter mapping table, the target joint parameters corresponding to the scene label are obtained by querying; wherein, the target joint parameters include a pair of first parameter groups and second parameter groups, and there is a cooperative constraint relationship between the first parameter group and the second parameter group. The cooperative constraint relationship is used to suppress the gain conflict between the first parameter group and the second parameter group, and the gain conflict is used to characterize that the first parameter group and the second parameter group have opposite image gain trends for the image. The first image processor is controlled to process the original image signal using the first parameter group to obtain the image to be displayed, and the image to be displayed is forwarded to the second image processor; The second image processor is controlled to optimize the image to be displayed using the second parameter group to obtain the target image, and the image display terminal is controlled to display the target image.

2. The image processing method according to claim 1, characterized in that, The step of optimizing the image to be displayed using the second parameter set to obtain the target image includes: Identify safe pixels and unsafe pixels in the image to be displayed; wherein, the safe pixels are pixels related to driving safety information, and the unsafe pixels are pixels in the image other than the safe pixels; The safe pixels are enhanced, and the non-safe pixels are optimized using the second parameter set to obtain the target image.

3. The image processing method according to claim 2, characterized in that, The enhancement process for the security pixel includes: The brightness of the safety pixel is increased, and the safety color in the safety pixel is enhanced; wherein the safety color is used to characterize the color features that convey driving safety information. After optimizing the image quality of the unsafe pixels using the second parameter set, the method further includes reducing the saturation of the unsafe pixels.

4. The image processing method according to claim 1, characterized in that, The process of constructing the joint parameter mapping table includes: A first parameter group corresponding to each scene label is determined, and the original parameter group of the second image processor is obtained; wherein, the first parameter group includes at least some of the following items: noise reduction intensity, exposure gain and high dynamic range, and the original parameter group includes at least some of the following items: sharpening intensity, first saturation of the image center region and second saturation of the image edge region; Based on the image gain corresponding to the first parameter group and the original parameter group, the original parameter group is adjusted to obtain the second parameter group; Construct a joint parameter mapping table based on the first and second parameter groups corresponding to each scene label.

5. The image processing method according to claim 4, characterized in that, The step of adjusting the original parameter set based on the image gain corresponding to the first parameter set and the original parameter set to obtain the second parameter set includes: Determine the first image gain of the first parameter group and the second image gain of the original parameter group, and determine whether there is a gain conflict between the first image gain and the second image gain; If there is a gain conflict between the first image gain and the second image gain, the original parameter set is adjusted in the direction of suppressing the gain conflict to obtain the second parameter set.

6. The image processing method according to any one of claims 1-5, characterized in that, The multimodal environment perception data includes navigation information and map information, and the target driving scenario includes the driving area located in front of the vehicle; After determining the scene label of the target driving scene based on the multimodal environment perception data, the method further includes: Based on the navigation information and the map information, determine the vehicle's current location and the target area range of the target driving scenario; When it is detected that the vehicle's current position is outside the target area and the distance between the vehicle's current position and the target area is less than a preset distance, the step of querying the target joint parameters corresponding to the scene label based on the pre-built joint parameter mapping table is executed. When the vehicle's current position is detected to be within the target area, the first image processor is controlled to process the original image signal using the first parameter group to obtain the image to be displayed.

7. The image processing method according to claim 6, characterized in that, The multimodal environmental perception data also includes images of the driver's eyes; Before displaying the target image on the image display terminal, the method further includes: Based on the driver's eye image, determine the driver's pupil diameter and the driver's gaze direction; If the driver's pupil diameter is detected to be larger than a preset pupil diameter, the driver's gaze area in the target image is determined based on the driver's gaze direction, and the saturation of the gaze area is increased.

8. The image processing method according to claim 6, characterized in that, The multimodal environmental sensing data also includes accelerometer vibration frequencies; Before displaying the target image on the image display terminal, the method further includes: If the accelerometer vibration frequency is detected to be greater than the preset vibration frequency, jitter compensation is performed on the target image.

9. An image processing apparatus, characterized in that, An application in an in-vehicle display system, the in-vehicle display system comprising a first image processor deployed at an image acquisition end and a second image processor deployed at an image display end; the device includes: The first processing module is used to acquire multimodal environmental perception data and raw image signals output by the image acquisition terminal, and determine the scene label of the target driving scene based on the multimodal environmental perception data. The second processing module is used to query the target joint parameters corresponding to the scene label based on a pre-built joint parameter mapping table; wherein the target joint parameters include a pair of first parameter groups and second parameter groups, there is a cooperative constraint relationship between the first parameter groups and the second parameter groups, the cooperative constraint relationship is used to suppress the gain conflict between the first parameter groups and the second parameter groups, and the gain conflict is used to characterize that the first parameter groups and the second parameter groups have opposite image gain trends for the image; The third processing module is used to control the first image processor to process the original image signal using the first parameter group to obtain the image to be displayed, and to forward the image to be displayed to the second image processor; The fourth processing module is used to control the second image processor to optimize the image to be displayed using the second parameter group to obtain the target image, and to control the image display terminal to display the target image.

10. A vehicle, characterized in that, The vehicle includes an in-vehicle display system, which includes a controller, a first image processor deployed at an image acquisition end, and a second image processor deployed at an image display end; the controller includes: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the image processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the image processing method according to any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the image processing method according to any one of claims 1 to 8.

Images