Image processing method and device, storage medium and processor

Abstract

The invention discloses an image processing method and device, a storage medium and a processor. According to the scheme, an original image of the surrounding environment of a vehicle is collected, downsampling processing is carried out on the original image, and a preview is generated; segmenting the preview into a plurality of image areas, and calculating image entropies of all pixels in each image area; determining the region type of the corresponding region in the original image according to the image entropy of each image region; the region type comprises a plurality of salient regions of different grades, and the grades of the salient regions are in positive correlation with the image entropy; adjusting the resolution of a corresponding region in the original image according to the region type, and outputting the adjusted image as a target image; the resolution ratio of the corresponding region is positively correlated with the grade of the salient region corresponding to the corresponding region. Differentiation processing is carried out on the image through a dynamic resolution control mechanism, the computing power resource utilization rate of the intelligent driving system is improved, and the operation cost of the intelligent driving system is remarkably reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, apparatus, storage medium and processor. Background Technology

[0002] In intelligent driving systems, the image perception module serves as the core input source for environmental modeling and decision-making, and its processing efficiency and robustness have a decisive impact on the overall vehicle intelligence level. Currently, existing intelligent driving systems generally use fixed resolution for full-frame image processing. However, when processing images using pixel-based full-frame processing mode, the system cannot effectively allocate computing resources, resulting in a significant amount of computing power being wasted on processing background areas such as the sky and road surface. This leads to low utilization of computing resources and high operating costs. For example, when an intelligent driving system uses full-frame processing mode (such as 8-megapixel processing mode) to process images, approximately 85% of the total computing power of 4.2 TOPS is consumed in processing background areas such as the sky and road surface, leaving only 1.8 TOPS of effective computing power consumed in processing key areas such as pedestrians and vehicles. The bandwidth utilization rate is only 32% (only 4.1 GB / s of effective data transmission bandwidth out of a total bandwidth of 12 GB / s), indicating a significant waste of computing power and bandwidth. This drastically reduces the utilization rate of computing resources in intelligent driving systems and fails to meet the low-power requirements of in-vehicle systems. Summary of the Invention

[0003] To address the aforementioned issues, this application provides an image processing method, apparatus, storage medium, and processor, with the aim of improving the utilization rate of computing resources in intelligent driving systems and reducing the operating costs of intelligent driving systems.

[0004] The embodiments of this application disclose the following technical solutions:

[0005] In a first aspect, this application provides an image processing method, the method comprising:

[0006] Acquire raw images of the environment surrounding the vehicle, and downsample the raw images to generate a preview image;

[0007] The preview image is divided into multiple image regions, and the image entropy of all pixels in each image region is calculated; the image entropy is used to characterize the information richness of the image region.

[0008] Based on the image entropy of each of the image regions, the region type of the corresponding region in the original image is determined; the region type includes multiple levels of salient regions, and the level of the salient region is positively correlated with the image entropy;

[0009] The resolution of the corresponding region in the original image is adjusted according to the region type, and the adjusted image is output as the target image; the resolution of the corresponding region is positively correlated with the level of the salient region corresponding to the corresponding region.

[0010] In one feasible implementation, the region type includes highly saliency regions, moderately saliency regions, and low saliency regions;

[0011] The step of determining the region type of the corresponding region in the original image based on the image entropy of each of the image regions includes:

[0012] If the image entropy of the image region is less than a first preset threshold, then the region type of the corresponding region in the original image is determined to be the low saliency region;

[0013] If the image entropy of the image region is greater than or equal to the first preset threshold, and the image entropy is less than or equal to the second preset threshold, then the region type of the corresponding region in the original image is determined to be the salient region; the first preset threshold is less than the second preset threshold.

[0014] If the image entropy of the image region is greater than the second preset threshold, then the region type of the corresponding region in the original image is determined to be the highly salient region.

[0015] In one feasible implementation, adjusting the resolution of the corresponding region in the original image according to the region type includes:

[0016] If the region type of the corresponding area in the original image is the highly salient region, then the corresponding region is controlled to maintain its original resolution;

[0017] If the region type of the corresponding area in the original image is the medium salient region, then a pixel averaging merge operation is performed on the corresponding region to adjust the resolution of the corresponding region to a first resolution; the first resolution is less than the original resolution, and the pixel averaging merge operation is used to merge multiple adjacent pixel values ​​in the corresponding region into one pixel value;

[0018] If the region type of the corresponding area in the original image is the low saliency region, then a weighted pixel merging operation is performed on the corresponding region to adjust the resolution of the corresponding region to a second resolution; the second resolution is less than the first resolution, and the weighted pixel merging operation is used to perform weighted fusion processing on the pixel values ​​of multiple preset positions in the corresponding region based on the center pixel weight coefficient, and the center pixel weight coefficient is used to adaptively adjust the degree of image detail preservation in the corresponding region according to the temperature change of the environment in which the vehicle is located.

[0019] In one feasible implementation, performing a weighted pixel merging operation on the corresponding region to adjust the resolution of the corresponding region to a second resolution includes:

[0020] The center pixel weight coefficient is determined based on the temperature change and the temperature compensation coefficient; the temperature change is the difference between the current temperature of the environment where the vehicle is located and the threshold temperature; the temperature compensation coefficient is used to adaptively adjust the weight according to the temperature change at a rate of 0.0005 / ℃.

[0021] Based on the center pixel weight coefficient, the pixel values ​​at multiple preset positions within the corresponding region are weighted and fused to obtain the fused pixel values.

[0022] Based on the fused pixel values, the resolution of the corresponding region is adjusted to the second resolution.

[0023] In one feasible implementation, the pixel values ​​at the plurality of preset positions include pixel values ​​in the center region, pixel values ​​in the left adjacent region, pixel values ​​in the right adjacent region, and pixel values ​​in the bottom adjacent region. The step of weighted fusion processing of the pixel values ​​at the plurality of preset positions within the corresponding region based on the center pixel weight coefficient to obtain the fused pixel value includes:

[0024] Based on the center pixel weight coefficient, the pixel values ​​of the center region, the left neighboring region, the right neighboring region, and the bottom neighboring region in the corresponding region are weighted and fused to obtain the fused pixel value.

[0025] In one feasible implementation, after adjusting the resolution of the corresponding region to the second resolution, the image processing method further includes:

[0026] The peak signal-to-noise ratio (PSNR) of the low saliency region is determined based on the difference between each pixel value of the low saliency region in the original image and the corresponding fused pixel value; the PSNR is used to reflect whether there is distortion in the low saliency region after the resolution is adjusted to the second resolution;

[0027] Determine whether the peak signal-to-noise ratio is greater than a third preset threshold;

[0028] If the peak signal-to-noise ratio is greater than the third preset threshold, it is determined that the low saliency region does not have the distortion condition after the resolution is adjusted to the second resolution, and the adjusted image is obtained based on each corresponding adjusted region;

[0029] If the peak signal-to-noise ratio is less than or equal to the third preset threshold, it is determined that the distortion exists in the low saliency region, and a parameter feedback adjustment mechanism is triggered; the parameter feedback adjustment mechanism is used to readjust the resolution of the low saliency region until the peak signal-to-noise ratio is greater than the third preset threshold.

[0030] In one feasible implementation, before outputting the adjusted image as the target image, the image processing method further includes:

[0031] Calculate the target detection rate corresponding to the adjusted image; the target detection rate is used to characterize the detection probability of a preset target in the adjusted image;

[0032] If the target detection rate corresponding to the adjusted image is greater than a preset value, then the adjusted image will be output as the target image.

[0033] A second aspect of this application provides an image processing apparatus, the apparatus comprising:

[0034] The acquisition module is used to acquire raw images of the environment surrounding the vehicle, and to downsample the raw images to generate a preview image;

[0035] The calculation module is used to segment the preview image into multiple image regions and calculate the image entropy of all pixels in each image region; the image entropy is used to characterize the information richness of the image region.

[0036] The determination module is used to determine the region type of the corresponding region in the original image based on the image entropy of each of the image regions; the region type includes multiple levels of salient regions, and the level of the salient region is positively correlated with the image entropy;

[0037] The adjustment module is used to adjust the resolution of the corresponding region in the original image according to the region type, and output the adjusted image as the target image; the resolution of the corresponding region is positively correlated with the level of the salient region corresponding to the corresponding region.

[0038] In a third aspect, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method described above.

[0039] In a fourth aspect, this application provides a processor for running a computer program, which executes the image processing method described above during runtime.

[0040] Compared with the prior art, this application has the following advantages:

[0041] In this technical solution, the original image of the vehicle's surrounding environment is first acquired. Then, the high-resolution original image is downsampled to generate a low-resolution preview image, avoiding the need for complex image entropy calculations directly on the high-resolution original image, thus effectively reducing computational resource consumption. Next, the preview image is divided into multiple image regions, and the image entropy of all pixels within each region is calculated. This quantifies the information richness of the image region through image entropy, providing a data basis for subsequent dynamic resolution adjustment and avoiding errors caused by subjective experience. Then, based on the image entropy of each region, the region type in the original image is determined. The region type includes multiple levels of salient regions, with a positive correlation between the salient region level and image entropy, and a positive correlation between the resolution of the corresponding region and the level of the corresponding salient region. Therefore, the resolution of different regions in the original image can be dynamically adjusted based on the region type, and the adjusted image is output as the target image. This achieves precise allocation of computational resources, significantly reducing computational consumption in non-critical areas (such as the sky, road surface, and other background areas), thereby effectively improving the utilization rate of computational resources in the intelligent driving system, significantly reducing the operating cost of the intelligent driving system, and meeting the low-power consumption requirements of the vehicle system. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0044] Figure 2 A flowchart illustrating another image processing method provided in this application embodiment;

[0045] Figure 3 A flowchart illustrating a process for determining a region type, as provided in an embodiment of this application;

[0046] Figure 4 A flowchart illustrating an image resolution adjustment process provided in this application embodiment;

[0047] Figure 5 A flowchart of an image quality verification process provided in this application embodiment;

[0048] Figure 6 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Detailed Implementation

[0049] As described earlier, current intelligent driving systems generally use fixed resolution for full-frame image processing. However, when processing images using pixel-based full-frame processing, the system cannot effectively allocate computing resources, resulting in a significant amount of computing power being wasted on processing background areas such as the sky and road surface. This leads to low utilization of computing resources and high operating costs. For example, when an intelligent driving system uses full-frame processing (such as 8-megapixel processing), approximately 85% of the total computing power of 4.2 TOPS is consumed in processing background areas such as the sky and road surface, leaving only 1.8 TOPS of effective computing power for processing critical areas such as pedestrians and vehicles. The bandwidth utilization rate is only 32% (only 4.1 GB / s of effective data transmission bandwidth out of a total bandwidth of 12 GB / s). This represents a significant waste of computing power and bandwidth, drastically reducing the utilization rate of computing resources in the intelligent driving system and failing to meet the low-power requirements of the vehicle system.

[0050] Furthermore, existing technologies, when processing images using a pixel-based full-frame processing mode, struggle to adapt to changes in ambient temperature. This can easily lead to feature layer distortion in the processed images (such as edge blurring, texture loss, and geometric distortion), severely impacting the perception reliability of intelligent driving systems. For example, when the performance of a CMOS sensor fluctuates by more than ±15% within a wide temperature range of -40℃ to 85℃, it directly causes feature layer distortion in the image processing output. This results in a 38% increase in the target false detection rate (i.e., the probability of missing pedestrians, vehicles, and other targets), seriously affecting the perception reliability of intelligent driving systems, and potentially triggering functional safety boundary failures, especially in extreme temperature scenarios.

[0051] The inventors have proposed an image processing method, apparatus, storage medium, and processor. This scheme involves acquiring raw images of the vehicle's surrounding environment, then downsampling the high-resolution raw images to generate a low-resolution preview image. This avoids the need for complex image entropy calculations directly on the high-resolution raw images, effectively reducing computational resource consumption. The preview image is then divided into multiple image regions, and the image entropy of all pixels within each region is calculated. This quantifies the information richness of the image regions through image entropy, providing a data basis for subsequent dynamic resolution adjustment and avoiding errors caused by subjective judgment. Finally, based on the data from each image region… Image entropy determines the region type of a corresponding area in the original image. The region type includes multiple levels of salient regions. The level of the salient region is positively correlated with the image entropy, and the resolution of the corresponding region is positively correlated with the level of the salient region. Therefore, the resolution of different regions in the original image can be dynamically adjusted according to the region type, and the adjusted image can be output as the target image. This achieves precise allocation of computing resources, significantly reduces the computing power consumption of non-critical areas (such as the sky, road surface, and other background areas), effectively improves the utilization rate of computing resources in the intelligent driving system, significantly reduces the operating cost of the intelligent driving system, and meets the low power consumption requirements of the vehicle system.

[0052] Furthermore, in the technical solution of this application, when the region type of the corresponding area in the original image is a low saliency region, the pixel values ​​of multiple preset positions in the corresponding region are weighted and fused based on the center pixel weight coefficient. This can adaptively adjust the degree of image detail preservation in the corresponding region according to the temperature change of the vehicle's environment, so that the system can adapt to the influence of environmental temperature changes and avoid the problem of feature layer distortion (such as edge blurring, texture loss, geometric distortion) in the processed image. This reduces the target false detection rate (i.e., the probability of false detection of pedestrians, vehicles, and other targets) and improves the perception reliability of the intelligent driving system.

[0053] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0054] Method Implementation Examples

[0055] This application provides an embodiment of an image processing method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although the flowchart shows a logical order, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0056] See Figure 1 The figure is a flowchart of an image processing method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:

[0057] Step S101: Acquire the original image of the environment around the vehicle, and perform downsampling processing on the original image to generate a preview image.

[0058] In one feasible embodiment, an image processing system can serve as the execution subject of the image processing method of this application embodiment. For ease of description, the image processing system will be referred to simply as the system below.

[0059] In this embodiment of the application, the system can acquire raw images of the vehicle's surrounding environment (i.e., ...) through visual sensors (such as monocular cameras, fisheye cameras, etc.) installed on the vehicle. Figure 2 The system takes an original image A, which may include targets such as pedestrians, vehicles, houses, and roads appearing around the vehicle; then, the system can downsample the high-resolution original image to generate a low-resolution preview image. For example, as... Figure 2 As shown, the system can perform an 8x downsampling process on the original image A to obtain an image B (i.e., a preview image) with a resolution of 1 / 8 of the original image A. This avoids the need to perform highly complex image entropy calculations directly on the high-resolution original image, thereby effectively reducing the consumption of computing resources.

[0060] It should be noted that the process of downsampling the original image by 8 times, reducing the width and height of the original image to 1 / 8 of their original size, to generate the preview image takes no more than 5 milliseconds, effectively improving the system response speed and optimizing the system resource utilization.

[0061] Step S102: Divide the preview image into multiple image regions and calculate the image entropy of all pixels in each image region.

[0062] In step S102, image entropy is used to characterize the information richness of an image region (i.e., the richness of targets such as pedestrians, vehicles, and houses in the image). The higher the image entropy, the higher the information value of the image region, and the lower the image entropy, the lower the information value of the image region.

[0063] In one implementation, the image entropy can be the gray-level distribution entropy, and the system can calculate the gray-level distribution entropy of all pixels in the image region based on the pixel gray-level values ​​of each image region.

[0064] In this embodiment of the application, the system can divide the preview image into multiple image regions of the same size, such as... Figure 2 As shown, the system can divide image B into multiple image regions of the same size according to the size of 8×8 pixels. Each image region is a local window in image B centered at coordinates (x,y). Then, the system can calculate the image entropy E(x,y) of all pixels in the image region based on the pixel gray value of each image region using the formula (1) shown below. This provides a data basis for subsequent dynamic resolution adjustment and avoids errors caused by subjective experience judgment.

[0065] (1)

[0066] Where i represents the pixel grayscale value, which in a standard 8-bit grayscale image ranges from 0 (representing pure black) to 255 (representing pure white), for a total of 256 grayscale levels; P i This represents the probability of pixel grayscale value i occurring. For example, when the image region contains preset targets such as pedestrians and vehicles, the calculated image entropy can be shown in Table 1.

[0067] Table 1

[0068]

[0069] As shown in Table 1, when the image region contains pedestrians, the image entropy calculated by the system based on the pixel grayscale values ​​of the image region is between 7.2 and 8.5 bits; when the image region contains cars, the image entropy calculated by the system based on the pixel grayscale values ​​of the image region is between 7.5 and 9.0 bits; and when the image region contains bicycles, the image entropy calculated by the system based on the pixel grayscale values ​​of the image region is between 6.8 and 7.9 bits.

[0070] Step S103: Determine the region type of the corresponding region in the original image based on the image entropy of each image region.

[0071] In step S103, the region type includes multiple levels of salient regions, and the level of the salient region is positively correlated with the image entropy. Specifically, the region type may include high salient regions, medium salient regions, and low salient regions, and the mapping relationship between image entropy and region type can be shown in the examples in Table 2.

[0072] Table 2

[0073]

[0074] In this embodiment, there is a spatial mapping relationship between the image region and its corresponding region in the original image. That is, the corresponding region is the region in the original image whose spatial position is consistent with the segmented region of the preview image (i.e., the aforementioned image region). For example, if the image region is a k×k region located at the target position in the preview image, the corresponding region is an Nk×Nk region in the original image corresponding to that target position. The system can determine the region type of the corresponding region in the original image based on the image entropy of each image region. For example, in the scenario shown in Table 2, when the calculated image entropy of the corresponding region is 7.6, the system can determine that the region type of the corresponding region is a highly salient region; when the calculated image entropy of the corresponding region is 5.5, the system can determine that the region type of the corresponding region is a moderately salient region; and when the calculated image entropy of the corresponding region is 3, the system can determine that the region type of the corresponding region is a lowly salient region.

[0075] See Figure 3 The figure is a flowchart of a region type determination process provided in an embodiment of this application, which includes the following steps:

[0076] Step S1031: If the image entropy of the image region is less than the first preset threshold, then the region type of the corresponding region in the original image is determined to be a low saliency region.

[0077] In the embodiments of this application, such as Figure 2 As shown, when the image entropy E(x,y) of the image region is less than 4 (i.e., the first preset threshold), the system can determine that the region type of the corresponding region in the original image is a low saliency region, that is, the corresponding region in the original image is a region containing background such as road surface and sky.

[0078] Step S1032: If the image entropy of the image region is greater than or equal to the first preset threshold and the image entropy is less than or equal to the second preset threshold, then the region type of the corresponding region in the original image is determined to be a salient region.

[0079] In step S1032, the first preset threshold is less than the second preset threshold. For example... Figure 2 As shown, the first preset threshold can be set to 4, and the second preset threshold can be set to 7.

[0080] In the embodiments of this application, such as Figure 2 As shown, when the image entropy E(x,y) of the image region is greater than or equal to 4 and E(x,y) is less than or equal to 7, the system can determine that the region type of the corresponding region in the original image is a salient region, that is, the corresponding region in the original image is a transition region containing pedestrians or vehicles and the background (such as road surface, sky, etc.).

[0081] Step S1033: If the image entropy of the image region is greater than the second preset threshold, then the region type of the corresponding region in the original image is determined to be a highly salient region.

[0082] In the embodiments of this application, such as Figure 2 As shown, when the image entropy E(x,y) of the image region is greater than 7 (i.e., the second preset threshold), the system can determine that the region type of the corresponding region in the original image is a highly significant region, that is, the corresponding region in the original image is a region containing preset targets such as pedestrians and vehicles.

[0083] It should be noted that determining the region type of a region by using the image entropy of the image region can effectively distinguish key information from the background in the image, enhance the system's ability to understand the image content, thereby reducing the subsequent investment of computing resources in non-important regions, while ensuring that important regions (such as regions containing pedestrians and vehicles) are effectively processed, thus optimizing the system's computing resource utilization efficiency.

[0084] Step S104: Adjust the resolution of the corresponding region in the original image according to the region type, and output the adjusted image as the target image.

[0085] In step S104, the resolution of the corresponding region is positively correlated with the level of the salient region corresponding to that region. For example, if the level of the salient region corresponding to the corresponding region is high (i.e., high salient region), the resolution of the corresponding region can be set to high resolution (e.g., 1080p resolution); if the level of the salient region corresponding to the corresponding region is medium (i.e., medium salient region), the resolution of the corresponding region can be set to medium resolution (e.g., 720p resolution); if the level of the salient region corresponding to the corresponding region is low (i.e., low salient region), the resolution of the corresponding region can be set to low resolution (e.g., 480p resolution).

[0086] In this embodiment, the system can dynamically adjust the resolution of different regions in the original image based on the region type. For example, image regions of the highly salient region type maintain high resolution; image regions of the moderately salient region type are adjusted to medium resolution; and image regions of the lowly salient region type are adjusted to low resolution. The system can then output the adjusted image as the target image, achieving precise allocation of computing resources, significantly reducing computing power consumption in non-critical areas (such as background areas like the sky and road surface), thereby effectively improving the utilization rate of computing resources in the intelligent driving system, significantly reducing the operating cost of the intelligent driving system, and meeting the low power consumption requirements of the vehicle system.

[0087] See Figure 4 The figure is a flowchart of an image resolution adjustment process provided in an embodiment of this application, which includes the following steps:

[0088] Step S1041: If the region type of the corresponding region in the original image is a highly salient region, then control the corresponding region to maintain its original resolution.

[0089] For example, when the original resolution of the original image is 1080p and the corresponding region is a highly salient region, the system can control the corresponding region to maintain its original resolution of 1080p.

[0090] Step S1042: If the region type of the corresponding area in the original image is a salient region, then perform a pixel averaging merge operation on the corresponding region to adjust the resolution of the corresponding region to the first resolution.

[0091] In step S1042, the first resolution is less than the original resolution, and the pixel averaging merge operation is used to merge multiple adjacent pixel values ​​in the corresponding region into one pixel value.

[0092] In this embodiment of the application, the system performs the following operation based on the formula (2): Figure 2 The 2×2 pixel average merging shown merges multiple adjacent pixel values ​​in the corresponding area into one pixel value, thereby adjusting the resolution of the corresponding area to a first resolution (e.g., from the original resolution 1080p to 720p), maximizing the saving of computing resources while ensuring the recognizability of the preset target.

[0093] P merged1 = (P1+P2+P3+P4) / 4 (2)

[0094] Among them, P merged1 The values ​​represent the merged pixel values; P1, P2, P3, and P4 represent the pixel values ​​at adjacent 2×2 positions in the corresponding regions.

[0095] It should be noted that the number of merged groups of pixel regions in the central salient region is the total number of pixels in the central salient region / 4.

[0096] Step S1043: If the region type of the corresponding area in the original image is a low saliency region, then a weighted pixel merging operation is performed on the corresponding region to adjust the resolution of the corresponding region to the second resolution.

[0097] In step S1043, the second resolution is smaller than the first resolution. The weighted pixel merging operation is used to perform weighted fusion processing on the pixel values ​​of multiple preset positions in the corresponding area based on the center pixel weight coefficient. The center pixel weight coefficient is used to adaptively adjust the degree of image detail preservation in the corresponding area according to the temperature change of the vehicle's environment.

[0098] In this embodiment, if the region type of the corresponding area in the original image is a low saliency region, the system can perform a weighted pixel merging operation on the corresponding region in the original image using the formula (3) shown below to obtain the fused pixel value P. merged2Based on the merged pixel values, the resolution of the corresponding area is adjusted to a second resolution (e.g., from the original resolution of 1080p to 480p), thereby significantly reducing the computing power consumption of the corresponding area (e.g., background areas such as the sky and road surface).

[0099] P merged2 =α·P c +[(1-α) / 3]·(P l +P r +P b (3)

[0100] Among them, P merged2 α represents the merged pixel value, ranging from 0 to 255; α represents the center pixel weight coefficient, ranging from [0.42, 0.46]; P c P represents the pixel value of the central region. l P represents the pixel value of the left neighboring region. r Pixel values ​​in the adjacent area on the right, P b The pixel values ​​in the bottom adjacent area range from [0, 255]; (1-α) / 3 represents the weight coefficient of the surrounding pixels, which is used to control the contribution of the surrounding pixels to the result (the total weight is always 1), and the range is (0, 0.2). Among them, the weight coefficient α of the center pixel can be calculated based on the temperature change of the environment in which the vehicle is located, as shown in the following formula (4).

[0101] α = 0.45 - 0.0005 × ΔT (4)

[0102] Here, 0.45 is the baseline weight for 25℃, used to balance the optimal value between image detail preservation and noise suppression at normal temperature; ΔT represents the temperature change of the vehicle's environment, which is the current temperature T of the environment where the vehicle is located. current The difference between the temperature and the threshold temperature (e.g., 25℃), i.e., ΔT = T current -25, ranging from [-45, +60]; 0.0005 is the temperature compensation coefficient, used to adjust the current temperature T. current The change in temperature (i.e., ΔT) is adaptively weighted at a rate of 0.0005 / ℃.

[0103] It should be noted that when the region type of the corresponding area in the original image is a low saliency region, the pixel values ​​of multiple preset positions in the corresponding region are weighted and fused based on the center pixel weight coefficient. This can adaptively adjust the degree of image detail preservation in the corresponding region according to the temperature change of the vehicle's environment, so that the system can adapt to the influence of environmental temperature changes and avoid the problem of feature layer distortion (such as edge blurring, texture loss, geometric distortion) in the processed image. This reduces the target false detection rate (i.e., the probability of missing the detection of pedestrians, vehicles, and other targets) and improves the perception reliability of the intelligent driving system.

[0104] Specifically, if the region type of the corresponding area in the original image is a low saliency region, the system can determine the center pixel weight coefficient based on the temperature change and the temperature compensation coefficient using the above formula (4); where the temperature change is the difference between the current temperature of the vehicle's environment and the threshold temperature, and the temperature compensation coefficient is used to adaptively adjust the weight according to the temperature change at a ratio of 0.0005 / ℃; then the system can perform weighted fusion processing on the pixel values ​​of multiple preset positions in the corresponding region based on the center pixel weight coefficient using the above formula (5) to obtain the fused pixel values; then the system can adjust the resolution of the corresponding region to the second resolution based on the fused pixel values.

[0105] In one feasible embodiment, the pixel values ​​at multiple preset positions may include the pixel values ​​of the center region, the pixel values ​​of the left neighboring region, the pixel values ​​of the right neighboring region, and the pixel values ​​of the bottom neighboring region. The system can perform weighted fusion processing on the pixel values ​​of the center region, the pixel values ​​of the left neighboring region, the pixel values ​​of the right neighboring region, and the pixel values ​​of the bottom neighboring region in the corresponding region based on the center pixel weight coefficient to obtain the fused pixel value.

[0106] Further, see Figure 5 The figure is a flowchart of an image quality verification process provided in an embodiment of this application. The process includes the following steps:

[0107] Step S501: After adjusting the resolution of the corresponding region to the second resolution, the peak signal-to-noise ratio of the low saliency region is determined based on the difference between each pixel value of the low saliency region in the original image and the corresponding fused pixel value.

[0108] In step S501, the peak signal-to-noise ratio is used to reflect whether there is distortion in the low saliency region after the resolution is adjusted to the second resolution.

[0109] In this embodiment of the application, after the system adjusts the resolution of the corresponding region to the second resolution, it can use formula (5) to determine the peak signal-to-noise ratio (PSNR) of the low saliency region based on the difference between each pixel value of the low saliency region in the original image and the corresponding fused pixel value, so as to evaluate whether there is distortion in the image of the low saliency region after pixel merging, so as to avoid the problem of blurring and structural distortion of the image of the low saliency region due to excessive aggregation, and ensure the stability of the image quality of the low saliency region after pixel merging.

[0110] (5)

[0111] Among them, P a,i and P b,i If the i-th pixel is located at the same coordinate position in the original image, and the i-th pixel is located in a key region (such as lane lines or other important regions), then P... a,i =X, P b,i =0, where X represents the pixel value of the key region; if the location of the i-th pixel is in a non-key region (such as the background region like the road surface), then P a,i =0, P b,i =Y, where Y represents the pixel value of the non-critical region; 0.7 and 0.3 are human visual perception weighting coefficients; P merged2,i is the merged pixel value; N is the total number of pixels in the low saliency region in the original image.

[0112] It should be noted that using PSNR to evaluate the image quality of low saliency areas after pixel merging can introduce visual saliency weights to evaluate image quality through a "human eye weighted fusion" strategy. This makes the evaluation results more in line with the human eye's focus on key areas, and is more suitable for the visual perception quality evaluation of ADAS (Advanced Driver Assistance Systems). This avoids misjudgments caused by errors in non-critical areas, reduces interference from low saliency areas, and thus solves the limitations of traditional PSNR in measuring image quality through mean square error (MSE), especially the problem that misjudgments are easily caused by errors in non-critical areas in low saliency areas after pixel merging.

[0113] Step S502: Determine whether the peak signal-to-noise ratio is greater than the third preset threshold.

[0114] In this embodiment of the application, the system can dynamically control the image processing quality by determining whether the peak signal-to-noise ratio is greater than a third preset threshold (e.g., 20dB), thereby optimizing the efficiency of computing resource utilization.

[0115] Step S503: If the peak signal-to-noise ratio is greater than the third preset threshold, it is determined that the low saliency region does not have distortion after the resolution is adjusted to the second resolution, and the adjusted image is obtained based on each corresponding adjusted region.

[0116] In this embodiment, if the peak signal-to-noise ratio is greater than a third preset threshold (e.g., 20dB), the system can determine that there is no distortion in the low saliency region. At this time, the adjusted image can be directly output without additional image processing, which effectively saves computing resources.

[0117] Step S504: If the peak signal-to-noise ratio is less than or equal to the third preset threshold, it is determined that there is distortion in the low saliency region, and the parameter feedback adjustment mechanism is triggered.

[0118] In step S504, the parameter feedback adjustment mechanism is used to readjust the resolution of the low saliency region until the peak signal-to-noise ratio is greater than the third preset threshold.

[0119] In this embodiment, if the peak signal-to-noise ratio is less than or equal to a third preset threshold, the system can determine that there is distortion in the low saliency region. At this time, only the corresponding region is optimized (i.e., the parameter feedback adjustment mechanism is triggered to readjust the resolution of the corresponding region), avoiding redundancy processing of the whole image and further optimizing the utilization efficiency of computing resources.

[0120] Furthermore, before outputting the adjusted image as the target image, such as Figure 2 As shown, the system can also calculate the target detection rate corresponding to the adjusted image; the target detection rate is used to characterize the detection probability of preset targets (such as pedestrians, vehicles, traffic signs and lane lines) in the adjusted image; if the target detection rate corresponding to the adjusted image is greater than the preset value (e.g., 98%), the adjusted image is output as the target image, thereby providing data preparation for downstream planning and control; if the target detection rate corresponding to the adjusted image is less than or equal to the preset value, a reprocessing mechanism is triggered to re-acquire the original image of the vehicle's surrounding environment and dynamically adjust the resolution of different regions in the original image.

[0121] In one feasible implementation, this application has significantly improved upon existing technologies in terms of reducing computing power consumption and storage bandwidth, as shown in Table 3.

[0122] Table 3

[0123]

[0124] In another feasible implementation, this application (taking Horizon Journey 6 + LPDDR5 as an example) has significantly improved cost optimization benefits compared to existing technologies (such as NVIDIA Orin NX + GDDR6), achieving savings on the order of millions: 148 × 1,000,000 = 148 million, as shown in Table 4.

[0125] Table 4

[0126]

[0127] In particular, this application has significantly improved the efficiency of heat dissipation cost optimization compared with the prior art. As shown in Table 5, when the ambient temperature of the vehicle is -40℃, the heat dissipation cost of the prior art is 82, while the heat dissipation cost required by the technical solution of this application is 45, which significantly reduces the heat dissipation cost of the vehicle.

[0128] Table 5

[0129]

[0130] The image processing method provided in this application avoids performing highly complex image entropy calculations directly on high-resolution original images, effectively reducing the consumption of computing resources; it realizes the quantification of information richness in image regions through image entropy, providing a data basis for subsequent dynamic resolution adjustment and avoiding errors caused by subjective experience judgment; it realizes the dynamic adjustment of the resolution of different regions in the original image and outputs the adjusted image as the target image, achieving precise allocation of computing resources, significantly reducing the computing power consumption of non-critical areas (such as background areas such as the sky and road surface), thereby effectively improving the utilization rate of computing resources in the intelligent driving system, significantly reducing the operating cost of the intelligent driving system, and meeting the low power consumption requirements of the vehicle system.

[0131] Furthermore, through the image processing method provided in this application embodiment, when the region type of the corresponding area in the original image is a low saliency region, the degree of image detail preservation in the corresponding area can be adaptively adjusted according to the temperature change of the vehicle's environment. This enables the system to adapt to the influence of environmental temperature changes, avoid the problem of feature layer distortion (such as edge blurring, texture loss, and geometric distortion) in the processed image, thereby reducing the target false detection rate (i.e., the probability of false detection of pedestrians, vehicles, and other targets) and improving the perception reliability of the intelligent driving system.

[0132] Device Examples

[0133] This application provides an image processing apparatus, wherein... Figure 6 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application, as shown below. Figure 6As shown, the device includes: a data acquisition module 11, a calculation module 12, a determination module 13, and an adjustment module 14. From... Figure 6 You can see the connections between several modules.

[0134] The acquisition module 11 is used to acquire original images of the environment around the vehicle and to downsample the original images to generate a preview image.

[0135] The calculation module 12 is used to divide the preview image into multiple image regions and calculate the image entropy of all pixels in each image region; the image entropy is used to characterize the information richness of the image region.

[0136] The determination module 13 is used to determine the region type of the corresponding region in the original image based on the image entropy of each image region; the region type includes multiple levels of salient regions, and the level of salient region is positively correlated with the image entropy;

[0137] The adjustment module 14 is used to adjust the resolution of the corresponding region in the original image according to the region type, and output the adjusted image as the target image; the resolution of the corresponding region is positively correlated with the level of the salient region corresponding to the corresponding region.

[0138] In one feasible implementation, the region types include highly saliency regions, moderately saliency regions, and low saliency regions; the determination module is specifically used for:

[0139] If the image entropy of an image region is less than a first preset threshold, then the region type of the corresponding region in the original image is determined to be a low saliency region.

[0140] If the image entropy of the image region is greater than or equal to the first preset threshold and the image entropy is less than or equal to the second preset threshold, then the region type of the corresponding region in the original image is determined to be a salient region; the first preset threshold is less than the second preset threshold.

[0141] If the image entropy of an image region is greater than the second preset threshold, then the region type of the corresponding region in the original image is determined to be a highly significant region.

[0142] In one feasible implementation, the adjustment module includes: a control unit, a first execution unit, and a second execution unit.

[0143] The control unit is used to control the corresponding region to maintain its original resolution if the region type of the corresponding region in the original image is a highly salient region.

[0144] The first execution unit is configured to perform a pixel averaging merge operation on the corresponding region if the region type of the corresponding region in the original image is a centrally salient region, so as to adjust the resolution of the corresponding region to a first resolution; the first resolution is less than the original resolution, and the pixel averaging merge operation is used to merge multiple adjacent pixel values ​​in the corresponding region into one pixel value.

[0145] The second execution unit is used to perform a weighted pixel merging operation on the corresponding region if the region type of the corresponding region in the original image is a low saliency region, so as to adjust the resolution of the corresponding region to a second resolution. The second resolution is less than the first resolution. The weighted pixel merging operation is used to perform weighted fusion processing on the pixel values ​​of multiple preset positions in the corresponding region based on the center pixel weight coefficient. The center pixel weight coefficient is used to adaptively adjust the degree of image detail preservation in the corresponding region according to the temperature change of the vehicle's environment.

[0146] In one feasible implementation, the second execution unit includes: a first determining unit, a processing unit, and an adjusting unit.

[0147] The first determining unit is used to determine the center pixel weight coefficient based on the temperature change and the temperature compensation coefficient; the temperature change is the difference between the current temperature of the vehicle's environment and the threshold temperature, and the temperature compensation coefficient is used to adaptively adjust the weight according to the temperature change at a rate of 0.0005 / ℃.

[0148] The processing unit is used to perform weighted fusion processing on the pixel values ​​of multiple preset positions in the corresponding region based on the center pixel weight coefficient, so as to obtain the fused pixel value;

[0149] The adjustment unit is used to adjust the resolution of the corresponding area to a second resolution based on the fused pixel values.

[0150] In one feasible implementation, the pixel values ​​at multiple preset positions include pixel values ​​in the center region, pixel values ​​in the left adjacent region, pixel values ​​in the right adjacent region, and pixel values ​​in the bottom adjacent region; the adjustment unit is specifically used for:

[0151] Based on the center pixel weight coefficient, the pixel values ​​of the center region, the left neighboring region, the right neighboring region, and the bottom neighboring region in the corresponding region are weighted and fused to obtain the fused pixel value.

[0152] In one feasible implementation, the image processing apparatus further includes: a first calculation unit, a judgment unit, a second determination unit, and a triggering unit.

[0153] The first calculation unit is used to determine the peak signal-to-noise ratio of the low saliency region based on the difference between each pixel value of the low saliency region in the original image and the corresponding fused pixel value after adjusting the resolution of the corresponding region to the second resolution. The peak signal-to-noise ratio is used to reflect whether there is distortion in the low saliency region after adjusting the resolution to the second resolution.

[0154] The judgment unit is used to determine whether the peak signal-to-noise ratio is greater than a third preset threshold.

[0155] The second determining unit is used to determine that if the peak signal-to-noise ratio is greater than the third preset threshold, the low saliency region does not have distortion after the resolution is adjusted to the second resolution, and obtains the adjusted image based on each corresponding adjusted region.

[0156] The triggering unit is used to determine that there is distortion in the low saliency region if the peak signal-to-noise ratio is less than or equal to the third preset threshold, and to trigger the parameter feedback adjustment mechanism. The parameter feedback adjustment mechanism is used to readjust the resolution of the low saliency region until the peak signal-to-noise ratio is greater than the third preset threshold.

[0157] In one feasible implementation, the image processing apparatus further includes a second computing unit and a third determining unit.

[0158] The second calculation unit is used to calculate the target detection rate corresponding to the adjusted image; the target detection rate is used to characterize the detection probability of a preset target in the adjusted image;

[0159] The third determining unit is used to output the adjusted image as the target image if the target detection rate corresponding to the adjusted image is greater than a preset value.

[0160] Storage Media Examples

[0161] This application provides a computer-readable storage medium storing a program, which, when executed by a processor, implements some or all of the steps of the image processing method described in the foregoing method embodiments of this application. The storage medium can be any medium capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0162] Processor Implementation

[0163] This application provides a processor for running a program, wherein some or all of the steps of the image processing method described in the foregoing method embodiments are executed during program execution.

[0164] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. The components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment solution according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0165] The above is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized by, The method comprises the following steps: Collecting a raw image of the environment around the vehicle, and performing down-sampling processing on the raw image to generate a preview image; Segmenting the preview image into multiple image regions, and calculating the image entropy of all pixels in each image region; The image entropy is used to represent the information richness in the image region; According to the image entropy of each image region, the region type of the corresponding region in the raw image is determined; the region type includes multiple different levels of salient regions, and the level of the salient region is positively correlated with the image entropy; According to the region type, the resolution of the corresponding region in the raw image is adjusted, and the adjusted image is output as a target image; the resolution of the corresponding region is positively correlated with the level of the salient region corresponding to the corresponding region.

2. The method of claim 1, wherein, The region type includes high salient region, medium salient region and low salient region; The region type of the corresponding region in the raw image is determined according to the image entropy of each image region, which comprises: If the image entropy of the image region is less than a first preset threshold, the region type of the corresponding region in the raw image is determined as the low salient region; If the image entropy of the image region is greater than or equal to the first preset threshold and less than or equal to a second preset threshold, the region type of the corresponding region in the raw image is determined as the medium salient region; the first preset threshold is less than the second preset threshold; If the image entropy of the image region is greater than the second preset threshold, the region type of the corresponding region in the raw image is determined as the high salient region.

3. The method of claim 2, wherein, According to the region type, the resolution of the corresponding region in the raw image is adjusted, which comprises: If the region type of the corresponding region in the raw image is the high salient region, the corresponding region is controlled to keep the original resolution; If the region type of the corresponding region in the raw image is the medium salient region, a pixel average merging operation is performed on the corresponding region to adjust the resolution of the corresponding region to a first resolution; the first resolution is less than the original resolution, and the pixel average merging operation is used to combine multiple adjacent pixel values in the corresponding region into one pixel value; If the region type of the corresponding region in the raw image is the low salient region, a weighted pixel merging operation is performed on the corresponding region to adjust the resolution of the corresponding region to a second resolution; the second resolution is less than the first resolution, and the weighted pixel merging operation is used to perform weighted fusion processing on the pixel values of multiple preset positions in the corresponding region based on a center pixel weight coefficient, and the center pixel weight coefficient is used to adaptively adjust the image detail retention degree of the corresponding region according to the temperature variation of the environment where the vehicle is located.

4. The method of claim 3, wherein, The weighted pixel merging operation is performed on the corresponding region to adjust the resolution of the corresponding region to a second resolution, which comprises: determining the center pixel weight coefficient based on the temperature variation and a temperature compensation coefficient; the temperature variation is a difference between a current temperature of an environment where the vehicle is located and a threshold temperature, and the temperature compensation coefficient is used to adaptively adjust the weight at a rate of 0.0005 / ℃ according to the temperature variation; performing weighted fusion processing on pixel values of a plurality of preset positions in the corresponding region based on the center pixel weight coefficient to obtain a fused pixel value; adjusting the resolution of the corresponding region to the second resolution based on the fused pixel value.

5. The method of claim 4, wherein, The pixel values of the plurality of preset positions include center region pixel values, left adjacent region pixel values, right adjacent region pixel values, and bottom adjacent region pixel values. The weighted fusion processing on the pixel values of the plurality of preset positions in the corresponding region based on the center pixel weight coefficient to obtain the fused pixel value includes: performing weighted fusion processing on the center region pixel values, the left adjacent region pixel values, the right adjacent region pixel values, and the bottom adjacent region pixel values in the corresponding region based on the center pixel weight coefficient to obtain the fused pixel value.

6. The method of claim 4, wherein, After adjusting the resolution of the corresponding region to the second resolution, the method further includes: determining a peak signal-to-noise ratio of the low salient region according to a difference between each pixel value of the low salient region in the original image and a corresponding fused pixel value; the peak signal-to-noise ratio is used to reflect whether there is a distortion condition after adjusting the resolution of the low salient region to the second resolution; determining whether the peak signal-to-noise ratio is greater than a third preset threshold value; if the peak signal-to-noise ratio is greater than the third preset threshold value, it is determined that there is no distortion condition after adjusting the resolution of the low salient region to the second resolution, and the adjusted image is obtained based on each adjusted corresponding region; if the peak signal-to-noise ratio is less than or equal to the third preset threshold value, it is determined that there is a distortion condition in the low salient region, and a parameter feedback adjustment mechanism is triggered; the parameter feedback adjustment mechanism is used to readjust the resolution of the low salient region until the peak signal-to-noise ratio is greater than the third preset threshold value.

7. The method as claimed in claim 1, wherein, Before outputting the adjusted image as a target image, the method further includes: calculating a target detection rate corresponding to the adjusted image; the target detection rate is used to represent a detection probability of a preset target in the adjusted image; if the target detection rate corresponding to the adjusted image is greater than a preset value, the adjusted image is output as the target image.

8. An image processing apparatus characterized by comprising: includes: a collection module configured to collect an original image of a surrounding environment of a vehicle, and perform down-sampling processing on the original image to generate a preview image; a calculation module configured to divide the preview image into a plurality of image regions, and calculate image entropy of all pixels in each of the image regions; the image entropy is used to represent an information richness in the image region; A determining module is configured to determine a region type of a corresponding region in the original image according to an image entropy of each of the image regions; the region type includes a plurality of different levels of salient regions, and the level of the salient region is positively correlated with the image entropy; An adjusting module is configured to adjust a resolution of the corresponding region in the original image according to the region type, and output an adjusted image as a target image; the resolution of the corresponding region is positively correlated with the level of the salient region corresponding to the corresponding region.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program is run by a processor, the image processing method in any one of claims 1-7 is implemented.

10. A processor, comprising: A computer program product for running a computer program, and when the computer program is run, the image processing method in any one of claims 1-7 is executed.

Images