An image processing method, device, vehicle and medium

Abstract

This invention discloses an image processing method, apparatus, vehicle, and medium. The method includes: acquiring images from multiple cameras and current computing power performance indicators; determining a panoramic image containing multi-granularity semantic annotations based on the multi-camera images, current computing power performance indicators, and a lightweight AI model, wherein the multi-granularity semantic annotations include global scene labels and local region masks; and processing the panoramic image based on the global scene labels and local region masks to obtain an enhanced final output image. By dynamically allocating resources using the current computing power performance indicators and accurately perceiving the scene in the image based on a lightweight AI model, a panoramic image containing multi-granularity semantic annotations is formed. The panoramic image is processed using a matching algorithm, enabling dynamic selection of the enhancement algorithm and differentiated processing of local problem areas. The final output image is clearer, more detailed, and reduces the risk of accidents.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and more particularly to an image processing method, apparatus, vehicle, and medium. Background Technology

[0002] With the rapid iteration of automotive intelligence and connectivity technologies, low-speed 360-degree surround view systems have become one of the core safety assistance configurations for passenger cars, commercial vehicles (such as logistics vehicles and park shuttle buses), and special-purpose vehicles (such as construction machinery and AGVs).

[0003] The low-speed 360° surround view system uses 4-6 fisheye cameras deployed around the vehicle to collect 360° environmental images around the vehicle without blind spots. After image stitching and distortion correction, a bird's-eye view (BEV) is generated, providing the driver with all-round environmental perception information in low-speed driving (such as parking, entering a garage, or passing on a narrow road) or low-speed operation (such as site loading and unloading and park dispatching), effectively reducing the risk of collision caused by blind spots and improving driving and operation safety.

[0004] However, existing low-speed 360-degree surround view systems typically use cost-constrained camera modules, which encounter the following problems in complex scenarios: significantly increased image noise in low-light conditions at night; localized overexposure due to backlighting and oncoming vehicle headlights; noticeable occlusion by raindrops and dirt; and low-cost lenses lacking hydrophobic coatings, resulting in severe impacts from rain. Furthermore, traditional static algorithms cannot dynamically adjust enhancement strategies based on the environment; and due to the limited computing power of the host SOC, real-time discrimination using medium- to large-scale models is not feasible. Summary of the Invention

[0005] This invention provides an image processing method, apparatus, vehicle, and medium to dynamically adjust enhancement strategies under low computing power conditions and improve the quality of the output image.

[0006] According to a first aspect of the present invention, an image processing method is provided, comprising:

[0007] Acquire images from multiple cameras and current computing power performance indicators;

[0008] Based on the multi-camera images, the current computing power performance indicators, and the lightweight AI model, a panoramic image containing multi-granularity semantic annotations is determined. The multi-granularity semantic annotations include global scene labels and local region masks.

[0009] The panoramic image is processed based on the global scene label and the local region mask to obtain the enhanced final output image.

[0010] According to a second aspect of the present invention, an image processing apparatus is provided, comprising:

[0011] The information acquisition module is used to acquire images from multiple cameras and current computing power performance indicators;

[0012] The image determination module is used to determine a panoramic image containing multi-granularity semantic annotations based on the multi-camera images, the current computing power performance indicators, and the lightweight AI model. The multi-granularity semantic annotations include global scene labels and local region masks.

[0013] The image processing module is used to process the panoramic image based on the global scene label and the local region mask to obtain the enhanced final output image.

[0014] According to a third aspect of the present invention, a vehicle is provided, the vehicle comprising:

[0015] At least one controller; and

[0016] A memory communicatively connected to the at least one controller; wherein,

[0017] The memory stores a computer program that can be executed by the at least one controller, which enables the at least one controller to perform the image processing method according to any embodiment of the present invention.

[0018] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a controller to execute and implement the image processing method according to any embodiment of the present invention.

[0019] According to a fifth aspect of the present invention, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, which, when executed by a controller, implements the image processing method of any embodiment of the present invention.

[0020] The technical solution of this invention involves acquiring images from multiple cameras and current computing power performance metrics; determining a panoramic image with multi-granularity semantic annotations based on the images, current computing power performance metrics, and a lightweight AI model. These multi-granularity semantic annotations include global scene labels and local region masks; and processing the panoramic image based on the global scene labels and local region masks to obtain an enhanced final output image. Dynamic resource allocation is performed using the current computing power performance metrics, and the lightweight AI model accurately perceives the scene in the image, forming a panoramic image with multi-granularity semantic annotations. The panoramic image is processed using a matching algorithm, enabling dynamic selection of the enhancement algorithm and differentiated processing of local problem areas. The final output image is clearer, more detailed, and reduces the risk of accidents.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of an image processing method provided according to Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of an image processing method provided according to Embodiment 2 of the present invention;

[0025] Figure 3 This is the original panoramic image of an image processing method provided in Embodiment 2 of the present invention;

[0026] Figure 4 This is a panoramic image containing multi-granularity semantic annotation, provided by an image processing method according to Embodiment 2 of the present invention.

[0027] Figure 5 This is the original image of an image processing method provided in Embodiment 2 of the present invention;

[0028] Figure 6 This is an optimized image obtained using an image processing method provided in Embodiment 2 of the present invention.

[0029] Figure 7 This is a schematic diagram of the structure of an image processing device according to Embodiment 3 of the present invention;

[0030] Figure 8 This is a structural schematic diagram of a vehicle that implements an embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Example 1

[0034] Figure 1 This is a flowchart of an image processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to panoramic image optimization on low-computing-power platforms. The method can be executed by an image processing device, which can be implemented in hardware and / or software, and can be configured in a vehicle. Figure 1 As shown, the method includes:

[0035] S110: Acquire images from multiple cameras and current computing power performance indicators.

[0036] In this embodiment, multi-camera images refer to the original images simultaneously acquired by 4-6 fisheye cameras in the vehicle-mounted 360-degree surround view system, covering the entire field of vision around the vehicle and serving as the basic data source for panoramic image generation and optimization. Computational performance indicators can be understood as the real-time operating load data of the vehicle-mounted SOC (System-on-a-Chip), with core metrics including average CPU utilization and average inference time across multiple consecutive frames.

[0037] Specifically, the vehicle can simultaneously acquire original images of the surrounding area through 4-6 fisheye cameras to ensure a comprehensive view without any omissions. The controller can receive images from multiple cameras and collect the hardware operation data of the vehicle SOC and the results of each calculation in real time to obtain the current computing power performance indicators.

[0038] S120. Based on images from multiple cameras, current computing power performance indicators, and a lightweight AI model, determine panoramic images containing multi-granularity semantic annotations, which include global scene labels and local region masks.

[0039] The lightweight AI model can be understood as an artificial intelligence model adapted to low-computing-power automotive platforms. It employs lightweight network structures such as MobileNet and ShuffleNet, and simplifies model complexity through techniques like deep separable convolutions, channel shuffling, and lightweight feature pyramids. This allows it to complete scene classification and region detection with inference times below 5-8ms. The lightweight AI model may include: a preprocessing module, three deep separable convolutional layers, a channel shuffling module, a lightweight feature pyramid, and a feature post-processing module.

[0040] In this embodiment, multi-granularity semantic annotation can be understood as containing two levels of annotation information. Multi-granularity reflects the differentiated recognition of global and local aspects. The global scene label is the classification result of the overall environment (3-4 categories), such as daytime and sunny, nighttime, rainy night, and backlighting, used to match the global optimization direction. The local region mask is the pixel-level annotation of local problems (3-4 channels), such as overexposed areas, underexposed areas, and raindrop / stain areas, used to accurately locate the local areas requiring differentiated processing. The panoramic image can be understood as an initial bird's-eye view generated from images captured by multiple fisheye cameras after distortion correction, stitching, color equalization, and other preprocessing. It covers a 360-degree field of view around the vehicle and is the object of subsequent optimization processing.

[0041] Specifically, the controller can first dynamically adjust the image processing resolution, processing area, and frame rate based on the current computing power performance indicators. Then, it can preprocess the acquired multi-camera images according to the adjusted image processing indicators to obtain a panoramic bird's-eye view. This image is then input into a lightweight AI model to perform scene classification and region detection on the global bird's-eye view, resulting in multi-granular semantic annotations that include global scene labels and local region masks.

[0042] S130. Based on the global scene label and local area mask, process the panoramic image to obtain the enhanced final output image.

[0043] In this embodiment, the final output image can be understood as a panoramic image that has been processed by a targeted optimization algorithm to solve problems such as insufficient brightness, noise, overexposure, and raindrop obstruction.

[0044] Specifically, the controller can determine the appropriate optimization algorithm based on global scene labels and local area masks, process different areas through the optimization algorithm to obtain optimized local areas, and perform weighted fusion of all optimized local areas to obtain the enhanced final output image.

[0045] The technical solution of this invention involves acquiring images from multiple cameras and current computing power performance metrics; determining a panoramic image with multi-granularity semantic annotations based on the images, current computing power performance metrics, and a lightweight AI model. These multi-granularity semantic annotations include global scene labels and local region masks; and processing the panoramic image based on the global scene labels and local region masks to obtain an enhanced final output image. Dynamic resource allocation is performed using the current computing power performance metrics, and the lightweight AI model accurately perceives the scene in the image, forming a panoramic image with multi-granularity semantic annotations. The panoramic image is processed using a matching algorithm, enabling dynamic selection of the enhancement algorithm and differentiated processing of local problem areas. The final output image is clearer, more detailed, and reduces the risk of accidents.

[0046] Example 2

[0047] Figure 2 This is a flowchart of an image processing method provided in Embodiment 2 of the present invention. This embodiment is a further refinement of the above embodiment. Figure 2 As shown, the method includes:

[0048] S201. Acquire images from multiple cameras and current computing power performance indicators.

[0049] S202. Determine the image processing parameters based on the current computing power performance indicators and the set threshold.

[0050] In this embodiment, the set threshold can be understood as a judgment criterion used to adapt to the preset computing power load. The image processing parameters can be understood as image optimization configurations dynamically determined based on the computing power status.

[0051] Specifically, the controller can compare the current computing power performance indicators with the set thresholds to determine the image processing parameters that are different from those below.

[0052] Furthermore, based on the above embodiments, the steps for determining image processing parameters according to the current computing power performance indicators and set thresholds can be refined as follows:

[0053] If the SOC load rate in the current computing power performance metrics is greater than the first set threshold and less than or equal to the second set threshold, then the preset target resolution is used as the image processing parameter; if the SOC load rate is greater than the second set threshold, then the target region of interest metrics is used as the image processing parameter; the average inference time under multiple historical frames in the current computing power performance metrics is determined by a sliding window; based on the average inference time, the image frame processing rate is determined and used as the image processing parameter.

[0054] In this embodiment, SOC load rate can be understood as the proportion of hardware resources occupied by the vehicle system-on-a-chip. The first set threshold can be understood as a preset first judgment standard for computing power load, such as a CPU utilization rate of 80% (average upper limit). The second set threshold can be understood as a preset second judgment standard for computing power load, such as a CPU utilization rate of 90%. The preset target resolution can be understood as a preset image processing resolution standard adapted to medium-high load scenarios, such as an original input resolution of 1080p and a preset target resolution of 720p. By reducing the resolution, computing power consumption is reduced, ensuring smooth processing. The target region of interest index can be understood as a processing range parameter set for extremely high load scenarios, with the core referring to the near-vehicle region ROI (Region of Interest). By focusing on the core field of view and reducing the processing area, computing power is concentrated to ensure the image quality of key areas.

[0055] A sliding window can be understood as a real-time data statistics tool. For example, it selects five consecutive frames of image data as a statistical window to calculate the average inference time across multiple frames, avoiding policy misadjustment caused by fluctuations in single-frame data and improving the stability of parameter adjustments. Historical frames refer to image frames that have been processed before the current frame; their inference time data serves as a statistical sample to calculate the average inference time, reflecting the computing power trend over a period of time. The average inference time can be understood as the average of the inference times of multiple historical frames statistically analyzed through the sliding window, used to determine the current computing power processing efficiency and provide a basis for frame rate adjustments. The image frame processing rate can be understood as the number of image frames processed per unit time (e.g., 30 frames / second, 24 frames / second, 15 frames / second), a parameter dynamically adjusted based on the average inference time, used to maintain system real-time performance during computing power fluctuations.

[0056] Specifically, the controller can obtain the current SOC load rate (average CPU utilization) in real time and compare it with a first set threshold (e.g., 80%) and a second set threshold (e.g., 90%). If the current SOC load rate in the computing power performance indicators is greater than the first set threshold but less than or equal to the second set threshold (80% < SoC load rate ≤ 90%), the preset target resolution (720p) is used as the image processing parameter. By reducing the image resolution, computing power consumption is reduced, and the frame processing rate is appropriately reduced (e.g., from 30 frames / second to 24 frames / second) to balance processing effect and real-time performance. When the load rate exceeds the second threshold (SoC load rate > 90%), the target region of interest (ROI near the vehicle) is used as the image processing parameter. Only the core visual area around the vehicle is optimized, reducing the scope of invalid processing and concentrating computing power to ensure the image quality of key areas. If the SOC load rate is less than or equal to 80%, the system switches to the initial resolution (1080p). The controller can use a sliding window mechanism to continuously select multiple consecutive historical frames before the current frame as statistical samples. Data statistics: Extract the inference time data of each of these 5 historical frames (i.e., the time for the lightweight AI model to process a single frame image) and calculate the average inference time. The controller can dynamically adjust the image frame processing rate based on the average inference time. For example, if the average inference time is short, it means that the computing power is sufficient and the frame processing rate can be maintained or increased (e.g., 30 frames / second); if the average inference time is long, it means that the computing power is tight and the frame processing rate needs to be reduced (e.g., from 30 frames / second to 15 frames / second).

[0057] S203. Based on image processing parameters, preprocess the images from multiple cameras to obtain intermediate images.

[0058] In this embodiment, the intermediate image can be understood as an image adapted to the current computing power conditions.

[0059] Specifically, the controller can preprocess images from multiple cameras based on image processing parameters, such as adjusting the resolution, selecting images based on the target region of interest index, or responding to the frame processing rate, to obtain intermediate images.

[0060] S204. Perform panoramic fusion processing on the intermediate image to obtain the initial panoramic image.

[0061] In this embodiment, the initial panoramic image can be understood as the complete bird's-eye view obtained after panoramic fusion processing.

[0062] Specifically, the controller can perform distortion correction, stitching, and color equalization on the intermediate images to obtain the initial panoramic image.

[0063] S205. Input the initial panoramic image into the lightweight AI model to obtain a panoramic image containing multi-granular semantic annotations.

[0064] Specifically, the controller can input the initial panoramic image into a lightweight AI model to obtain a panoramic image containing multi-granular semantic annotations.

[0065] Furthermore, the global scene labels include daytime and sunny days, nighttime, rainy nights, and backlighting, while the local area masks include overexposed areas, underexposed areas, raindrop areas, and smudged areas.

[0066] For example, a specific image example can be used as a demonstration. Figure 3 This is the original panoramic image of an image processing method provided in Embodiment 2 of the present invention. Figure 4 The image processing method provided in Embodiment 2 of the present invention includes panoramic images with multi-granularity semantic annotations, from... Figure 3 It can be seen that there are overexposure and raindrops, which are identified and classified by a lightweight AI model. Figure 4 The result, such as Figure 4 As shown, red represents overexposed areas, green represents underexposed areas, and blue represents raindrop / stain areas. Accurate classification and identification of each area provides a basis for the selection of subsequent algorithms.

[0067] S206. If the global scene label is night or the local area mask is an underexposed area, then the panoramic image and / or the underexposed area are processed by the first algorithm to obtain the first enhancement mask.

[0068] In this embodiment, the first algorithm can be understood as an enhanced combination of algorithms adapted to nighttime / underexposed scenes. The first enhanced mask can be understood as an optimized mask for nighttime / underexposed problems obtained after processing the panoramic image or only processing the underexposed area through the first algorithm.

[0069] Specifically, when the global scene label is "nighttime" or the local area mask contains "underexposed areas", the first algorithm is triggered if either condition is met. For the panoramic image (global optimization) or only for the underexposed area (local optimization), fast bilateral filtering noise reduction (suppressing image noise in nighttime / underexposed scenes), contrast-limited adaptive histogram equalization (CLAHE, improving local brightness and contrast), and local gain adjustment (precisely brightening underexposed areas) are performed in sequence to obtain the first enhanced mask that solves the nighttime / underexposed problem.

[0070] For example, to suppress overexposure, optimization can be achieved through local tone remapping. By constructing a non-linear mapping curve, different brightness areas in the image are adjusted differently (such as peak areas). This means that the brightness range of highlight / overexposed areas is compressed, while the brightness of dark areas is moderately increased. Ultimately, this achieves the effect of "suppressing overexposure, restoring details, and balancing contrast". Moreover, it only applies to local problem areas, avoiding image distortion caused by global adjustments.

[0071] S207. If the global scene label is "rainy night" or the local area mask is "raindrop area", then the panoramic image and / or raindrop area are processed by the second algorithm to obtain the second enhanced mask.

[0072] In this embodiment, the second algorithm can be understood as an enhanced combination of algorithms adapted to rainy night / raindrop scenes. The second enhanced mask can be understood as an optimized mask for rainy night / raindrop problems obtained after processing the panoramic image or only processing the raindrop area through the second algorithm.

[0073] Specifically, if the global scene label is "rainy night" or the local area mask contains "raindrop area", the second algorithm is triggered if either condition is met. For panoramic images (global optimization) or only for raindrop areas (local optimization), guided filtering (preserving image edges), local contrast restoration with previous frames (restoring details occluded by raindrops), and edge protection blur repair (filling in raindrop occluded areas) are performed in sequence to obtain the second enhanced mask that solves the rainy night / raindrop problem.

[0074] For example, consider the treatment of raindrops. Figure 5 This is the original image of an image processing method provided in Embodiment 2 of the present invention. Figure 6 An optimized image obtained from an image processing method provided in Embodiment 2 of the present invention, such as... Figure 5 As shown in the figure, there are raindrop areas that are obscured. After processing by the second algorithm, the result is as follows: Figure 6 The results shown, after processing the raindrop portion, yielded a clearer effect.

[0075] S208. If the global scene label is backlight or the local area mask is an overexposed area, then the panoramic image and / or raindrop area are processed by the third algorithm to obtain the third enhancement mask.

[0076] In this embodiment, the third algorithm can be understood as a combination of enhancement algorithms adapted to backlight / overexposure scenes, including local tone mapping, highlight suppression, shadow compensation, and dynamic range compression, used to solve the problems of local overexposure and loss of detail caused by backlight. The third enhancement mask can be understood as an optimized mask for backlight / overexposure problems obtained after processing the panoramic image or only processing the overexposed area through the third algorithm.

[0077] Specifically, if the global scene label is "backlight" or the local area mask contains "overexposed areas", the third algorithm is triggered if either condition is met. For panoramic images (global optimization) or only for overexposed areas (local optimization), local tone mapping (compressing the highlight range), highlight suppression + shadow compensation (balancing the brightness of bright and dark areas), and dynamic range compression (restoring the details of overexposed areas) are performed in sequence to obtain the third enhanced mask that solves the backlight / overexposure problem.

[0078] S209. If the local area mask is a stained area, then the stained area is processed by the fourth algorithm to obtain the fourth enhanced mask.

[0079] In this embodiment, the fourth algorithm can be understood as a combination of enhanced algorithms adapted to the stained scene. The fourth enhanced mask can be understood as an optimized mask obtained after the fourth algorithm specifically processes the stained area.

[0080] Specifically, if the local area mask contains a "stain area", the fourth algorithm is triggered if the condition is met; only for the stain area, the following steps are performed in sequence: accurate positioning of the stain area, enhancement of dark details (enhancing the clarity of the background around the stain), and image repair of the stain area (completing the background information occluded by the stain); thus obtaining the fourth enhanced mask that solves the stain occlusion problem.

[0081] S210. Perform weighted fusion on the first enhancement mask, the second enhancement mask, the third enhancement mask and the fourth enhancement mask to obtain the enhanced final output image.

[0082] Specifically, the controller can analyze the processing areas corresponding to the first to fourth enhancement masks, and fuse them according to the corresponding weights to obtain the enhanced final output image. For example, if there are conflicting processing requirements between adjacent areas (such as the same area needing both brightening and underexposure reduction), the weights are set with reference to the processing methods of surrounding areas (such as a 20% weighting for the surrounding areas of a small area). The enhancement masks are fused according to the set weights to ensure a natural transition and color coordination between different optimization effects, resulting in an enhanced final output image that takes into account various scene issues and is then output to the vehicle display system for the driver to view.

[0083] Furthermore, the first, second, third, and fourth algorithms are executed in a parallel pipeline manner through a multi-core heterogeneous architecture, which includes a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU).

[0084] In this embodiment, the multi-core heterogeneous architecture can be understood as a hardware architecture composed of multiple processors (CPU, GPU, NPU) with different functions. The parallel pipeline approach can be understood as breaking down the tasks of the first to fourth algorithms and distributing them to the CPU, GPU, and NPU for simultaneous execution. Each processor advances synchronously according to the pipeline rhythm, avoiding serial waiting and significantly shortening the overall processing time.

[0085] Specifically, the controller can accurately allocate the tasks of the first to fourth algorithms to the CPU, GPU, and NPU based on the core requirements of each algorithm and the characteristics of the processor hardware, and execute the algorithms in a parallel pipeline manner.

[0086] For example, lightweight AI inference and region mask generation can be performed using the CPU, while simultaneously coordinating task startup, progress synchronization, and anomaly handling between the GPU and NPU to ensure a smooth pipeline. The GPU executes pixel-level batch processing algorithms, such as the first and third algorithms, for low-light enhancement, highlight suppression, and local contrast enhancement. The NPU executes AI-assisted restoration algorithms, such as the second and fourth algorithms, for raindrop area processing and rapid noise reduction.

[0087] As a first optional embodiment of this second embodiment, after weighted fusing the first enhancement mask, the second enhancement mask, the third enhancement mask, and the fourth enhancement mask to obtain the enhanced final output image, the method further includes:

[0088] A lightweight object detection model is used to detect objects in the final output image to determine semantic objects; the semantic objects are then highlighted at their edges to obtain an enhanced output image.

[0089] In this embodiment, the lightweight object detection model can be understood as an AI model for object recognition adapted to low-computing-power automotive SoC platforms. For example, it can be a lightweight structure that has been pruned for NanoDet and YOLOv5s. By simplifying model complexity and optimizing the inference process, it can accurately identify specific targets in images while ensuring real-time performance, avoiding excessive hardware resource consumption. Semantic targets can be understood as specific objects in the in-vehicle scene that have practical driving reference significance, including pedestrians, vehicles, warning signs, and license plates. Enhanced output image can be understood as a panoramic image after semantic target edge highlighting processing of the final output image.

[0090] Specifically, the controller can input the final output image into a lightweight object detection model. The model accurately identifies semantic objects in the image through feature extraction, object localization, and classification. The controller can extract the edge contours of the semantic objects, determine the areas that need to be highlighted, and perform highlighting processing (such as increasing edge brightness or optimizing edge color saturation) to obtain an enhanced output image for the driver to view on the vehicle's infotainment system.

[0091] The technical solution of this invention, by employing a combination of lightweight AI models and image processing algorithms, has low hardware computing power requirements, making it possible to achieve intelligent image optimization on low-to-mid-range automotive platforms and meeting the strong cost reduction needs of the automotive industry. Through the lightweight AI model's accurate environmental perception, the processing strategy is "scene-specific," providing specially optimized solutions for scenarios where traditional algorithms perform poorly, such as rainy nights, strong light, and low light, significantly improving the usability and safety of the image. Through parallel processing and a dynamic resource allocation strategy based on current computing power performance indicators, the system maintains stable real-time performance even in complex automotive environments, meeting the stringent low-latency requirements of driver assistance systems. The final output image is clearer and richer in detail, effectively reducing the visual stress on drivers in adverse weather and lighting conditions, and lowering the risk of accidents.

[0092] Example 3

[0093] Figure 7 This is a schematic diagram of the structure of an image processing device provided in Embodiment 3 of the present invention. Figure 7 As shown, the device includes:

[0094] Information acquisition module 71 is used to acquire images from multiple cameras and current computing power performance indicators;

[0095] The image determination module 72 is used to determine a panoramic image containing multi-granularity semantic annotations based on the multi-camera images, the current computing power performance indicators, and the lightweight AI model. The multi-granularity semantic annotations include global scene labels and local region masks.

[0096] The image processing module 73 is used to process the panoramic image according to the global scene label and the local area mask to obtain the enhanced final output image.

[0097] The technical solution of this invention involves acquiring images from multiple cameras and current computing power performance metrics; determining a panoramic image with multi-granularity semantic annotations based on the images, current computing power performance metrics, and a lightweight AI model. These multi-granularity semantic annotations include global scene labels and local region masks; and processing the panoramic image based on the global scene labels and local region masks to obtain an enhanced final output image. Dynamic resource allocation is performed using the current computing power performance metrics, and the lightweight AI model accurately perceives the scene in the image, forming a panoramic image with multi-granularity semantic annotations. The panoramic image is processed using a matching algorithm, enabling dynamic selection of the enhancement algorithm and differentiated processing of local problem areas. The final output image is clearer, more detailed, and reduces the risk of accidents.

[0098] Furthermore, the image determination module 72 includes:

[0099] The first determining unit is used to determine image processing parameters based on the current computing power performance index and the set threshold.

[0100] The second determining unit is used to preprocess the multi-camera images based on the image processing parameters to obtain an intermediate image;

[0101] The third determining unit is used to perform panoramic fusion processing on the intermediate image to obtain an initial panoramic image;

[0102] The fourth determining unit is used to input the initial panoramic image into a lightweight AI model to obtain a panoramic image containing multi-granular semantic annotations.

[0103] Specifically, the first determining unit is used for:

[0104] If the SOC load rate in the current computing power performance index is greater than the first set threshold and less than or equal to the second set threshold, then the preset target resolution will be used as the image processing parameter.

[0105] If the SOC load rate is greater than the second set threshold, then the target region of interest index is used as the image processing parameter;

[0106] The average inference time across multiple historical frames in the current computing power performance metrics is determined using a sliding window.

[0107] Based on the average inference time, the image frame processing rate is determined and used as the image processing parameter.

[0108] The lightweight AI model includes: a preprocessing module, three depthwise separable convolutional layers, a channel shuffling module, a lightweight feature pyramid, and a feature post-processing module.

[0109] Furthermore, the global scene labels include daytime and sunny days, nighttime, rainy nights, and backlighting; the local area mask includes overexposed areas, underexposed areas, raindrop areas, and smudge areas; correspondingly, the image processing module 73 is specifically used for:

[0110] If the global scene label is the night or the local area mask is the underexposed area, then the panoramic image and / or the underexposed area are processed by the first algorithm to obtain the first enhancement mask;

[0111] If the global scene label is the rainy night or the local area mask is the raindrop area, then the panoramic image and / or the raindrop area are processed by the second algorithm to obtain the second enhanced mask;

[0112] If the global scene label is the backlight or the local area mask is the overexposed area, then the panoramic image and / or the raindrop area are processed by the third algorithm to obtain the third enhancement mask;

[0113] If the local area mask is the stained area, then the stained area is processed by the fourth algorithm to obtain the fourth enhanced mask;

[0114] The first enhancement mask, the second enhancement mask, the third enhancement mask, and the fourth enhancement mask are weighted and fused to obtain the enhanced final output image.

[0115] The first algorithm, the second algorithm, the third algorithm, and the fourth algorithm are executed in a parallel pipeline manner through a multi-core heterogeneous architecture, which includes a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU).

[0116] Optionally, the device may also include:

[0117] The target detection module is used to perform target detection on the final output image by using a lightweight target detection model after the first enhancement mask, the second enhancement mask, the third enhancement mask and the fourth enhancement mask are weighted and fused to obtain the enhanced final output image, thereby determining the semantic target in the final output image;

[0118] The semantic target is highlighted at the edges to obtain an enhanced output image.

[0119] The image processing apparatus provided in the embodiments of the present invention can execute the image processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0120] Example 4

[0121] Figure 8 This is a structural schematic diagram of a vehicle provided in Embodiment 4 of the present invention, as shown below. Figure 8 As shown, the vehicle includes a controller 81, a memory 82, an input device 83, and an output device 84; the number of controllers 81 in the vehicle can be one or more. Figure 8 Taking a controller 81 as an example; the controller 81, memory 82, input device 83, and output device 84 in the vehicle can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0122] The memory 82, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the image processing method in this embodiment of the invention (e.g., information acquisition module 71, image determination module 72, and image processing module 73 in the image processing device). The controller 81 executes various functional applications and data processing of the vehicle by running the software programs, instructions, and modules stored in the memory 82, thereby realizing the above-described image processing method.

[0123] The memory 82 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 82 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 82 may further include memory remotely configured relative to the controller 81, which can be connected to the vehicle via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0124] Input device 83 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the cloud platform. Output device 84 may include display devices such as a display screen.

[0125] Example 5

[0126] Embodiment 5 of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a controller, are used to perform an image processing method, including:

[0127] Acquire images from multiple cameras and current computing power performance indicators;

[0128] Based on the multi-camera images, the current computing power performance indicators, and the lightweight AI model, a panoramic image containing multi-granularity semantic annotations is determined. The multi-granularity semantic annotations include global scene labels and local region masks.

[0129] The panoramic image is processed based on the global scene label and the local region mask to obtain the enhanced final output image.

[0130] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0131] It is worth noting that in the embodiments of the above image processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of the present invention.

[0132] In one embodiment, the present invention further includes a computer program product, which includes a computer program that, when executed by a processor, implements the transaction rate limiting method of any embodiment of the present invention.

[0133] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. An image processing method, characterized in that, include: Acquire images from multiple cameras and current computing power performance indicators; Based on the multi-camera images, the current computing power performance indicators, and the lightweight AI model, a panoramic image containing multi-granularity semantic annotations is determined. The multi-granularity semantic annotations include global scene labels and local region masks. The panoramic image is processed based on the global scene label and the local region mask to obtain the enhanced final output image.

2. The method according to claim 1, characterized in that, The step of determining a panoramic image containing multi-granularity semantic annotations based on the multi-camera images, the current computing power performance indicators, and the lightweight AI model includes: Based on the current computing power performance indicators and the set threshold, determine the image processing parameters; The images from the multiple cameras are preprocessed based on the image processing parameters to obtain an intermediate image; The intermediate image is subjected to panoramic fusion processing to obtain an initial panoramic image; The initial panoramic image is input into a lightweight AI model to obtain a panoramic image with multi-granular semantic annotations.

3. The method according to claim 2, characterized in that, The step of determining image processing parameters based on the current computing power performance indicators and a set threshold includes: If the SOC load rate in the current computing power performance index is greater than the first set threshold and less than or equal to the second set threshold, then the preset target resolution will be used as the image processing parameter. If the SOC load rate is greater than the second set threshold, then the target region of interest index is used as the image processing parameter; The average inference time across multiple historical frames in the current computing power performance metrics is determined using a sliding window. Based on the average inference time, the image frame processing rate is determined and used as the image processing parameter.

4. The method according to claim 1, characterized in that, The lightweight AI model includes: a preprocessing module, three depthwise separable convolutional layers, a channel shuffling module, a lightweight feature pyramid, and a feature post-processing module.

5. The method according to claim 1, characterized in that, The global scene labels include daytime (sunny), nighttime, rainy night, and backlighting. The local area mask includes overexposed areas, underexposed areas, raindrop areas, and smudged areas. Correspondingly, the process of processing the panoramic image based on the global scene labels and the local area mask to obtain the enhanced final output image includes: If the global scene label is the night or the local area mask is the underexposed area, then the panoramic image and / or the underexposed area are processed by the first algorithm to obtain the first enhancement mask; If the global scene label is the rainy night or the local area mask is the raindrop area, then the panoramic image and / or the raindrop area are processed by the second algorithm to obtain the second enhanced mask; If the global scene label is the backlight or the local area mask is the overexposed area, then the panoramic image and / or the raindrop area are processed by the third algorithm to obtain the third enhancement mask; If the local area mask is the stained area, then the stained area is processed by the fourth algorithm to obtain the fourth enhanced mask; The first enhancement mask, the second enhancement mask, the third enhancement mask, and the fourth enhancement mask are weighted and fused to obtain the enhanced final output image.

6. The method according to claim 5, characterized in that, The first algorithm, the second algorithm, the third algorithm, and the fourth algorithm are executed in a parallel pipeline manner through a multi-core heterogeneous architecture, which includes a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU).

7. The method according to claim 1, characterized in that, After performing weighted fusion on the first enhancement mask, the second enhancement mask, the third enhancement mask, and the fourth enhancement mask to obtain the enhanced final output image, the method further includes: A lightweight object detection model is used to detect objects in the final output image to determine the semantic objects in the final output image. The semantic target is highlighted at the edges to obtain an enhanced output image.

8. An image processing apparatus, characterized in that, include: The information acquisition module is used to acquire images from multiple cameras and current computing power performance indicators; The image determination module is used to determine a panoramic image containing multi-granularity semantic annotations based on the multi-camera images, the current computing power performance indicators, and the lightweight AI model. The multi-granularity semantic annotations include global scene labels and local region masks. The image processing module is used to process the panoramic image based on the global scene label and the local region mask to obtain the enhanced final output image.

9. A vehicle, characterized in that, The vehicles include: At least one controller; and A memory communicatively connected to the at least one controller; wherein, The memory stores a computer program that can be executed by the at least one controller, the computer program being executed by the at least one controller to enable the at least one controller to perform the image processing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause the controller to execute the image processing method according to any one of claims 1-7.

Images