Image processing method and device, electronic equipment and medium
By performing Gaussian filtering and correcting parameter processing on the camera images of autonomous driving vehicles, the problem of insufficient or excessive image exposure under different lighting conditions is solved, and the accuracy and stability of the visual perception system are improved.
Patent Information
- Application Number
- CN202510240346.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-28
- Publication Date
- 2025-06-17
AI Technical Summary
Cameras in autonomous driving vehicles are easily affected under different lighting conditions, resulting in insufficient or excessive image exposure, which in turn affects the accuracy and stability of the visual perception system.
By acquiring the image to be processed and filtering it Gaussianly, an exposure adjustment coefficient and correction parameter diagram are obtained, and the image is corrected based on these parameters to obtain the processed enhanced image.
Two-way exposure correction of the image is achieved, and the accuracy and stability of the visual perception system are improved, especially under different lighting conditions.
Smart Images
Figure CN120163749A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence, and in particular, to the fields of image processing, deep learning, and autonomous driving technology. Specifically, it relates to an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Art
[0002] Artificial intelligence is a discipline that studies how to make a computer simulate certain human thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It has both hardware-level technologies and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, and knowledge graph technology.
[0003] With the rapid development of autonomous driving technology, the visual perception system has become an essential key component in autonomous driving vehicles. In practical applications, cameras are often affected by different lighting conditions, such as strong light, backlight, and shadows, resulting in under-exposed or over-exposed images, which in turn affect the accuracy and stability of the visual perception system. Summary of the Invention
[0004] The present disclosure provides an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product.
[0005] According to one aspect of the present disclosure, there is provided an image processing method, including: obtaining a first image to be processed; passing the first image through a Gaussian filter to obtain a second image; inputting the second image into a first network model to obtain an exposure adjustment coefficient; inputting the second image into a second network model to obtain at least one correction parameter map, wherein for each correction parameter map in the at least one correction parameter map, each correction parameter in the correction parameter map is used to correct the corresponding pixel point of the corresponding image; and correcting the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image.
[0006] According to another aspect of the present disclosure, there is provided a model training method, including: obtaining a first image to be processed and a preset label image, where the label image is a normally exposed image; passing the first image through a Gaussian filter to obtain a second image; inputting the second image into a first network model to obtain an exposure adjustment coefficient; inputting the second image into a second network model to obtain at least one correction parameter map, where for each correction parameter map in the at least one correction parameter map, each correction parameter in the correction parameter map is used to correct the corresponding pixel point of the corresponding image; correcting the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image; obtaining an exposure loss value through a preset loss function based on the enhanced image and the label image; and adjusting parameters of at least one of the first network model and the second network model based on the exposure loss value.
[0007] According to another aspect of the present disclosure, there is provided an image processing apparatus, including: a first acquisition unit configured to acquire a first image to be processed; a first Gaussian filtering unit configured to pass the first image through a Gaussian filter to obtain a second image; a first input unit configured to input the second image into a first network model to obtain an exposure adjustment coefficient; a second input unit configured to input the second image into a second network model to obtain at least one correction parameter map, where for each correction parameter map in the at least one correction parameter map, each correction parameter in the correction parameter map is used to correct the corresponding pixel point of the corresponding image; and a first calculation unit configured to correct the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image.
[0008] According to another aspect of the present disclosure, there is provided a model training apparatus, including: a second acquisition unit configured to acquire a first image to be processed and a preset label image, wherein the label image is an image with normal exposure; a second Gaussian filtering unit configured to obtain a processed image by passing the first image through a Gaussian filter; a third input unit configured to input the second image into a first network model to obtain an exposure adjustment coefficient; a fourth input unit configured to input the second image into a second network model to obtain at least one correction parameter map, wherein for each correction parameter map in the at least one correction parameter map, each correction parameter in the correction parameter map is used to correct the corresponding pixel point of the corresponding image; a second calculation unit configured to correct the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image; a third calculation unit configured to obtain an exposure loss value based on the enhanced image and the label image through a preset loss function; and a training unit configured to adjust parameters of at least one of the first network model and the second network model based on the exposure loss value.
[0009] According to another aspect of the present disclosure, there is provided an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the method described in the present disclosure.
[0010] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method described in the present disclosure.
[0011] According to another aspect of the present disclosure, there is provided a computer program product including a computer program which, when executed by a processor, implements the method described in the present disclosure.
[0012] According to another aspect of the present disclosure, there is provided a vehicle including the electronic device described in the present disclosure.
[0013] According to one or more embodiments of the present disclosure, by adding a Gaussian filtering process, high-frequency information interference on subsequent model output results can be effectively avoided; and through the action of the exposure adjustment coefficient and the correction parameter map, two-way exposure correction can be achieved, which can further help improve the accuracy and stability of the visual perception system.
[0014] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. Description of the Drawings
[0015] The accompanying drawings exemplarily illustrate embodiments and form a part of the specification, and are used together with the written description of the specification to explain the exemplary embodiments of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. In all the drawings, the same reference numerals refer to similar but not necessarily identical elements.
[0016] Figure 1 A schematic diagram of an exemplary system in which various methods described herein can be implemented according to an embodiment of the present disclosure is shown;
[0017] Figure 2 A flowchart of an image processing method according to an embodiment of the present disclosure is shown;
[0018] Figure 3 A schematic diagram of an image processing method according to an embodiment of the present disclosure is shown;
[0019] Figure 4 A flowchart of a model training method according to an embodiment of the present disclosure is shown;
[0020] Figure 5 A structural block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown;
[0021] Figure 6 A structural block diagram of a model training apparatus according to an embodiment of the present disclosure is shown; and
[0022] Figure 7 A structural block diagram of an exemplary electronic device capable of implementing the embodiments of the present disclosure is shown. Detailed Embodiments
[0023] The following describes exemplary embodiments of the present disclosure in conjunction with the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0024] In the present disclosure, unless otherwise specified, the terms "first", "second", etc. are used to describe various elements and do not intend to limit the positional relationship, timing relationship, or importance relationship of these elements. Such terms are only used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, and in certain cases, based on the context description, they may also refer to different instances.
[0025] In the description of the various examples in this disclosure, the terms used are for the purpose of describing specific examples only and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure encompasses any one of the listed items and all possible combinations.
[0026] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
[0027] Figure 1 FIG. shows a schematic diagram of an exemplary system 100 in which the various methods and apparatuses described herein can be implemented according to an embodiment of the present disclosure. Referring Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 that couple the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more applications.
[0028] In an embodiment of the present disclosure, the server 120 can run one or more services or software applications that enable methods for performing image processing or model training.
[0029] In certain embodiments, the server 120 can also provide other services or software applications, which can include non-virtual environments and virtual environments. In certain embodiments, these services can be provided as web-based services or cloud services, for example, provided to users of the client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.
[0030] In Figure 1 the configuration shown, the server 120 can include one or more components that implement the functions performed by the server 120. These components can include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 can in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that various different system configurations are possible, which can be different from the system 100. Therefore, Figure 1 is an example of a system for implementing the various methods described herein and is not intended to be limiting.
[0031] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to obtain images to be processed. The client devices can provide an interface that enables the users of the client devices to interact with the client devices. The client devices can also output information to the users via this interface. Although Figure 1 only six client devices are depicted, those skilled in the art will be able to understand that the present disclosure can support any number of client devices.
[0032] Client devices 101, 102, 103, 104, 105, and / or 106 can include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices, etc. These computer devices can run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux, or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT WindowsMobile OS, iOS, Windows Phone, Android. Portable handheld devices can include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices can include head-mounted displays (such as smart glasses) and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices, etc. The client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.
[0033] Network 110 can be any type of network well-known to those skilled in the art, which can support data communication using any one of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, token ring, wide area network (WAN), the Internet, virtual network, virtual private network (VPN), intranet, extranet, blockchain network, public switched telephone network (PSTN), infrared network, wireless network (such as Bluetooth, WIFI), and / or any combination of these and / or other networks.
[0034] Server 120 may include one or more general-purpose computers, dedicated server computers (such as PC (Personal Computer) servers, UNIX servers, midrange servers), blade servers, mainframes, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (such as one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices of the server). In various embodiments, Server 120 may run one or more services or software applications that provide the functions described below.
[0035] The computing units in Server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. Server 120 may also run any one of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.
[0036] In some embodiments, Server 120 may include one or more applications to analyze and combine data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.
[0037] In some embodiments, Server 120 may be a server of a distributed system, or a server incorporating a blockchain. Server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, which solves the defects of difficult management and weak business scalability existing in traditional physical hosts and virtual private server (VPS, Virtual Private Server) services.
[0038] System 100 may also include one or more databases 130. In certain embodiments, these databases can be used to store data and other information. For example, one or more of the databases 130 can be used to store information such as image files. The databases 130 can reside in various locations. For example, the databases used by the server 120 can be local to the server 120, or can be remote from the server 120 and can communicate with the server 120 via a network-based or dedicated connection. The databases 130 can be of different types. In certain embodiments, the databases used by the server 120 can be, for example, relational databases. One or more of these databases can store, update, and retrieve data to and from the databases in response to commands.
[0039] In certain embodiments, one or more of the databases 130 can also be used by an application to store application data. The databases used by the application can be different types of databases, such as key-value repositories, object repositories, or conventional repositories supported by a file system.
[0040] Figure 1 System 100 can be configured and operated in various ways to enable the application of the various methods and apparatuses described in this disclosure.
[0041] With the rapid development of autonomous driving technology, the visual perception system has become an essential and crucial component in autonomous vehicles. In practical applications, cameras are often affected by different lighting conditions, such as strong light, backlight, shadows, etc., resulting in under-exposed or over-exposed images, which in turn affect the accuracy and stability of the visual perception system.
[0042] Therefore, according to an embodiment of the present disclosure, an image processing method is provided. Figure 2 A flowchart of an image processing method according to an embodiment of the present disclosure is shown, as Figure 2 shown, method 200 includes: obtaining a first image to be processed (step 210); passing the first image through a Gaussian filter to obtain a second image (step 220); inputting the second image into a first network model to obtain an exposure adjustment coefficient (step 230); inputting the second image into a second network model to obtain at least one correction parameter map, wherein for each correction parameter map in the at least one correction parameter map, each correction parameter in the correction parameter map is used to correct the corresponding pixel point of the corresponding image (step 240); and correcting the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image (step 250).
[0043] According to an embodiment of the present disclosure, by adding a Gaussian filtering process, high-frequency information interference on subsequent model output results can be effectively avoided; and through the effects of the exposure adjustment coefficient and the correction parameter map, two-way exposure correction can be achieved, which can further help improve the accuracy and stability of the visual perception system.
[0044] According to some embodiments, the first network model includes a convolutional neural network model. Exemplarily, the first network model can be implemented by a multi-layer convolutional neural network to obtain an exposure adjustment coefficient Ev. Through this exposure adjustment coefficient Ev, overexposed images can be corrected.
[0045] It can be understood that the first network model can also be implemented by other network models to obtain corresponding exposure adjustment coefficients, which are not limited herein.
[0046] According to some embodiments, correcting the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image includes: based on the first correction parameter map in the at least one correction parameter map, using the correction parameters in the first correction parameter map and the exposure adjustment coefficient to correct corresponding pixel points in the first image to obtain a corrected image; in response to determining that there is a next correction parameter map of the first correction parameter map in the at least one correction parameter map, using the next correction parameter map as the new first correction parameter map and the corrected image as the new first image, and iteratively operating to correct corresponding pixel points in the new first image until all of the at least one correction parameter map have participated in the correction operation; and obtaining the image obtained after correction by the at least one correction parameter map as the enhanced image.
[0047] Specifically, each correction parameter map in the at least one correction parameter map is used in turn through iterative operations to perform pixel-level correction on the first image to be corrected. The corrected image obtained after each correction operation is used as the new first image to be corrected in the next iterative operation until all of the at least one correction parameter map have participated in the correction operation.
[0048] According to some embodiments, the enhanced image is obtained based on the following formula:
[0049] LE n (x) = Ev * [LE n-1 (x) + A n (x)LE n-1 (x)(1 - LE n-1 (x))]
[0050] where A n(x) represents the first correction parameter map in the at least one correction parameter map in the n-th iteration operation, where n = 1, 2, …, N, N is the number of correction parameter maps in the at least one correction parameter map, and x is the pixel coordinate of the image. Among them, when n = 1, LE0(x) represents the first image before correction, and when n = 2, …, N, LE n-1 (x) represents the corrected image obtained after the (n - 1)-th iteration operation. When n = N, LE N (x) represents the enhanced image, where Ev is the exposure adjustment coefficient. Specifically, the set of correction parameter maps is also called a Light-Enhancement Curve (LE-curve), and the curve can be described by the following formula:
[0051] LE(I(x), α) = I(x) + αI(x)(1 - I(x))
[0052] Among them, x is the pixel coordinate, α is the learnable parameter, usually α ∈ [-1, 1]; I(x) is the input image, and LE(I(x), α) is the enhanced image of the input image. In order to fit a more complex curve, the above function can be iteratively nested multiple times, that is:
[0053] LE1(x) = I(x) + α1I(x)(1 - I(x))
[0054] LE2(x) = LE1(x) + α2LE1(x)(1 - LE1(x))
[0055] LE n (x) = LE n-1 (x) + α n LE n-1 (x)(1 - LE n-1 (x))
[0056] Through iterative nesting, a higher-order function can be represented. In each round of iteration, there is a new parameter α n . However, it is usually not desirable to use the same light-enhancement function for each pixel. For example, if there is a lighted area in the image, then the pixel values in this area do not need to be changed. Therefore, each pixel should have an independent α. Therefore, the final light-enhancement function is:
[0057] LE n (x) = LE n-1 (x) + A n (x)LE n-1 (x)(1 - LE n-1 (x))
[0058] That is, α corresponding to each pixel nComposing the A n (x), which is the correction parameter map corresponding to the nth iteration operation.
[0059] According to some embodiments, the second network model includes a Deep Curve Estimation network model (DCE-Net). When the input of the Deep Curve Estimation network model is the first low-light image, it can output at least one correction parameter map for correcting the first image. That is, it takes the low-light image as the input and generates a high-order curve as its output. Thus, the problem of light enhancement is transformed into the problem of estimating a specific image curve using a deep network.
[0060] In some examples, the deep curve estimation model can be a Zero-Reference Deep Curve Estimation model (Zero-DCE). Zero-DCE uses a lightweight network to predict a pixel-level, high-order curve. Moreover, Zero-DCE has an extremely fast inference speed, better brightening effect, and faster training speed.
[0061] The Zero-DCE model is a neural network algorithm only for low-light image processing and enhancement, and it is usually difficult for network models to achieve overexposure correction of images at low cost. Therefore, through the method described in the present disclosure, Zero-Dec can not only maintain its original characteristics but also correct overexposed images.
[0062] In some embodiments, the second network model can also be implemented based on a U-Net structure network, which is not limited herein.
[0063] According to some embodiments, adjusting the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image includes: based on the exposure adjustment coefficient and the at least one correction parameter map, using the GPU to correct the first image to obtain a processed enhanced image.
[0064] In some examples, the final result restores the full image through the GPU, so that the system load for completing the exposure correction task is extremely low, which is particularly suitable for multi-channel image processing tasks in the fields of autonomous driving and assisted driving.
[0065] In some examples, applying a lightening curve (LE-Curve) to the RGB channels (instead of just the illumination channel) of the input image can better maintain the inherent color and avoid overfitting.
[0066] According to some embodiments, the method according to the present disclosure may further include: separating the first image by color channels to obtain a plurality of third images after separation. The step of obtaining the second image by filtering out high-frequency information from the first image through a Gaussian filter includes: separately passing the plurality of third images through the Gaussian filter to obtain a plurality of fourth images corresponding one-to-one to the plurality of third images and having high-frequency information filtered out; and combining the plurality of fourth images to obtain the second image.
[0067] Specifically, separating the first image by color channels (spilt operation) can obtain the images of the three RGB channels after separation, i.e., the third images. Separately passing the three third images through a Gaussian filtering operation to filter out high-frequency information to obtain a low-frequency image. The filtered low-frequency images are then passed through a merging layer (i.e., concat operation) to obtain the second image.
[0068] In some examples, the Gaussian filter can be implemented only through a convolutional neural network. For example, an N-layer Gaussian filtering process is implemented through a convolutional neural network to obtain a low-frequency image with high-frequency information filtered out. Since it is necessary to correct the exposure of the image, and the exposure is low-frequency information, adding a Gaussian filtering process can avoid high-frequency information interfering with the subsequent model output results.
[0069] It can be understood that the Gaussian filtering process can also be implemented in other ways, which is not limited herein.
[0070] Figure 3 A schematic diagram of an image processing method according to an embodiment of the present disclosure is shown. As Figure 3 shown, the input image (i.e., the image to be processed) is separated by channels through a separation layer (spilt operation) to obtain an RGB three-channel image; then the separated images are separately input into a Gaussian filter (GaussFilter) to obtain filtered three-channel low-frequency images; the three-channel low-frequency images are merged by channels through a merging layer (concat operation) to obtain a merged image. The merged image passes through a CNN network (convolutional neural network) and a DCE network respectively to obtain an exposure adjustment coefficient Ev and a multi-layer correction parameter map α. Finally, the GPU can sample the original input image and perform calculations based on the exposure adjustment coefficient and the correction parameter map to obtain the restored enhanced image.
[0071] According to an embodiment of the present disclosure, as Figure 4As shown, a model training method 400 is also provided, including: obtaining a first image to be processed and a preset labeled image, where the labeled image is a normally exposed image (step 410); passing the first image through a Gaussian filter to obtain a second image (step 420); inputting the second image into a first network model to obtain an exposure adjustment coefficient (step 430); inputting the second image into a second network model to obtain at least one correction parameter map, where for each correction parameter map in the at least one correction parameter map, each correction parameter in the correction parameter map is used to correct the corresponding pixel point of the corresponding image (step 440); correcting the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image (step 450); obtaining an exposure loss value based on the enhanced image and the labeled image through a preset loss function (step 460); adjusting parameters of at least one of the first network model and the second network model based on the exposure loss value (step 470).
[0072] It can be understood that the loss function plays a crucial role in the design of the low-light enhancement model. It is a key indicator for evaluating the model performance. By measuring the difference between the model output and the real image, it guides the model to continuously optimize during the training process. In the low-light image enhancement task, the selection and design of the loss function directly affect the model's ability to restore image details and the improvement of the overall quality. Therefore, during the model design process, it is necessary to select an appropriate loss function according to the specific task requirements and image characteristics to ensure that the model can effectively enhance low-light images.
[0073] For example, the preset loss function may include, but is not limited to, mean absolute error (MAE), SSIM loss function, smoothness loss, color loss, etc., and is not limited here.
[0074] In some examples, the preset loss function may be, for example, exposure loss. Therefore, the loss function can be constructed by calculating the difference in gray mean values between the enhanced image and the labeled image. Generally, the image can be divided into blocks for calculation and then the total error can be obtained, which can suppress areas with poor exposure and make them reach a better exposure level.
[0075] According to some embodiments, it further includes: separating the first image by color channels to obtain multiple separated third images. Passing the first image through a Gaussian filter to obtain a second image with high-frequency information filtered out includes: passing the multiple third images through a Gaussian filter respectively to obtain multiple fourth images with high-frequency information filtered out corresponding to the multiple third images one by one; and combining the multiple fourth images to obtain the second image.
[0076] According to some embodiments, adjusting the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image includes: correcting corresponding pixel points in the first image based on a first correction parameter map in the at least one correction parameter map, using the correction parameters in the first correction parameter map and the exposure adjustment coefficient, to obtain a corrected image; in response to determining that there is a next correction parameter map of the first correction parameter map in the at least one correction parameter map, taking the next correction parameter map as the new first correction parameter map and the corrected image as the new first image, and correcting the corresponding pixel points in the new first image through iterative operations until all of the at least one correction parameter maps have participated in the correction operation; and obtaining the image obtained after correction through the at least one correction parameter map as the enhanced image.
[0077] Specifically, each correction parameter map in the at least one correction parameter map is used in sequence through iterative operations to perform pixel-level correction on the first image to be corrected. The corrected image obtained after each correction operation is used as the new first image to be corrected in the next iterative operation until all of the at least one correction parameter maps have participated in the correction operation.
[0078] According to some embodiments, the enhanced image LE is obtained based on the following formula n (x):
[0079] LE n (x) = Ev * [LE n-1 (x) + α n (x)LE n-1 (x)(1 - LE n-1 (x))]
[0080] where, A n (x) represents the first correction parameter map in the at least one correction parameter map in the nth iterative operation, where n = 1, 2,..., N, and N is the number of correction parameter maps in the at least one correction parameter map, and x is the pixel point coordinate of the image. When n = 1, LE0(x) represents the first image before correction. When n = 2,..., N, LE n-1 (x) represents the corrected image obtained after the (n - 1)th iterative operation. When n = N, LE n (x) represents the enhanced image, where Ev is the exposure adjustment coefficient.
[0081] According to some embodiments, the first network model includes a convolutional neural network model.
[0082] According to some embodiments, the second network model includes a Deep Curve Estimation network model (DCE-Net). In some examples, the deep curve estimation model can be a Zero-DCE (Zero-reference Deep Curve Estimation) model. Zero-DCE uses a lightweight network to predict a pixel-level, high-order curve. Moreover, Zero-DCE has an extremely fast inference speed, and its brightening effect and training speed are also better.
[0083] In the present disclosure, the model trained by the model training method described in any one of the above embodiments can be used to implement the image processing method described in any one of the embodiments of the present disclosure.
[0084] Here, for the embodiments for implementing the model training method and the embodiments for implementing the image processing method, the corresponding operations are similar and will not be elaborated here.
[0085] According to an embodiment of the present disclosure, as Figure 5 shown, there is also provided an image processing apparatus 500, including: a first acquisition unit 510 configured to acquire a first image to be processed; a first Gaussian filtering unit 520 configured to pass the first image through a Gaussian filter to obtain a second image; a first input unit 530 configured to input the second image into a first network model to obtain an exposure adjustment coefficient; a second input unit 540 configured to input the second image into a second network model to obtain at least one correction parameter map, wherein for each correction parameter map in the at least one correction parameter map, each correction parameter in the correction parameter map is used to correct the corresponding pixel point of the corresponding image; and a first calculation unit 550 configured to correct the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image.
[0086] According to some embodiments, it further includes a separation unit configured to: separate the first image by color channels to obtain a plurality of separated third images. The first Gaussian filtering unit includes: a unit configured to pass the plurality of third images through a Gaussian filter respectively to obtain a plurality of fourth images corresponding to the plurality of third images and filtering out high-frequency information; and a unit configured to merge the plurality of fourth images to obtain the second image.
[0087] According to some embodiments, the second network model includes a deep curve estimation network model.
[0088] According to some embodiments, the first computing unit 550 includes: a unit configured to correct corresponding pixel points in the first image based on a first correction parameter map among the at least one correction parameter map, by using correction parameters in the first correction parameter map and the exposure adjustment coefficient, to obtain a corrected image; a unit configured to, in response to determining that there is a next correction parameter map of the first correction parameter map in the at least one correction parameter map, use the next correction parameter map as a new first correction parameter map, use the corrected image as a new first image, and perform iterative operations to correct corresponding pixel points in the new first image until all of the at least one correction parameter map have participated in the correction operation; and a unit configured to obtain the image obtained by correction through the at least one correction parameter map as the enhanced image.
[0089] According to some embodiments, the enhanced image is obtained based on the following formula:
[0090] LE n (x) = Ev * [LE n-1 (x) + α n (x)LE n-1 (x)(1 - LE n-1 (x))]
[0091] where, A n (x) represents the first correction parameter map among the at least one correction parameter map in the n-th iterative operation, where n = 1, 2, …, N, N is the number of correction parameter maps in the at least one correction parameter map, and x is the pixel point coordinate of the image; where, when n = 1, LE0(x) represents the first image before correction, and when n = 2, …, N, LE n-1 (x) represents the corrected image obtained after the (n - 1)-th iterative operation, and when n = N, LE N (x) represents the enhanced image, where Ev is the exposure adjustment coefficient.
[0092] According to some embodiments, the first network model includes a convolutional neural network model.
[0093] According to some embodiments, the first computing unit 550 includes: a unit configured to use a GPU to correct the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image.
[0094] Here, the operations of the above units 510 to 550 of the image processing device 500 are respectively similar to the operations of steps 210 to 250 described above, and will not be elaborated here.
[0095] According to an embodiment of the present disclosure, as Figure 6As shown in the figure, a model training device 600 is further provided, including: a second acquisition unit 610 configured to acquire a first image to be processed and a preset labeled image, where the labeled image is an image with normal exposure; a second Gaussian filtering unit 620 configured to pass the first image through a Gaussian filter to obtain a second image; a third input unit 630 configured to input the second image into a first network model to obtain an exposure adjustment coefficient; a fourth input unit 640 configured to input the second image into a second network model to obtain at least one correction parameter map, where for each correction parameter map in the at least one correction parameter map, each correction parameter in the correction parameter map is used to correct the corresponding pixel point of the corresponding image; a second calculation unit 650 configured to correct the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image; a third calculation unit 660 configured to obtain an exposure loss value based on the enhanced image and the labeled image through a preset loss function; a training unit 670 configured to adjust parameters of at least one of the first network model and the second network model based on the exposure loss value.
[0096] Here, the operations of the above units 610-670 of the model training device 600 are respectively similar to the operations of steps 410-470 described above, and will not be elaborated here.
[0097] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.
[0098] According to an embodiment of the present disclosure, an electronic device, a readable storage medium, and a computer program product are further provided.
[0099] According to an embodiment of the present disclosure, a vehicle is further provided, including the electronic device according to the present disclosure.
[0100] With the rapid development of autonomous driving technology, the visual perception system has become an essential key component in autonomous driving vehicles. Through the embodiments of the present disclosure, not only can over-dark images be corrected, but over-exposed images can also be corrected. Finally, the entire image is restored through a processing device such as a GPU, so that the system load for completing the exposure correction task is extremely low, which is particularly suitable for multi-channel image processing tasks in the fields of autonomous driving and assisted driving.
[0101] Reference Figure 7, a block diagram of an electronic device 700 that can be a server or a client of the present disclosure will now be described. It is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0102] As Figure 7 shown, the electronic device 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0103] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 can be any type of device that can input information into the electronic device 700. The input unit 706 can receive input digital or character information, and generate key signal inputs related to the user settings and / or function controls of the electronic device, and can include but are not limited to a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone, and / or a remote control. The output unit 707 can be any type of device that can present information, and can include but are not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 can include but are not limited to magnetic disks and optical discs. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and can include but are not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0104] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 executes the various methods and processes described above, such as method 200 or 400. For example, in some embodiments, method 200 or 400 may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed onto the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the method 200 or 400 described above may be executed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute method 200 or 400 in any other suitable manner (e.g., by means of firmware).
[0105] Various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field-programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), application-specific standard products (ASSP), system-on-a-chip systems (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special or general-purpose programmable processor, receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting the data and instructions to the storage system, the at least one input device, and the at least one output device.
[0106] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program codes can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0107] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0108] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide for interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0109] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.
[0110] A computer system may include a client and a server. The client and the server are generally far away from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.
[0111] It should be understood that various forms of the processes shown above can be used, steps can be reordered, added or deleted. For example, the steps recited in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this is not limited herein.
[0112] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is only defined by the authorized claims and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, the steps can be executed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. Importantly, with the evolution of technology, many of the elements described herein can be replaced by equivalent elements that emerge after the present disclosure.
Claims
1. An image processing method, comprising: Acquire a first image to be processed; Pass the first image through a Gaussian filter to obtain a second image; Inputting the second image into the first network model to obtain an exposure adjustment coefficient; Inputting the second image into a second network model to obtain at least one correction parameter map, wherein, for each correction parameter map in the at least one correction parameter map, each correction parameter in the correction parameter map is used to correct the corresponding pixel point of the corresponding image; as well as The first image is corrected based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image.
2. The method of claim 1, further comprising: The first image is separated according to color channels to obtain a plurality of separated third images, and wherein: Passing the first image through a Gaussian filter to obtain a second image comprises: Passing the plurality of third images through Gaussian filters respectively to obtain a plurality of fourth images corresponding to the plurality of third images one by one and filtering out high-frequency information; and The plurality of fourth images are combined to obtain the second image.
3. The method of claim 1, wherein: Correcting the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image comprises: Based on a first correction parameter map among the at least one correction parameter map, correcting corresponding pixel points in the first image by using correction parameters in the first correction parameter map and the exposure adjustment coefficient to obtain a corrected image; In response to determining that there is a correction parameter map next to the first correction parameter map in the at least one correction parameter map, taking the next correction parameter map as a new first correction parameter map and taking the corrected image as a new first image, correcting corresponding pixels in the new first image through an iterative operation until all the at least one correction parameter map has participated in the correction operation; and An image obtained after correction by using the at least one correction parameter map is acquired as the enhanced image.
4. The method of claim 3, wherein: The enhanced image is obtained based on the following formula: THE n (x)=Ev*[LE n-1 (x)+A n (x)THE n-1 (x)(1-LE n-1 (x))] Among them, A n (x) represents the first correction parameter map in the at least one correction parameter map in the nth iteration operation, wherein n=1, 2, ..., N, N is the number of correction parameter maps in the at least one correction parameter map, x is the pixel coordinate of the image, When n=1, LE0(x) represents the first image before correction, and when n=2, ..., N, LE n-1 (x) represents the corrected image obtained after the n-1th iteration operation. When n=N, LE N (x) represents the enhanced image, wherein Ev is the exposure adjustment coefficient.
5. The method of claim 1, wherein: The first network model includes a convolutional neural network model, and the second network model includes a deep curve estimation network model.
6. The method of claim 1, wherein: Adjusting the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image comprises: Based on the exposure adjustment coefficient and the at least one correction parameter map, the first image is corrected by a GPU to obtain a processed enhanced image.
7. A model training method, comprising: Acquire a first image to be processed and a preset label image, wherein the label image is an image with normal exposure; Pass the first image through a Gaussian filter to obtain a second image; Inputting the second image into the first network model to obtain an exposure adjustment coefficient; Inputting the second image into a second network model to obtain at least one correction parameter map, wherein, for each correction parameter map in the at least one correction parameter map, each correction parameter in the correction parameter map is used to correct the corresponding pixel point of the corresponding image; Correcting the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image; Based on the enhanced image and the label image, obtaining an exposure loss value through a preset loss function; and A parameter of at least one of the first network model and the second network model is adjusted based on the exposure loss value.
8. The method of claim 7, further comprising: The first image is separated according to color channels to obtain a plurality of separated third images, and wherein: Passing the first image through a Gaussian filter to obtain a second image comprises: Passing the plurality of third images through Gaussian filters respectively to obtain a plurality of fourth images corresponding to the plurality of third images one by one and filtering out high-frequency information; and The plurality of fourth images are combined to obtain the second image.
9. The method of claim 7, wherein: Correcting the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image comprises: Based on a first correction parameter map among the at least one correction parameter map, correcting corresponding pixel points in the first image by using correction parameters in the first correction parameter map and the exposure adjustment coefficient to obtain a corrected image; In response to determining that there is a correction parameter map next to the first correction parameter map in the at least one correction parameter map, taking the next correction parameter map as a new first correction parameter map and taking the corrected image as a new first image, correcting corresponding pixels in the new first image through an iterative operation until all the at least one correction parameter map has participated in the correction operation; and An image obtained after correction by using the at least one correction parameter map is acquired as the enhanced image.
10. The method of claim 9, wherein: The enhanced image is obtained based on the following formula: THE n (x)=Ev*[LE n-1 (x)+A n (x)THE n-1 (x)(1-LE n-1 (x))] Among them, A n (x) represents the first correction parameter map in the at least one correction parameter map in the nth iteration operation, wherein n=1, 2, ..., N, N is the number of correction parameter maps in the at least one correction parameter map, x is the pixel coordinate of the image, When n=1, LE0(x) represents the first image before correction, and when n=2, ..., N, LE n-1 (x) represents the corrected image obtained after the n-1th iteration operation. When n=N, LE N (x) represents the enhanced image, wherein Ev is the exposure adjustment coefficient.
11. The method of claim 7, wherein: The first network model includes a convolutional neural network model, and the second network model includes a deep curve estimation network model.
12. An image processing device, comprising: A first acquisition unit, configured to acquire a first image to be processed; A first Gaussian filtering unit, configured to obtain a second image by passing the first image through a Gaussian filter; A first input unit, configured to input the second image into a first network model to obtain an exposure adjustment coefficient; a second input unit configured to input the second image into a second network model to obtain at least one correction parameter map, wherein, for each correction parameter map in the at least one correction parameter map, each correction parameter in the correction parameter map is used to correct the corresponding pixel point of the corresponding image; as well as The first calculation unit is configured to correct the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image.
13. A model training device, comprising: A second acquisition unit is configured to acquire a first image to be processed and a preset label image, wherein the label image is an image with normal exposure; A second Gaussian filtering unit is configured to obtain a passed image by passing the first image through a Gaussian filter; A third input unit is configured to input the second image into the first network model to obtain an exposure adjustment coefficient; a fourth input unit, configured to input the second image into a second network model to obtain at least one correction parameter map, wherein, for each correction parameter map in the at least one correction parameter map, each correction parameter in the correction parameter map is used to correct the corresponding pixel point of the corresponding image; a second calculation unit configured to correct the first image based on the exposure adjustment coefficient and the at least one correction parameter map to obtain a processed enhanced image; a third calculation unit, configured to obtain an exposure loss value through a preset loss function based on the enhanced image and the label image; and A training unit is configured to adjust parameters of at least one of the first network model and the second network model based on the exposure loss value.
14. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.
15. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.
16. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.
17. A vehicle comprising the electronic device according to claim 14.