Image processing methods, related devices, equipment and storage media

Abstract

The embodiment of the present application discloses an image processing method, related apparatus, device and storage medium, the method comprising: obtaining a bright channel image of a target image to be processed, and performing feature extraction on the bright channel image to obtain a bright channel detail map; obtaining a dark channel image of the target image, and performing feature extraction on the dark channel image to obtain a dark channel detail map; performing feature extraction on the target image to obtain a basic reconstruction map of the target image; generating a reconstructed image based on the dark channel detail map, the bright channel detail map and the basic reconstruction map. In the embodiment of the present application, a bright channel image and a dark channel image of the target image are first generated, and then a dark channel detail map and a bright channel detail map are generated, and the image is reconstructed by combining the dark channel detail map and the bright channel detail map, which enhances the reconstruction of details of underexposed and overexposed areas of the target image, expands the dynamic range of the image, and makes the generated reconstructed image have richer image details and higher image quality.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, related apparatus, equipment and storage medium. Background Art

[0002] Scenes in nature have a large dynamic range, often with bright and dark areas appearing at the same time. The dynamic range of ordinary digital cameras is only two to three levels, and the brightness range of the obtained digital images will be compressed, and the details of the bright and dark areas will be lost and the colors will be distorted. The distinct brightness effects in the original scene cannot be displayed, and it is impossible to accurately capture all the information in the real situation. Professional high-dynamic cameras are expensive, and the effect will also be limited by the hardware equipment conditions. Therefore, it is necessary to use high-dynamic imaging methods to process the images.

[0003] High-Dynamic Range imaging (HDRI) is mainly achieved through multi-exposure fusion. Its principle is to obtain multiple low dynamic range (LDR) images according to different exposure methods, and then select the best exposed pixels as the corresponding pixels in the output image according to the brightness information of the pixels in each LDR image, so as to retain the details of the bright and dark parts of the original image and present a wider dynamic range.

[0004] At present, traditional schemes such as weighted fusion and alignment-based fusion have many parameters, are difficult to debug, and have poor reconstruction effects on bright and dark areas. Summary of the invention

[0005] The present application provides an image processing method, related apparatus, device and storage medium, which enhance the reconstruction of details of underexposed and overexposed areas of a target image, expand the image dynamic range, and make the generated reconstructed image have richer image details and higher image quality.

[0006] In a first aspect, the present application provides an image processing method, the method comprising:

[0007] Acquire a bright channel image of the target image to be processed, and perform feature extraction on the bright channel image to obtain a bright channel detail image;

[0008] Acquire a dark channel image of the target image, and perform feature extraction on the dark channel image to obtain a dark channel detail image;

[0009] Performing feature extraction on the target image to obtain a basic reconstruction image of the target image;

[0010] A reconstructed image is generated according to the dark channel detail image, the bright channel detail image and the basic reconstruction image.

[0011] In a second aspect, the present application provides a model training method, the method comprising:

[0012] Acquire an image training set, wherein the image training set includes multiple groups of image training data, and each group of image training data includes a sample image and a target real image;

[0013] Inputting the sample images in the image training set into the initial image processing model for training, and outputting a predicted image;

[0014] Obtain the target real image corresponding to the sample image in each set of image training data;

[0015] A preset loss function is used to converge the predicted image and the target real image to obtain a trained image processing model. The preset loss function is used to calculate the total loss value of the initial image processing model, and the total loss value includes a bright channel loss value and / or a dark channel loss value. The bright channel loss value is the loss value of the predicted image and the target real image under the bright channel image, and the dark channel loss value is the loss value of the predicted image and the target real image under the dark channel image.

[0016] In a third aspect, the present application provides an image processing device, the device comprising:

[0017] A first acquisition module is used to acquire a bright channel image of a target image to be processed, and perform feature extraction on the bright channel image to obtain a bright channel detail image;

[0018] The second acquisition module acquires a dark channel image of the target image and performs feature extraction on the dark channel image to obtain a dark channel detail image;

[0019] A basic reconstruction module, used for extracting features from the target image to obtain a basic reconstruction image of the target image;

[0020] A generating module is used to generate a reconstructed image according to the dark channel detail image, the bright channel detail image and the basic reconstruction image.

[0021] In a fourth aspect, the present application provides a model training device, the device comprising:

[0022] A first acquisition module is used to acquire an image training set, wherein the image training set includes multiple groups of image training data, and each group of image training data includes a sample image and a target real image;

[0023] An output module, used for inputting the sample images in the image training set into the initial image processing model for training, and outputting a predicted image;

[0024] The second acquisition module is used to acquire the target real image corresponding to the sample image in each set of image training data;

[0025] A convergence module is used to converge the predicted image and the target real image using a preset loss function to obtain a trained image processing model, wherein the preset loss function is used to calculate the total loss value of the initial image processing model, wherein the total loss value includes a bright channel loss value and / or a dark channel loss value, wherein the bright channel loss value is the loss value of the predicted image and the target real image under the bright channel image, and the dark channel loss value is the loss value of the predicted image and the target real image under the dark channel image.

[0026] In a fifth aspect, the present application provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image processing method described in any one of the first aspect or the second aspect.

[0027] In a sixth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is loaded by a processor to execute the steps in the method as described in any one of the first aspect or the second aspect.

[0028] In the embodiment of the present application, an image processing method, related apparatus, device and storage medium are provided, which obtains a bright channel image of a target image to be processed and performs feature extraction on the bright channel image to obtain a bright channel detail map; obtains a dark channel image of the target image and performs feature extraction on the dark channel image to obtain a dark channel detail map; performs feature extraction on the target image to obtain a basic reconstruction map of the target image; and generates a reconstructed image based on the dark channel detail map, the bright channel detail map and the basic reconstruction map. In the embodiment of the present application, a bright channel image and a dark channel image of the target image are first generated, and then a dark channel detail map with more underexposed areas and a bright channel detail map with more overexposed areas are generated through the bright channel image and the dark channel image, and the image is reconstructed by combining the dark channel detail map and the bright channel detail map. Since the dark channel detail map and the bright channel detail map enhance the reconstruction of the details of the underexposed and overexposed areas of the target image, the dynamic range of the image is expanded, so that the generated reconstructed image has richer image details and higher image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 is a schematic diagram of the structure of an embodiment of an image processing system provided in an embodiment of the present application;

[0031] Figure 2 This is a flow chart of an embodiment of the image processing method in the embodiment of the present application;

[0032] Figure 3 is a schematic diagram of a network architecture of the image processing method provided in an embodiment of the present application;

[0033] Figure 4 It is a schematic diagram of an embodiment of the process of extracting features of a target image to obtain a basic reconstruction image of the target image provided in an embodiment of the present application;

[0034] Figure 5 is a schematic diagram of the structure of an embodiment of the image processing model in the embodiment of the present application;

[0035] Figure 6 is a schematic diagram of the structure of an embodiment of a feature extraction unit provided in an embodiment of the present application;

[0036] Figure 7 is a comparative schematic diagram of the dynamic range expansion results provided in the embodiments of the present application;

[0037] Figure 8 is a schematic diagram of the structure of an embodiment of an image processing device provided in an embodiment of the present application;

[0038] Fig. 9 It is a schematic diagram of the structure of an embodiment of the model training device provided in the embodiments of the present application;

[0039] Fig.10 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0041] In the following description, the specific embodiments of the present application will be described with reference to the steps and symbols performed by one or more computers, unless otherwise specified. Therefore, these steps and operations will be mentioned several times as being performed by a computer, and the computer execution referred to herein includes the operation of a computer processing unit by an electronic signal representing data in a structured form. This operation converts the data or maintains it at a location in the memory system of the computer, which can be reconfigured or otherwise change the operation of the computer in a manner familiar to testers in the field. The data structure maintained by the data is a physical location in the memory, which has specific characteristics defined by the data format. However, the principles of the present application are described in the above text, which does not represent a limitation, and testers in the field will understand that the various steps and operations described below can also be implemented in hardware.

[0042] The terms "module" or "unit" used herein may be considered as software objects executed on the computing system. The different components, modules, engines and services described herein may be considered as implementation objects on the computing system. The devices and methods described herein are preferably implemented in software, but may also be implemented in hardware, all within the scope of protection of this application.

[0043] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0044] The embodiments of the present application provide an image processing method, a related device and a storage medium.

[0045] See also Figure 1 , Figure 1 This is a schematic diagram of a scenario of an image processing system provided in an embodiment of the present application, and the image processing system may include an electronic device 100, and an image processing method device is integrated in the electronic device 100. In the embodiment of the present application, the electronic device 100 may be a terminal device or a server.

[0046] In the embodiment of the present application, when the electronic device 100 is a server, the server can be an independent server, or a server network or server cluster composed of servers. For example, the server described in the embodiment of the present application includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud server composed of a plurality of servers. Wherein, the cloud server is composed of a large number of computers or network servers based on cloud computing (Cloud Computing). In the embodiment of the present application, communication between the server and the client can be achieved by any communication method, including but not limited to, mobile communication based on the third generation partnership project (3rd Generation Partnership Project, 3GPP), long term evolution (Long Term Evolution, LTE), Worldwide Interoperability for Microwave Access (Worldwide Interoperability for Microwave Access, WiMAX), or computer network communication based on TCP / IP protocol suite (TCP / IP Protocol Suite, TCP / IP), user datagram protocol (User Datagram Protocol, UDP) protocol, etc.

[0047] It is understandable that when the electronic device 100 used in the embodiment of the present application is a terminal device, the terminal device may be a device including both receiving hardware and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. Such terminal devices may include: cellular or other communication devices having a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The specific electronic device 100 may be a desktop terminal or a mobile terminal, and the electronic device 100 may be one of a mobile phone, a tablet computer, a laptop computer, etc.

[0048] The terminal device involved in the embodiments of the present application may also refer to a device that provides voice and / or data connectivity to a user, a handheld device with a wireless connection function, or other processing devices connected to a wireless modem. For example, a mobile phone (or "cellular" phone) and a computer with a mobile terminal, for example, a portable, pocket-sized, handheld, computer-built-in or vehicle-mounted mobile device, which exchanges voice and / or data with a wireless access network. For example, a personal communication service (English full name: Personal Communication Service, English abbreviation: PCS) phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (Wireless Local Loop, English abbreviation: WLL) station, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA) and other devices.

[0049] Those skilled in the art will understand that Figure 1 The application environment shown in the figure is only one application scenario of the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or fewer electronic devices shown in, or electronic device network connection relationships, such as Figure 1 Only one electronic device is shown in the figure. It can be understood that the image processing system can also include one or more other electronic devices, or / and one or more other electronic devices connected to the electronic device 100 through a network, which is not specifically limited here.

[0050] In addition, if Figure 1 As shown, the image processing system may further include a memory 200 for storing data, such as storing a captured original image, a reconstructed image generated after image reconstruction, and the like.

[0051] It should be noted that Figure 1 The scene diagram of the image processing system shown is only an example. The image processing system and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the image processing system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0052] The solution provided in the embodiments of the present application involves technologies such as artificial intelligence (AI), computer vision (CV), and machine learning (ML), which are specifically described by the following embodiments:

[0053] Among them, AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0054] AI technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0055] CV is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify, track and measure targets, and further performs graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes adversarial disturbance generation, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0056] In the embodiments of the present application, an image processing method, related apparatus, device and storage medium are provided, which first generate a bright channel image and a dark channel image of a target image, then generate a dark channel detail map and a bright channel detail map, and reconstruct an image by combining the dark channel detail map and the bright channel detail map, thereby enhancing the detail reconstruction of underexposed and overexposed areas of the target image and expanding the image dynamic range, so that the generated reconstructed image has richer image details and higher image quality.

[0057] The following describes it in detail with reference to specific embodiments.

[0058] In this embodiment, the image processing method and apparatus will be described from the perspective of the image processing method and apparatus, which can be specifically integrated into the electronic device 100 .

[0059] The present application provides an image processing method, which includes: obtaining a bright channel image of a target image to be processed, and performing feature extraction on the bright channel image to obtain a bright channel detail image; obtaining a dark channel image of the target image, and performing feature extraction on the dark channel image to obtain a dark channel detail image; performing feature extraction on the target image to obtain a basic reconstruction image of the target image; and generating a reconstructed image according to the dark channel detail image, the bright channel detail image and the basic reconstruction image.

[0060] The following describes it in detail with reference to specific embodiments.

[0061] In the embodiments of the present application, description will first be made from the perspective of an image processing apparatus, which may be specifically integrated into a receiving device.

[0062] See also Figure 2 , Figure 2 The image processing method provided in the embodiment of the present application includes the following steps 201 to 204:

[0063] 201. Obtain a bright channel image of a target image to be processed, and perform feature extraction on the bright channel image to obtain a bright channel detail image.

[0064] Among them, the target image can be a low dynamic range (LDR) image obtained in the imaging process. Specifically, the target image can be one image or multiple images. For example, the target image includes one LDR image, or the target image includes multiple LDR images. The specific details are not limited here. Further, the multiple LDR images can be multiple LDR images obtained by different exposure methods. In this way, the multiple LDR images with multiple exposures can be reconstructed to obtain a high dynamic range (HDR) image, thereby improving the image reconstruction quality.

[0065] The inventor of the present application has found through research that in most highlighted images, local pixels always have a large value in a certain color channel, and this value represents the maximum light intensity in the area. Since the bright channel represents the maximum light intensity, it can be used to estimate the light distribution in non-highlight areas and capture more underexposure information. Therefore, in the embodiment of the present application, more underexposure information can be captured by calculating the bright channel image to generate a reconstructed image. Specifically, the bright channel image is an image that reflects the maximum light intensity in each color channel of the target image.

[0066] When the target image is multiple images, the bright channel image of each image is calculated separately. After the bright channel image of each image is obtained, they are spliced ​​according to the color channel dimension to obtain the bright channel image of the target image.

[0067] 202. Obtain a dark channel image of the target image, and perform feature extraction on the dark channel image to obtain a dark channel detail image.

[0068] After research, the inventors of the present application found that in most non-highlight images, local pixels always have a very small value in a certain color channel. This value represents the minimum light intensity in the area, and due to the influence of natural scene projection or color vividness, the dark channel of all non-highlight images should be close to 0. Since the dark channel represents the minimum light intensity, it can be used to estimate the light distribution in the highlight area and capture more exposure information. Therefore, in the embodiment of the present application, more exposure information can be captured by calculating the dark channel image to generate a reconstructed image. Specifically, the dark channel image is an image that reflects the minimum light intensity in each color channel of the target image.

[0069] Similarly, when the target image is multiple images, the dark channel image of each image is calculated separately. After the dark channel image of each image is obtained, they are spliced ​​according to the color channel dimension to obtain the dark channel image of the target image.

[0070] 203. Perform feature extraction on the target image to obtain a basic reconstruction image of the target image.

[0071] It should be noted that the manners of extracting features from the dark channel image of the target image, the bright channel image of the target image, and the target image in steps 201, 202, and 203 may be the same or different, and are not specifically limited here.

[0072] 204. Generate a reconstructed image according to the dark channel detail image, the bright channel detail image and the basic reconstruction image.

[0073] In the embodiment of the present application, an image processing method, related apparatus, device and storage medium are provided, which obtains a bright channel image of a target image to be processed and performs feature extraction on the bright channel image to obtain a bright channel detail map; obtains a dark channel image of the target image and performs feature extraction on the dark channel image to obtain a dark channel detail map; performs feature extraction on the target image to obtain a basic reconstruction map of the target image; and generates a reconstructed image based on the dark channel detail map, the bright channel detail map and the basic reconstruction map. In the embodiment of the present application, a bright channel image and a dark channel image of the target image are first generated, and then a dark channel detail map with more underexposed areas and a bright channel detail map with more overexposed areas are generated through the bright channel image and the dark channel image, and the image is reconstructed by combining the dark channel detail map and the bright channel detail map. Since the dark channel detail map and the bright channel detail map enhance the reconstruction of the details of the underexposed and overexposed areas of the target image, the dynamic range of the image is expanded, so that the generated reconstructed image has richer image details and higher image quality.

[0074] It should be noted that Figure 2 In the illustrated embodiment, the reconstructed image is generated by combining the dark channel image and the bright channel image. It is understandable that in some other embodiments of the present application, the reconstructed image may be generated by combining only the dark channel image and the bright channel image, which is explained below with examples.

[0075] For example, in one embodiment of the present application, the image processing method may include: obtaining a bright channel image of a target image to be processed, and performing feature extraction on the bright channel image to obtain a bright channel detail map; performing feature extraction on the target image to obtain a basic reconstruction map of the target image; and generating a reconstructed image based on the bright channel detail map and the basic reconstruction map. By reconstructing the image in combination with the bright channel detail map, the reconstruction of underexposed details of the target image is enhanced, the image dynamic range is expanded, and the generated reconstructed image has richer image details and higher image quality.

[0076] For another example, in another embodiment of the present application, the image processing method may include: obtaining a dark channel image of a target image to be processed, and performing feature extraction on the dark channel image to obtain a dark channel detail image; performing feature extraction on the target image to obtain a basic reconstruction image of the target image; and generating a reconstructed image based on the dark channel detail image and the basic reconstruction image. By reconstructing the image in combination with the dark channel detail image, the detail reconstruction of the overexposed area of ​​the target image is enhanced, the image dynamic range is expanded, and the generated reconstructed image has richer image details and higher image quality.

[0077] In the embodiments of the present application, based on the inventor's research on prior knowledge of bright channel images and dark channel images, the bright channel image of the target image and the dark channel image of the target image can be obtained based on different means, which are explained below with examples.

[0078] 1. Obtain the bright channel image of the target image to be processed

[0079] Obtaining the bright channel image of the target image to be processed in step 201 may further include: calculating the maximum channel pixel value in the target image, the maximum channel pixel value being the maximum value of the channel pixel values ​​of the target image in each color channel; taking each pixel in the target image as the first target pixel, and determining the maximum neighborhood pixel value of the first target pixel within the first target neighborhood range according to the maximum channel pixel value; determining the bright channel pixel value of the first target pixel according to the maximum neighborhood pixel value; after determining the bright channel pixel value corresponding to each pixel in the target image, obtaining the bright channel image of the target image. The maximum neighborhood pixel value is the neighborhood pixel value of the first target pixel with the largest value within the first target neighborhood range.

[0080] Among them, the channel pixel value is the pixel value of a pixel in a color channel in the target image, the maximum channel pixel value is the pixel value of a pixel in a color channel in the corresponding target image, and the pixel value of the pixel in the color channel is exactly the maximum value of the channel pixel values ​​of the target image in each color channel. The neighborhood pixel value with the largest value of the first target pixel in the first target neighborhood range can be the neighborhood pixel value with the largest value of the first target pixel in the first target neighborhood range of the color channel where the maximum channel pixel value is located, or it can be the neighborhood pixel value with the largest value of the first target pixel in the first target neighborhood range of the three color channels. The first target neighborhood range can be set according to the actual application scenario, and can be specifically divided according to the pixel point as a unit. For example, the first target neighborhood range can be a neighborhood range of n*n, where n is a positive integer. In a specific example of the present application, n can be 1, that is, the first target neighborhood range can be a neighborhood range of 1*1, representing that the neighborhood range can be the pixel point itself to retain more image details.

[0081] In one embodiment, the bright channel image of the target image to be processed can be obtained by using the following formula (1) for calculation:

[0082]

[0083] Among them I c represents each color channel of the color image, Ω(x) is the neighborhood pixel point within the first target neighborhood range centered on the pixel point x, and y represents other pixel points within the first target neighborhood range centered on the pixel point x.

[0084] In a specific example, taking the target image as an image, pixel 1 in the target image has a channel pixel value in the three color channels of R, G, and B, assuming it is (R1, G1, B1), and the channel pixel values ​​of pixel 1 in each color channel are (R1, 0, 0), (0, 0, B1) and (0, G1, 0), assuming R1>B1>G1, then the maximum channel pixel value of pixel 1 in each color channel is determined to be the channel pixel value (R1, 0, 0) of pixel 1 in the red channel. If the first target neighborhood range is 1*1, the maximum neighborhood pixel value is also the channel pixel value (R1, 0, 0) of the red channel. If the first target neighborhood range is 2*2, there are other pixels within the first target neighborhood range. The maximum channel pixel value (R1, 0, 0) is compared with the pixel values ​​of pixels in other neighborhood ranges, and the maximum channel pixel value within the first target neighborhood range is taken as the maximum neighborhood pixel value.

[0085] like Figure 3As shown, it is an example comparison diagram of the target image and the bright channel image and dark channel image obtained by the method of the present application. Obviously, the details of the underexposed area of ​​the bright channel image are clearer, which is more helpful for reconstructing the details of the dark area in the target image.

[0086] 2. Obtaining a dark channel image of the target image

[0087] Obtaining the dark channel image of the target image in step 202 may further include: calculating the minimum channel pixel value in the target image; taking each pixel in the target image as a second target pixel, and determining the minimum neighborhood pixel value of the second target pixel within the second target neighborhood range according to the minimum channel pixel value; determining the dark channel pixel value of the second target pixel according to the minimum neighborhood pixel value; after determining the dark channel pixel value corresponding to each pixel in the target image, obtaining the dark channel image of the target image.

[0088] Among them, the minimum channel pixel value is the pixel value of a pixel in a color channel corresponding to the target image, and the pixel value of the pixel in the color channel is exactly the minimum value of the channel pixel values ​​of the target image in each color channel. The neighborhood pixel value with the smallest value of the first target pixel in the second target neighborhood range can be the neighborhood pixel value with the smallest value of the first target pixel in the second target neighborhood range of the color channel where the minimum channel pixel value is located, or it can be the neighborhood pixel value with the smallest value of the first target pixel in the second target neighborhood range of the three color channels respectively.

[0089] It should be noted that the first target neighborhood range can be the same neighborhood range as the second target neighborhood range, or it can be a different neighborhood range. The second target neighborhood range can be set according to the actual application scenario, and can also be divided according to the pixel point as a unit. It is not limited here. In one example, the second target neighborhood range can be an n*n neighborhood range, where n is a positive integer. In a specific example of the present application, n can be 1, that is, the second target neighborhood range can be a 1*1 neighborhood range, which represents the neighborhood range can be the pixel point itself.

[0090] In one embodiment, the dark channel image of the target image can be calculated using the following formula (2):

[0091]

[0092] Among them I c represents each color channel of the color image, Ω(x) is the neighborhood pixel point within the second target neighborhood range centered on the pixel point x, and y represents other pixel points within the second target neighborhood range centered on the pixel point x.

[0093] In a specific example, taking the target image as an image, pixel 1 in the target image has a pixel value in the three color channels of R, G, and B, assuming (R1, G1, B1), and the channel pixel values ​​of pixel 1 in each color channel are (R1, 0, 0), (0, 0, B1) and (0, G1, 0), respectively. Assuming R1>B1>G1, the minimum channel pixel value of pixel 1 in each color channel is determined to be the channel pixel value of pixel 1 in the green channel (0, 0, G1). If the second target neighborhood range is 1*1, the minimum neighborhood pixel value is also the channel pixel value of the green channel (0, 0, G1). If the second target neighborhood range is 2*2, there are other pixels in the second target neighborhood range. The minimum channel pixel value (0, 0, G1) is compared with the pixel values ​​of pixels in other neighborhood ranges, and the minimum channel pixel value in the second target neighborhood range is taken as the minimum neighborhood pixel value.

[0094] like Figure 3 As shown, it is an example diagram comparing the target image with the bright channel image and the dark channel image obtained by the method of the present application. It can be seen that in the non-highlight area, the dark channel of the image is close to 0; in the highlight area, the dark channel image hierarchy is more obvious, which can provide rich detail information for the brightness component.

[0095] Further, such as Figure 4 As shown, in step 203, feature extraction is performed on the target image to obtain a basic reconstruction image of the target image, which may include the following steps 2031 to 2034:

[0096] 2031. Acquire a target image.

[0097] The target image may be directly acquired, that is, the target image is not processed in any way, or the target image may be acquired after basic image operations such as denoising that do not affect image details are performed, which is not specifically limited here.

[0098] 2032. Perform local feature extraction on the target image to obtain local features of the image.

[0099] Specifically, performing local feature extraction on the target image to obtain the local features of the image may further include: performing two dilated convolutions on the target image in sequence, wherein the dilation coefficient of the second dilated convolution layer is twice the dilation coefficient of the first dilated convolution, for example, the dilation coefficient of the first dilated convolution is 1, and the dilation coefficient of the second dilated convolution layer is 2.

[0100] 2033. Perform global feature extraction on the target image to obtain the global feature of the image.

[0101] Among them, performing global feature extraction on the target image to obtain the global feature of the image can further include: performing multi-layer pooling processing on the target image to obtain a first image feature; weighting the first image feature using a preset global average value to obtain a second image feature; upsampling the second image feature to the size of the target image to obtain the global feature of the image.

[0102] 2034. According to the target image, local features of the image and global features of the image are fused to obtain a basic reconstruction image of the target image.

[0103] Specifically, according to the target image, local features of the image and global features of the image are subjected to feature fusion to obtain a basic reconstruction map of the target image, which may further include: using a preset activation function to map the global features of the image to 0 to 1 as a fusion weight; performing point-to-point global scaling on the target image according to the fusion weight to obtain a fused feature map; performing feature fusion based on the fused feature map and local features of the image to obtain a basic reconstruction map of the target image.

[0104] It should be noted that the specific method of extracting features of the target image and obtaining the basic reconstruction image of the target image described in the above steps 2031 to 2034 also includes feature extraction for the bright channel image and the dark channel image in steps 201 and 202. The feature extraction method for the bright channel image and the dark channel image can refer to the feature extraction method for the target image described in the above steps 2031 to 2034, and the details are not repeated here.

[0105] In some embodiments of the present application, steps 201 to 204 are performed by inputting the target image to be processed into a neural network model, for example, by a trained image processing model. Specifically, the image processing method in the above embodiment can be applied to a trained image processing model.

[0106] Among them, the image processing model may include a bright channel image processing layer, a dark channel image processing layer, a basic image processing layer and an image fusion layer.

[0107] At this time, in step 201, obtaining a bright channel image of the target image to be processed, and performing feature extraction on the bright channel image to obtain a bright channel detail image may include: inputting the target image into a bright channel image processing layer, and outputting a bright channel detail image.

[0108] In step 202, a dark channel image of the target image is obtained, and features are extracted from the dark channel image to obtain a dark channel detail image, including: inputting the target image into a dark channel image processing layer, and outputting a dark channel detail image;

[0109] In step 203, feature extraction is performed on the target image to obtain a basic reconstruction image of the target image, which may include: inputting the target image into a basic image processing layer and outputting a basic reconstruction image;

[0110] In step 204, a reconstructed image is generated according to the dark channel detail image, the bright channel detail image and the basic reconstruction image, including: inputting the dark channel detail image, the bright channel detail image and the basic reconstruction image into an image fusion layer for image fusion, and outputting a reconstructed image.

[0111] In the embodiment of the present application, the prior knowledge of the bright channel image and the dark channel image is combined with deep learning, and the channel prior is used to supplement the details of the underexposed and overexposed areas. The dark channel detail map, the bright channel detail map and the basic reconstructed image are extracted according to the input target image, and finally the three are superimposed to generate the final HDR reconstructed image.

[0112] like Figure 5 FIG. 1 is a schematic diagram of the structure of an embodiment of an image processing model in an embodiment of the present application. Figure 5 The rightmost layer is the loss layer, which is only present in the initial image processing model training stage. In the application stage of the image processing model, the rightmost structure can be removed. Figure 5 The top branch is the bright channel image (bright channel) and the bright channel detail image is obtained after feature extraction. The middle branch is the target image and the basic reconstruction image is obtained after feature extraction. The bottom branch is the dark channel image (dark channel) and the dark channel detail image is obtained after feature extraction. The reconstructed image is obtained after merging and fusion.

[0113] In an embodiment of the present application, when generating a basic reconstructed image, a feature extraction unit is designed to fuse the target image, local features of the image, and global features of the image. Specifically, the basic image processing layer includes a feature extraction unit, and the feature extraction unit includes a basic branch module, a local branch module, a global branch module, and a fusion module. At this time, the target image is input into the basic image processing layer, and the basic reconstruction image is output, which further includes: inputting the target image into the basic branch module and outputting the target image; inputting the target image into the local branch module for feature extraction and outputting the local features of the image; inputting the target image into the global branch module for feature extraction and outputting the global features of the image; inputting the target image, the local features of the image, and the global features of the image into the fusion module for feature fusion, and outputting the basic reconstruction image of the target image. Furthermore, the local branch module includes two stacked dilated convolution layers, each dilated convolution layer contains a preset number of dilated convolution kernels, the dilation coefficient of the first dilated convolution layer is 1, and the dilation coefficient of the second dilated convolution layer is 2.

[0114] like Figure 6As shown in the figure, in a specific example, the feature extraction unit consists of three branches: base part, local part and global part, aiming to extract sufficient feature information. Among them, the local branch uses two layers of stacked dilated convolution layers to increase the image receptive field to obtain a wider range of image information without adding additional parameters. Each layer contains 64 3×3 dilated convolution kernels, the only difference is that the dilation coefficient of the first layer of convolution kernels is 1, and the dilation coefficient of the second layer is 2. Finally, this branch can extract neighborhood features in the range of 7×7. The global features of the image are extracted by the global branch, which is composed of three layers of pooling layers with a coefficient of 2. Considering that the global average can represent the overall characteristics of the image to a certain extent, the third layer feature map is weighted by the global average, and the result is upsampled to the original target image size, mapped to 0~1 as the weight using the Sigmoid activation function, and globally scaled point-to-point with the input target image. Finally, the local features are fused with the scaled base reconstructed image to generate multi-granular features.

[0115] In the above example, the basic image processing layer includes a feature extraction unit. It can be understood that in actual application scenarios, the bright channel image processing layer and the dark channel image processing layer may also include a feature extraction unit for feature extraction in the bright channel image and the dark channel image in step 201 and step 202. The feature extraction unit included in the bright channel image processing layer and the dark channel image processing layer may be similar to the feature extraction unit included in the basic image processing layer. For details, reference may be made to the structural setting of the feature extraction unit of the basic image processing layer. The feature extraction unit included in the bright channel image processing layer and the dark channel image processing layer may also include the basic branch module, local branch module, global branch module and fusion module described above.

[0116] In some embodiments of the present application, some subtle adjustments can be made based on the feature extraction unit of the basic image processing layer to obtain the feature extraction units included in the bright channel image processing layer and the dark channel image processing layer. For example, an additional layer of dilated convolution can be added to the basic branch module according to actual conditions, and an additional layer of pooling processing can be added to the global branch module, etc. The specific adjustments are not limited here.

[0117] In the embodiment of the present application, before using the trained image processing model, the initial image processing model needs to be trained to obtain the trained image processing model. The initial image processing model can be a convolutional neural network (CNN) or other neural network models, which can be selected according to actual conditions and are not specifically limited here.

[0118] Specifically, in some embodiments of the present application, the image processing method may further include a training step of an initial image processing model. For example, the image processing method includes: obtaining an image training set, the image training set including multiple groups of image training data, each group of image training data including sample images and target real images; training a preset initial image processing model according to the image training set to obtain a trained image processing model.

[0119] Furthermore, training a preset initial image processing model according to an image training set to obtain a trained image processing model can include: inputting sample images in the image training set into the initial image processing model for training, and outputting predicted images; obtaining target real images corresponding to the sample images in each group of image training data; using a preset loss function to converge the predicted images and the target real images to obtain a trained image processing model, and the preset loss function is used to calculate the total loss value of the initial image processing model.

[0120] Among them, different from the current common model training strategy, the embodiment of the present application also uses channel priors for auxiliary optimization during model training, that is, the total loss value may include a bright channel loss value and / or a dark channel loss value, the bright channel loss value is the loss value of the predicted image and the target real image under the bright channel image, and the dark channel loss value is the loss value of the predicted image and the target real image under the dark channel image. The basic loss value can be the loss value calculated by the predicted image and the target real image based on the mean structural similarity index measure (SSIM) function, or it can be the loss value calculated by other existing loss functions, such as the basic loss value calculated by multi-scale structural similarity (MS-SSIM).

[0121] The following examples illustrate different calculation methods of the total loss value in various scenarios.

[0122] 1. The total loss value includes the bright channel loss value

[0123] At this time, a preset loss function is used to converge the sample image and the target real image to obtain a trained image processing model, which may include: calculating the basic loss value of the predicted image and the target real image; calculating the bright channel loss value of the predicted image and the target real image; calculating the total loss value of the predicted image and the target real image based on the basic loss value and the bright channel loss value; and optimizing the initial image processing model based on the total loss value until the total loss value of the model reaches the preset requirements to obtain a trained image processing model.

[0124] Further, according to the basic loss value and the bright channel loss value, the total loss value of the predicted image and the target real image can be calculated as follows: according to the basic loss value and the basic loss weight, the bright channel loss value and the bright channel loss weight, the total loss value of the predicted image and the target real image is calculated. Among them, the basic loss weight and the bright channel loss weight can be preset according to the actual application scenario.

[0125] In one example, the preset loss function may be the following formulas (3) to (5):

[0126] Loss(X,Y)=Loss1(X,Y)+0.1×Loss2(X,Y) (3)

[0127]

[0128] Loss2(X,Y)=L1(X,Y)=|XY| (5)

[0129] Among them, X is the predicted image, Y is the target real image, Loss(X,Y) is the total loss value, Loss1(X,Y) is the basic loss value, Loss2(X,Y) is the bright channel loss value, L1(X,Y) means the loss value of the predicted image and the target real image calculated by the L1 loss function, 0.1 is the bright channel loss weight, the basic loss weight is 1, u x is the mean value of x, u y is the mean value of y, σ x 2 is the variance of x, σ y 2 is the variance of y, σ xy is the covariance of x and y, C1=(K1L) 2 , C2=(K2L) 2 is a preset constant used to maintain stability, L is the dynamic range of pixel values, K1 and K2 are preset parameters. In a specific embodiment, K1=0.01, K2=0.03.

[0130] 2. The total loss value includes the dark channel loss value

[0131] At this time, a preset loss function is used to converge the sample image and the target real image to obtain a trained image processing model, which may include: calculating the basic loss value of the predicted image and the target real image; calculating the dark channel loss value of the predicted image and the target real image; calculating the total loss value of the predicted image and the target real image based on the basic loss value and the dark channel loss value; and optimizing the initial image processing model based on the total loss value until the total loss value of the model reaches the preset requirements to obtain a trained image processing model.

[0132] Further, according to the basic loss value and the dark channel loss value, the total loss value of the predicted image and the target real image can be calculated by: according to the basic loss value and the basic loss weight, the bright channel loss value and the bright channel loss weight, the dark channel loss value and the dark channel loss weight, the total loss value of the predicted image and the target real image is calculated. Among them, the basic loss weight, the bright channel loss weight and the dark channel loss weight can be preset according to the actual application scenario.

[0133] In one example, the preset loss function may be the following formulas (6) to (7):

[0134] Loss(X,Y)=Loss1(X,Y)+0.1×Loss3(X,Y) (6)

[0135] Loss3(X,Y)=L1(X,Y)=|XY| (7)

[0136] Among them, X is the predicted image, Y is the target real image, Loss(X,Y) is the total loss value, Loss1(X,Y) is the basic loss value, Loss3(X,Y) is the dark channel loss value, L1(X,Y) represents the loss value of the predicted image and the target real image calculated by the L1 loss function, 0.1 is the dark channel loss weight, and the calculation of the basic loss value Loss1(X,Y) refers to the method described in the above embodiment, which is not described in detail here.

[0137] 3. The total loss value includes both the bright channel loss value and the dark channel loss value

[0138] At this time, a preset loss function is used to converge the sample image and the target real image to obtain a trained image processing model, which may include: calculating the basic loss value of the predicted image and the target real image; calculating the bright channel loss value of the predicted image and the target real image; calculating the dark channel loss value of the predicted image and the target real image; calculating the total loss value of the predicted image and the target real image based on the basic loss value, the bright channel loss value and the dark channel loss value; and optimizing the initial image processing model based on the total loss value until the total loss value of the model reaches the preset requirements to obtain a trained image processing model.

[0139] Further, according to the basic loss value, the bright channel loss value and the dark channel loss value, the total loss value of the predicted image and the target real image can be calculated as follows: according to the basic loss value and the basic loss weight, the dark channel loss value and the dark channel loss weight, the total loss value of the predicted image and the target real image is calculated. Among them, the basic loss weight and the dark channel loss weight can be preset according to the actual application scenario.

[0140] In one example, the preset loss function may be the following formulas (8) to (9):

[0141] Loss(X,Y)=Loss1(X,Y)+0.1×Loss2(X,Y)+0.1×Loss3(X,Y) (8)

[0142] Loss2(X,Y)=Loss3(X,Y)=L1(X,Y)=|XY| (9)

[0143] Among them, X is the predicted image, Y is the target real image, Loss(X,Y) is the total loss value, Loss1(X,Y) is the basic loss value, Loss2(X,Y) is the bright channel loss value, Loss3(X,Y) is the dark channel loss value, L1(X,Y) represents the loss value of the predicted image and the target real image calculated by the L1 loss function, the dark channel loss weight and the bright channel loss weight are both 0.1, and the calculation of the basic loss value Loss1(X,Y) refers to the method described in the above embodiment, and will not be described in detail here.

[0144] It should be noted that in the above embodiments, the function for calculating the loss is illustrated by taking the L1 loss function as an example. It can be understood that other types of loss functions may also be used. Specifically, common loss functions in existing network models may be used, such as L1 loss function, L2 loss function, SSIM loss function, MS-SSIM loss function, etc., a combination of one or more. It can be set according to the actual application scenario and is not limited here.

[0145] Since a large amount of training data that meets the requirements needs to be collected to train the initial image processing model, high-quality training data is crucial to the effect of the model in later applications. The collection of the image training set for this application is explained below.

[0146] In order to train the initial image processing model, the multi-exposure data is used to construct a training set in the embodiment of the present application. First, three LDR images are randomly selected from each scene, and then a pixel point is randomly selected as the center pixel in each LDR image, and square local blocks with side lengths of 64, 128, and 256 are randomly intercepted with radii of 32, 64, and 128. After that, the intercepted image blocks are randomly rotated and resized to 128×128, and Gaussian noise with a variance of 5-12 is added, and finally a preset number (for example, 200,000) of image training sets with different scenes and different exposures are generated.

[0147] After collecting the above training data, an image training set that meets the requirements can be collected to improve the image processing effect of the trained image processing model.

[0148] In order to facilitate better implementation of the image processing method provided in the embodiment of the present application, the embodiment of the present application also provides a model training method for the initial search network model used in the above image processing method. The meanings of the terms are the same as those in the above image processing method, and the specific implementation details can refer to the description in the method embodiment.

[0149] The model training method comprises the following steps: obtaining an image training set, wherein the image training set comprises multiple groups of image training data, each group of image training data comprises a sample image and a target real image; inputting the sample image in the image training set into the initial image processing model for training, and outputting a predicted image;

[0150] Obtain a target real image corresponding to a sample image in each group of image training data; use a preset loss function to converge the predicted image and the target real image to obtain a trained image processing model, wherein the preset loss function is used to calculate a total loss value of the initial image processing model, wherein the total loss value includes a bright channel loss value and / or a dark channel loss value, wherein the bright channel loss value is a loss value of the predicted image and the target real image under a bright channel image, and the dark channel loss value is a loss value of the predicted image and the target real image under a dark channel image.

[0151] The relevant contents in the model training method in this embodiment can be specifically referred to the contents of the relevant steps in the above image processing embodiment, and will not be repeated here.

[0152] The trained image processing model obtained by the above-mentioned model training method improves the image quality of the output superimposed image, making the subsequent image reconstruction effect better.

[0153] In the embodiment of the present application, the Keras software package can be used for training, and the network can be optimized by stochastic gradient descent (SGD) with a batch size of 64. The parameters of momentum and weight decay are 0.9 and 0.0001 respectively. In addition, the learning rate is initialized to 0.01, and it decreases by one tenth every 30 rounds, and it takes 30 hours to train 100 rounds.

[0154] Figure 7A comparative schematic diagram of the dynamic range extension results in the embodiment of the present application is shown. The lower right corner is the HDR image output by the image processing model, and the rest are LDR images of the other three exposure settings in the related art. It can be seen that due to the bright sky background in the LDR image, the ground scene is underexposed and the details are difficult to see. The method of the present invention effectively expands the dynamic range of the LDR image, and the image contrast is high. While ensuring the highlight area, it effectively brightens the brightness of the underexposed area, the distant peaks are clearly visible, and the highlighted sky area is clearly layered, and the visual effect is good. Closer to the real reference HDR image. In addition, as shown in Table 1 below, compared with the relevant existing algorithms, the HDR image reconstructed in the embodiment of the present application has the largest PU-PSNR, PU-SSIM and Q_score and the smallest NIQE, indicating that the generated HDR image has rich image details and structure, and the image quality is higher. Therefore, it has been verified that the image processing method of the present application is an effective and highly accurate multi-exposure image fusion solution.

[0155] Among them, NIQE is Natural Image Quality Evaluator; PU-PSNR is Perceptually Uniform-Peak Signal-to-Noise Ratio, which is the peak signal-to-noise ratio based on perceptual consistency coding; PU-SSIM is Perceptually Uniform-Multi Scale Structural Similarity IndexMeasure, which is the multi-scale structural similarity based on perceptual consistency coding; Q_score is a common image quality evaluation indicator in related technologies.

[0156] Table 1

[0157]

[0158] In order to better implement the image processing method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above image processing method. The meanings of the terms are the same as those in the above image processing method, and the specific implementation details can refer to the description in the method embodiment.

[0159] See also Figure 8 , Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is applied to an electronic device, wherein the image processing device 700 may include a first acquisition module 701, a second acquisition module 702, a basic reconstruction module 703 and a generation module 704, wherein:

[0160] The first acquisition module 701 is used to acquire a bright channel image of a target image to be processed, and perform feature extraction on the bright channel image to obtain a bright channel detail image;

[0161] The second acquisition module 702 acquires a dark channel image of the target image, and performs feature extraction on the dark channel image to obtain a dark channel detail image;

[0162] A basic reconstruction module 703 is used to extract features from the target image to obtain a basic reconstruction image of the target image;

[0163] The generating module 704 is configured to generate a reconstructed image according to the dark channel detail image, the bright channel detail image and the basic reconstruction image.

[0164] In some embodiments of the present application, the first acquisition module 701 is specifically used to:

[0165] Calculate the maximum channel pixel value in the target image, where the maximum channel pixel value is the maximum value among the channel pixel values ​​of each color channel of the target image;

[0166] Taking each pixel in the target image as a first target pixel, determining a maximum neighborhood pixel value of the first target pixel within a first target neighborhood range according to the maximum channel pixel value;

[0167] Determining a bright channel pixel value of the first target pixel according to the maximum neighborhood pixel value;

[0168] After taking each pixel in the target image as a first target pixel and determining a bright channel pixel value of each pixel in the target image, a bright channel image of the target image is obtained according to the bright channel pixel value of each pixel in the target image.

[0169] In some embodiments of the present application, the second acquisition module 702 is specifically used to:

[0170] Calculate the minimum channel pixel value in the target image, where the minimum channel pixel value is the minimum value among the channel pixel values ​​of each color channel of the target image;

[0171] Taking each pixel in the target image as a second target pixel, and determining the minimum neighborhood pixel value of the second target pixel within a second target neighborhood range according to the minimum channel pixel value;

[0172] Determining a dark channel pixel value of the second target pixel according to the minimum neighborhood pixel value;

[0173] After taking each pixel in the target image as a second target pixel and determining the dark channel pixel value of each pixel in the target image, a dark channel image of the target image is obtained according to the dark channel pixel value of each pixel in the target image.

[0174] In some embodiments of the present application, the basic reconstruction module 703 is specifically used to:

[0175] Acquire the target image;

[0176] Extracting local features of the target image to obtain local features of the image;

[0177] Performing global feature extraction on the target image to obtain image global features;

[0178] According to the target image, the local features of the image and the global features of the image are fused to obtain a basic reconstruction image of the target image.

[0179] In some embodiments of the present application, the basic reconstruction module 703 is specifically used to:

[0180] Performing multi-layer pooling processing on the target image to obtain a first image feature;

[0181] The first image feature is weighted by using a preset global average value to obtain a second image feature;

[0182] The second image feature is upsampled to the size of the target image to obtain an image global feature.

[0183] In some embodiments of the present application, the basic reconstruction module 703 is specifically used to:

[0184] Using a preset activation function to map the global features of the image to 0-1 as a fusion weight;

[0185] Performing point-to-point global scaling on the target image according to the fusion weight to obtain a fusion feature map;

[0186] Feature fusion is performed based on the fused feature map and the local features of the image to obtain a basic reconstruction map of the target image.

[0187] In some embodiments of the present application, the image processing method is applied to a trained image processing model; wherein the image processing model includes a bright channel image processing layer, a dark channel image processing layer, a basic image processing layer and an image fusion layer;

[0188] The first acquisition module 701 is specifically used to: input the target image into the bright channel image processing layer, and output the bright channel detail image;

[0189] The second acquisition module 702 is specifically used to: input the target image into the dark channel image processing layer, and output the dark channel detail image;

[0190] The basic reconstruction module 703 is specifically used to: input the target image into the basic image processing layer, and output the basic reconstruction image;

[0191] The generating module 704 is specifically used for: inputting the dark channel detail image, the bright channel detail image and the basic reconstruction image into the image fusion layer for image fusion, and outputting the reconstructed image.

[0192] In some embodiments of the present application, the basic image processing layer includes a feature extraction unit, and the feature extraction unit includes a basic branch module, a local branch module, a global branch module and a fusion module;

[0193] The basic reconstruction module 703 is specifically used for:

[0194] Input the target image into the basic branch module, and output the target image;

[0195] Input the target image into the local branch module for feature extraction, and output the local features of the image;

[0196] Input the target image into the global branch module for feature extraction, and output the global features of the image;

[0197] The target image, the local features of the image and the global features of the image are input into the fusion module for feature fusion, and a basic reconstruction image of the target image is output.

[0198] In some embodiments of the present application, the local branch module includes two stacked dilated convolution layers, each dilated convolution layer contains a preset number of dilated convolution kernels, the dilation coefficient of the first dilated convolution layer is 1, and the dilation coefficient of the second dilated convolution layer is 2.

[0199] In some embodiments of the present application, the device further includes a training module, and the training module is used to:

[0200] Acquire an image training set, wherein the image training set includes multiple groups of image training data, and each group of image training data includes a sample image and a target real image;

[0201] A preset initial image processing model is trained according to the image training set to obtain a trained image processing model.

[0202] In some embodiments of the present application, the training module is specifically used for:

[0203] Inputting the sample images in the image training set into the initial image processing model for training, and outputting a predicted image;

[0204] Obtain the target real image corresponding to the sample image in each set of image training data;

[0205] A preset loss function is used to converge the predicted image and the target real image to obtain a trained image processing model. The preset loss function is used to calculate the total loss value of the initial image processing model, and the total loss value includes a bright channel loss value and / or a dark channel loss value. The bright channel loss value is the loss value of the predicted image and the target real image under the bright channel image, and the dark channel loss value is the loss value of the predicted image and the target real image under the dark channel image.

[0206] In some embodiments of the present application, the training module is specifically used for:

[0207] Calculating a base loss value of the predicted image and the target real image;

[0208] Calculating bright channel loss values ​​of the predicted image and the target real image;

[0209] Calculating dark channel loss values ​​of the predicted image and the target real image;

[0210] Calculate the total loss value of the predicted image and the target real image according to the basic loss value, the bright channel loss value and the dark channel loss value;

[0211] According to the total loss value, the initial image processing model is optimized until the total loss value of the model reaches a preset requirement, thereby obtaining a trained image processing model.

[0212] In the embodiment of the present application, a bright channel image and a dark channel image of the target image are first generated, and then a dark channel detail image with more underexposed areas and a bright channel detail image with more overexposed areas are generated through the bright channel image and the dark channel image. The dark channel detail image and the bright channel detail image are combined to reconstruct the image. Since the dark channel detail image and the bright channel detail image enhance the reconstruction of the details of the underexposed and overexposed areas of the target image and expand the image dynamic range, the generated reconstructed image has richer image details and higher image quality.

[0213] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is applied to an electronic device, wherein the model training device 800 may include a first acquisition module 801, an output module 802, a second acquisition module 803 and a convergence module 804, wherein:

[0214] The first acquisition module 801 is used to acquire an image training set, wherein the image training set includes multiple groups of image training data, each group of image training data includes a sample image and a target real image;

[0215] An output module 802 is used to input the sample images in the image training set into the initial image processing model for training, and output a predicted image;

[0216] The second acquisition module 803 is used to acquire the target real image corresponding to the sample image in each set of image training data;

[0217] The convergence module 804 is used to use a preset loss function to converge the predicted image and the target real image to obtain a trained image processing model. The preset loss function is used to calculate the total loss value of the initial image processing model, and the total loss value includes a bright channel loss value and / or a dark channel loss value. The bright channel loss value is the loss value of the predicted image and the target real image under the bright channel image, and the dark channel loss value is the loss value of the predicted image and the target real image under the dark channel image.

[0218] The relevant contents in the model training device in this embodiment can be specifically referred to the relevant contents in the above-mentioned embodiments of the image processing method and the image processing device, and will not be described in detail here.

[0219] The trained image processing model obtained by training with the above-mentioned model training device improves the image quality of the output reconstructed image, making the subsequent image reconstruction effect better.

[0220] The present application also provides an electronic device, such as Fig.10 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0221] The electronic device may include components such as a processor 901 with one or more processing cores, a memory 902 with one or more computer-readable storage media, a power supply 903, and an input unit 904. Those skilled in the art will appreciate that Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0222] The processor 901 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 902 and calling data stored in the memory 902, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 901 may include one or more processing cores; preferably, the processor 901 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operation storage medium, user interface and application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 901.

[0223] The memory 902 can be used to store software programs and modules. The processor 901 executes various functional applications and data processing by running the software programs and modules stored in the memory 902. The memory 902 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating storage medium, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 902 may also include a memory controller to provide the processor 901 with access to the memory 902.

[0224] The electronic device also includes a power supply 903 for supplying power to various components. Preferably, the power supply 903 can be logically connected to the processor 901 through a power management storage medium, so as to manage charging, discharging, and power consumption management through the power management storage medium. The power supply 903 can also include any components such as one or more DC or AC power supplies, recharge storage media, power failure detection circuits, power converters or inverters, and power status indicators.

[0225] The electronic device may further include an input unit 904, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0226] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 901 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 902 according to the following instructions, and the processor 901 will run the application programs stored in the memory 902, thereby implementing the steps in the above-mentioned image processing method or model training method embodiment.

[0227] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0228] To this end, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. The computer program is loaded by a processor to execute the steps in any one of the image processing methods provided in the embodiments of the present application.

[0229] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0230] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0231] Since the computer program stored in the computer-readable storage medium can execute the steps in any image processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any image processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0232] The above is a detailed introduction to an image processing method, related devices, equipment and storage medium provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a bright channel image of the target image to be processed, and perform feature extraction on the bright channel image to obtain a bright channel detail image; Acquire a dark channel image of the target image, and perform feature extraction on the dark channel image to obtain a dark channel detail image; Performing feature extraction on the target image to obtain a basic reconstruction image of the target image; A reconstructed image is generated according to the dark channel detail image, the bright channel detail image and the basic reconstruction image.

2. The image processing method according to claim 1, characterized in that: The step of obtaining a bright channel image of a target image to be processed comprises: Calculate the maximum channel pixel value in the target image, where the maximum channel pixel value is the maximum value among the channel pixel values ​​of each color channel of the target image; Taking each pixel in the target image as a first target pixel, determining a maximum neighborhood pixel value of the first target pixel within a first target neighborhood range according to the maximum channel pixel value; Determining a bright channel pixel value of the first target pixel according to the maximum neighborhood pixel value; After taking each pixel in the target image as a first target pixel and determining a bright channel pixel value of each pixel in the target image, a bright channel image of the target image is obtained according to the bright channel pixel value of each pixel in the target image.

3. The image processing method according to claim 1, characterized in that: The step of acquiring a dark channel image of the target image comprises: Calculate the minimum channel pixel value in the target image, where the minimum channel pixel value is the minimum value among the channel pixel values ​​of each color channel of the target image; Taking each pixel in the target image as a second target pixel, and determining the minimum neighborhood pixel value of the second target pixel within a second target neighborhood range according to the minimum channel pixel value; Determining a dark channel pixel value of the second target pixel according to the minimum neighborhood pixel value; After taking each pixel in the target image as a second target pixel and determining the dark channel pixel value of each pixel in the target image, a dark channel image of the target image is obtained according to the dark channel pixel value of each pixel in the target image.

4. The image processing method according to claim 1, characterized in that: The step of extracting features from the target image to obtain a basic reconstruction image of the target image includes: Acquire the target image; Extracting local features of the target image to obtain local features of the image; Performing global feature extraction on the target image to obtain image global features; According to the target image, the local features of the image and the global features of the image are fused to obtain a basic reconstruction image of the target image.

5. The image processing method according to claim 4, characterized in that: The step of extracting global features from the target image to obtain global features of the image includes: Performing multi-layer pooling processing on the target image to obtain a first image feature; The first image feature is weighted by using a preset global average value to obtain a second image feature; The second image feature is upsampled to the size of the target image to obtain an image global feature.

6. The image processing method according to claim 4, characterized in that: The step of fusing the local features of the image and the global features of the image according to the target image to obtain a basic reconstruction image of the target image includes: Using a preset activation function to map the global features of the image to 0-1 as a fusion weight; Performing point-to-point global scaling on the target image according to the fusion weight to obtain a fusion feature map; Feature fusion is performed based on the fused feature map and the local features of the image to obtain a basic reconstruction map of the target image.

7. The image processing method according to claim 4, characterized in that: The image processing method is applied to a trained image processing model; wherein the image processing model includes a bright channel image processing layer, a dark channel image processing layer, a basic image processing layer and an image fusion layer; The step of acquiring a bright channel image of a target image to be processed and performing feature extraction on the bright channel image to obtain a bright channel detail image comprises: inputting the target image into the bright channel image processing layer and outputting the bright channel detail image; The step of acquiring a dark channel image of the target image and performing feature extraction on the dark channel image to obtain a dark channel detail image comprises: inputting the target image into the dark channel image processing layer and outputting the dark channel detail image; The step of extracting features from the target image to obtain a basic reconstruction image of the target image includes: inputting the target image into the basic image processing layer and outputting the basic reconstruction image; The generating a reconstructed image according to the dark channel detail map, the bright channel detail map and the basic reconstruction map comprises: inputting the dark channel detail map, the bright channel detail map and the basic reconstruction map into the image fusion layer for image fusion, and outputting the reconstructed image.

8. The image processing method according to claim 7, characterized in that: The basic image processing layer includes a feature extraction unit, and the feature extraction unit includes a basic branch module, a local branch module, a global branch module and a fusion module; The step of inputting the target image into the basic image processing layer and outputting the basic reconstruction image comprises: Input the target image into the basic branch module, and output the target image; Input the target image into the local branch module for feature extraction, and output the local features of the image; Input the target image into the global branch module for feature extraction, and output the global features of the image; The target image, the local features of the image and the global features of the image are input into the fusion module for feature fusion, and a basic reconstruction image of the target image is output.

9. The image processing method according to claim 8, characterized in that: The local branch module includes two stacked dilated convolution layers, each of which includes a preset number of dilated convolution kernels, a dilation coefficient of the first dilated convolution layer is 1, and a dilation coefficient of the second dilated convolution layer is 2.

10. The image processing method according to any one of claims 7 to 9, characterized in that: The method further comprises: Acquire an image training set, wherein the image training set includes multiple groups of image training data, and each group of image training data includes a sample image and a target real image; A preset initial image processing model is trained according to the image training set to obtain a trained image processing model.

11. The image processing method according to any one of claim 10, characterized in that: The step of training a preset initial image processing model according to the image training set to obtain a trained image processing model comprises: Inputting the sample images in the image training set into the initial image processing model for training, and outputting a predicted image; Obtain the target real image corresponding to the sample image in each set of image training data; A preset loss function is used to converge the predicted image and the target real image to obtain a trained image processing model. The preset loss function is used to calculate the total loss value of the initial image processing model, and the total loss value includes a bright channel loss value and / or a dark channel loss value. The bright channel loss value is the loss value of the predicted image and the target real image under the bright channel image, and the dark channel loss value is the loss value of the predicted image and the target real image under the dark channel image.

12. The image processing method according to claim 11, characterized in that: The method of using a preset loss function to converge the sample image and the target real image to obtain a trained image processing model includes: Calculating a base loss value of the predicted image and the target real image; Calculating bright channel loss values ​​of the predicted image and the target real image; Calculating dark channel loss values ​​of the predicted image and the target real image; Calculate the total loss value of the predicted image and the target real image according to the basic loss value, the bright channel loss value and the dark channel loss value; According to the total loss value, the initial image processing model is optimized until the total loss value of the model reaches a preset requirement, thereby obtaining a trained image processing model.

13. An image processing device, characterized in that: The device comprises: A first acquisition module is used to acquire a bright channel image of a target image to be processed, and perform feature extraction on the bright channel image to obtain a bright channel detail image; The second acquisition module acquires a dark channel image of the target image and performs feature extraction on the dark channel image to obtain a dark channel detail image; A basic reconstruction module, used for extracting features from the target image to obtain a basic reconstruction image of the target image; A generating module is used to generate a reconstructed image according to the dark channel detail image, the bright channel detail image and the basic reconstruction image.

14. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps in the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the method according to any one of claims 1 to 12.

Images