Image processing method and device, electronic device, and storage medium

Abstract

The disclosed embodiments are about an image processing method and device, electronic device, and storage medium, and relate to the field of image processing technology. The image processing method includes: performing panoramic image segmentation on a first image to obtain multiple image regions, and classifying the multiple image regions to obtain multiple categories; performing local color mapping on the image region where each category in the first image is located separately according to the grayscale image corresponding to the first image and the second image, and determining the mapping relationship between the pixels of the first image and the pixels of the second image; mapping the second image according to the mapping relationship to obtain a mapping result, and guiding and optimizing the mapping result to determine the mapping image corresponding to the second image. The technical solution disclosed in the present invention can improve the accuracy of image color mapping and image quality.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to an image processing method, an image processing device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the fusion algorithm of RGB image and NIR (Near Infrared) image, it is necessary to match the feature points of the dual-camera images, and then align and fuse the images. In the feature point matching process, due to the imaging characteristics of the near-infrared image, the feature descriptors at the same position will be quite different, so it is necessary to map the images first to make the RGB image and the near-infrared image more similar.

[0003] In the related art, a histogram matching method is generally used to perform one-to-one mapping on all pixels. Specifically, distribution statistics are performed on each pixel value, and the mapping relationship is calculated using probability distribution information so that the histograms of the two are similar.

[0004] In the above method, since it is based on color mapping between RGB images, it is not suitable for the case of RGB images to near-infrared images. The imaging law of near-infrared images is different from that of RGB images and has certain limitations, resulting in large errors in the results.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide an image processing method and device, an electronic device, and a storage medium, thereby overcoming the problem of poor image mapping accuracy caused by the limitations and defects of related technologies at least to a certain extent.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of the present disclosure, there is provided an image processing method, including: performing panoramic image segmentation on a first image to obtain multiple image areas, and classifying the multiple image areas to obtain multiple categories; performing local color mapping on the image area where each category in the first image is located separately according to the grayscale image and the second image corresponding to the first image, and determining the mapping relationship between the pixels of the first image and the pixels of the second image; mapping the second image according to the mapping relationship to obtain a mapping result, and guiding and optimizing the mapping result to determine the mapping image corresponding to the second image.

[0009] According to one aspect of the present disclosure, there is provided an image processing device, comprising: an image segmentation module, used to perform panoramic image segmentation on a first image to obtain multiple image areas, and classify the multiple image areas to obtain multiple categories; a mapping relationship determination module, used to perform local color mapping on the image area where each category in the first image is located separately according to the grayscale image corresponding to the first image and the second image, and determine the mapping relationship between the pixels of the first image and the pixels of the second image; an image mapping module, used to map the second image according to the mapping relationship to obtain a mapping result, and guide and optimize the mapping result to determine a mapping image corresponding to the second image.

[0010] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above-mentioned image processing methods by executing the executable instructions.

[0011] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method described in any one of the above is implemented.

[0012] In the image processing method, image processing device, electronic device and computer-readable storage medium provided in the embodiments of the present disclosure, local color mapping is performed separately for the image area where each category in the first image is located, according to the grayscale image corresponding to the first image and the second image, to determine the mapping relationship between the pixels of the first image and the pixels of the second image; the mapping result of the second image is guided and optimized to determine the mapping image corresponding to the second image. On the one hand, according to the grayscale image corresponding to the first image and the second image, local color mapping is performed separately for the image area where each category in the first image is located, which can realize local mapping of the image by region, avoid the problem of large errors caused by different imaging rules, improve the accuracy of local mapping, and improve the accuracy of color mapping based on local mapping, thereby improving the accuracy of feature point matching. On the other hand, local color mapping is performed on the first image and the second image by region, which can realize color mapping between different types of images, avoid the limitations of color mapping in related technologies, perform color mapping in a suitable manner, and increase the scope of application.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0015] Figure 1 A schematic diagram is shown of an application scenario to which the image processing method or image processing device of an embodiment of the present disclosure can be applied.

[0016] Figure 2 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown.

[0017] Figure 3 A schematic diagram schematically illustrates an image processing method in an embodiment of the present disclosure.

[0018] Figure 4 A schematic diagram of panoramic segmentation in an embodiment of the present disclosure is schematically shown.

[0019] Figure 5 A schematic diagram schematically illustrates an RGB image and a near-infrared image in an embodiment of the present disclosure.

[0020] Figure 6 The flowchart of color mapping in the embodiment of the present disclosure is schematically shown.

[0021] Figure 7 A schematic diagram of a process for determining a mapping relationship in an embodiment of the present disclosure is schematically shown.

[0022] Figure 8 The following is a schematic diagram of the equalization processing of the second histogram data in the embodiment of the present disclosure.

[0023] Fig. 9 The overall process diagram of color mapping in the embodiment of the present disclosure is schematically shown.

[0024] Fig.10 A block diagram of an image processing device in an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0026] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0027] An image processing method is provided in an embodiment of the present disclosure, which can be applied to a scene requiring color mapping.

[0028] Figure 1 A schematic diagram is shown of an application scenario to which the image processing method or image processing device of an embodiment of the present disclosure can be applied.

[0029] This image processing method can be applied to the application scenario of image fusion of RGB images and near-infrared images. Figure 1As shown in , it can be specifically applied to the process of using the client 101 to perform image color mapping on the first image 102 and the second image 103 received on the client. Among them, the client can be various types of devices with computing functions, such as smart phones, tablet computers, desktop computers, vehicle-mounted devices, wearable devices, etc. The first image 102 can be any type of high-definition image in various scenes, and can be an image obtained by shooting, or an image obtained from the network or other terminals, and the type of the image is not limited. The first image can be an RGB image. The second image 103 can be an image obtained at the same time as the first image, and the type of the second image is different from the type of the first image, and the second image can be a near-infrared light image NIR. The client 101 can perform panoramic image segmentation on the first image, and then perform local color mapping on each area in the segmentation result separately according to the grayscale image corresponding to the first image and the second image, so as to determine the mapping relationship between pixels. Further, the second image is mapped according to the mapping relationship to obtain a mapping result. The mapping result is further guided filtering to obtain the final mapping image, thereby realizing the entire color mapping process.

[0030] It should be noted that the image processing method provided in the embodiment of the present disclosure can be completely executed by the client. Accordingly, the image processing device can be set in the client.

[0031] Figure 2 Schematic diagram of an electronic device suitable for implementing an exemplary embodiment of the present disclosure is shown. The terminal of the present disclosure may be configured as follows Figure 2 The electronic device shown, however, needs to be explained. Figure 2 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0032] The electronic device of the present disclosure includes at least a processor and a memory, wherein the memory is used to store one or more programs. When the one or more programs are executed by the processor, the processor can implement the method of the exemplary embodiment of the present disclosure.

[0033] Specifically, Figure 2As shown, the electronic device 200 may include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a Subscriber Identification Module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, and a bone conduction sensor, etc.

[0034] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0035] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors. In addition, a memory may be provided in the processor 210 for storing instructions and data.

[0036] The USB interface 230 is an interface that complies with the USB standard specification, and specifically can be a MiniUSB interface, a MicroUSB interface, a USB Type C interface, etc. The USB interface 230 can be used to connect a charger to charge the electronic device 200, and can also be used to transmit data between the electronic device 200 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, etc.

[0037] The charging management module 240 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the display screen 290, the camera module 291 and the wireless communication module 260.

[0038] The wireless communication function of the electronic device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor, the baseband processor, and the like.

[0039] The mobile communication module 250 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the electronic device 200 .

[0040] The wireless communication module 260 can provide wireless communication solutions for application in the electronic device 200, including Wireless Local Area Networks (WLAN) (such as Wireless Fidelity (Wi-Fi) network), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), infrared technology (IR), etc.

[0041] The electronic device 200 implements the display function through a GPU, a display screen 290, and an application processor. The GPU is a microprocessor for image blurring, connecting the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0042] The electronic device 200 can realize the shooting function through the ISP, the camera module 291, the video codec, the GPU, the display screen 290 and the application processor. In some embodiments, the electronic device 200 may include 1 or N camera modules 291, N is a positive integer greater than 1, if the electronic device 200 includes N cameras, one of the N cameras is a main camera, and the others may be secondary cameras, such as telephoto cameras.

[0043] The internal memory 221 can be used to store computer executable program codes, which include instructions. The internal memory 221 can include a program storage area and a data storage area. The external memory interface 222 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200.

[0044] The electronic device 200 can implement audio functions such as music playing and recording through the audio module 270 , the speaker 271 , the receiver 272 , the microphone 273 , the headphone jack 274 and the application processor.

[0045] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 can be arranged in the processor 210.

[0046] The speaker 271 is used to convert audio electrical signals into sound signals. The electronic device 200 can listen to music or listen to hands-free calls through the speaker 271. The receiver 272, also called the "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 200 answers a call or voice message, the voice can be answered by placing the receiver 272 close to the human ear. The microphone 273, also called the "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by placing the human mouth close to the microphone 273 to input the sound signal into the microphone 273. The electronic device 200 can be provided with at least one microphone 273. The headphone jack 274 is used to connect wired headphones.

[0047] Regarding the sensors included in the electronic device 200, the depth sensor is used to obtain the depth information of the scene. The pressure sensor is used to sense the pressure signal and can convert the pressure signal into an electrical signal. The gyroscope sensor can be used to determine the motion posture of the electronic device 200. The air pressure sensor is used to measure the air pressure. The magnetic sensor includes a Hall sensor. The electronic device 200 can use the magnetic sensor to detect the opening and closing of the flip leather case. The acceleration sensor can detect the magnitude of the acceleration of the electronic device 200 in various directions (generally three axes). The distance sensor is used to measure the distance. The proximity light sensor may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The fingerprint sensor is used to collect fingerprints. The temperature sensor is used to detect the temperature. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. The visual output related to the touch operation can be provided through the display screen 290. The ambient light sensor is used to sense the ambient light brightness. The bone conduction sensor can obtain a vibration signal.

[0048] The button 294 includes a power button, a volume button, etc. The button 294 can be a mechanical button. It can also be a touch button. The motor 293 can generate a vibration prompt. The motor 293 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. The indicator 292 can be an indicator light, which can be used to indicate the charging status, power changes, messages, missed calls, notifications, etc. The SIM card interface 295 is used to connect the SIM card. The electronic device 200 interacts with the network through the SIM card to realize functions such as calls and data communications.

[0049] The present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.

[0050] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0051] Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0052] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments.

[0053] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0054] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0055] Next, the image processing method in the embodiment of the present disclosure will be described in detail with reference to the drawings.

[0056] In step S310, panoramic image segmentation is performed on the first image to obtain a plurality of image regions, and the plurality of image regions are classified to obtain a plurality of categories.

[0057] In the disclosed embodiment, the first image may be an image captured by the terminal, specifically an RGB image. The terminal may be a binocular camera, and the RGB image and the near-infrared image are acquired respectively by the binocular camera on the terminal. It should be noted that parallax may exist in the process of shooting with the binocular camera.

[0058] Panoramic image segmentation is used to segment the regions of different objects in an image (including different objects in the foreground and the background) to obtain a segmented image. That is, the panoramic segmentation task requires that each pixel in the image must be assigned to a semantic label and an instance identifier. Among them, the semantic label refers to the category of the object, and the instance identifier corresponds to the different numbers of the same type of objects. Among them, the panoramic segmentation result corresponding to each object can be represented by another feature map. The distribution of the eigenvalue size in the feature map can be mapped to obtain the area of ​​the object in the image. In this way, the area of ​​each object (including each object and the background) in the image can be obtained, thereby realizing the panoramic segmentation of the image.

[0059] After the first image is divided into multiple image regions, the multiple image regions can be classified into multiple categories, that is, image regions of the same type are determined as one category. The purpose of classifying multiple image regions is to perform color mapping on the image region of each category separately to improve local accuracy. Figure 4 As shown in Figure 4 By classifying each image region in the first image represented by Figure A in FIG, we can obtain Figure 4 Figure B shows multiple categories.

[0060] In step S320, based on the grayscale image corresponding to the first image and the second image, local color mapping is performed separately on the image area where each category in the first image is located to determine the mapping relationship between the pixels of the first image and the pixels of the second image.

[0061] In the disclosed embodiment, the grayscale image of the first image refers to a single-channel grayscale image of the first image, which can be obtained by converting the first image. In a single-channel grayscale image, each pixel has only one value representing the color, that is, a grayscale image. The second image refers to a near-infrared image, and the second image can be acquired by a binocular camera while capturing the first image. Furthermore, the single-channel grayscale image of the first image can be consistent with the single channel of the near-infrared image. The imaging law of the second image is different from that of the first image, and the feature descriptors at the same position are quite different. Figure 5 As shown in the figure, Figure A shows the RGB image, and Figure B shows the near-infrared image. Figure 5 It can be seen that in the first area (plant part), the brightness of the near-infrared image is greater than that of the RGB image, while in the second area (sky area), the brightness of the near-infrared image is less than that of the RGB image. Since different areas have different brightness, it is necessary to map the colors by area to improve the accuracy of color mapping.

[0062] On this basis, the image region where each category is located can be color mapped locally according to the grayscale image of the first image and the second image. Figure 6 A flowchart for color mapping is schematically shown in FIG. Figure 6 As shown in , it mainly includes the following steps:

[0063] In step S610, a target area is selected from multiple image areas, histogram statistics are performed on the grayscale image corresponding to the first image in the target area to obtain first histogram data, and histogram statistics are performed on the second image in the target area to obtain second histogram data.

[0064] In this step, the target area can be any one of the multiple image areas, and can be randomly selected or selected according to a preset rule, and the preset rule can be set according to actual needs. The histogram represents the data distribution by a series of longitudinal stripes or line segments of unequal heights. In the disclosed embodiment, the histogram is used to count the distribution of pixels corresponding to the grayscale image of the first image and the pixels corresponding to the second image.

[0065] Specifically, a histogram statistics may be performed on the pixels of the first image in the target area to obtain the first histogram data. The first histogram data may be used to represent the pixel statistics result of the first image. Among them, in order to ensure accuracy, a histogram statistics may be performed on the pixels in the grayscale image whose grayscale values ​​satisfy the pixel conditions. The pixel conditions are used to describe the conditions that need to be met for performing histogram statistics, so as to filter the pixels for statistical analysis and improve the effectiveness of the operation. For example, the pixel condition may be that the grayscale value is within a pixel range, and the pixel range may be that the grayscale value is within a range between 15 and 240. That is, a histogram statistics is performed on the pixels in the grayscale image whose grayscale values ​​are greater than 15 and whose grayscale values ​​are less than 240, to obtain the first histogram data.

[0066] Similarly, histogram statistics can be performed on the second image in the target area to obtain second histogram data. The second histogram data can be used to represent the pixel statistics of the second image. In order to ensure accuracy, histogram statistics can be performed on pixels in the second image whose grayscale values ​​meet the pixel conditions. The pixel conditions can be used to describe the pixel range where the grayscale value is located, and the pixel range can be a range between 15-240 grayscale values. That is, histogram statistics are performed on pixels in the second image in the target area whose grayscale values ​​are greater than 15 and whose grayscale values ​​are less than 240 to obtain second histogram data.

[0067] By performing histogram statistics on the grayscale image and the pixels in the second image whose grayscale values ​​meet the pixel conditions, and not performing statistics on the pixels whose grayscale values ​​are outside the pixel range, the influence of overexposed and underexposed areas on pixel statistics can be avoided, thereby improving the accuracy of histogram data statistics.

[0068] In step S620, pixels corresponding to the grayscale image are determined according to the first histogram data, and pixels corresponding to the second image are determined according to the second histogram data.

[0069] In this step, since the first histogram data represents the distribution of pixels of the grayscale image, the pixels of the grayscale image can be determined by the first histogram data. Similarly, since the second histogram data represents the distribution of pixels of the second image, the pixels of the second image can be determined by the second histogram data.

[0070] In step S630, a mapping relationship between pixels corresponding to the grayscale image and pixels corresponding to the second image is calculated according to a histogram matching algorithm.

[0071] In this step, the histogram matching method is based on the histogram of an image, so that the grayscale distribution of the original image is similar to the grayscale distribution of the reference image (that is, the histogram of an image is converted to make it similar to the histogram of another image), or the two images have similar tones and contrasts. The histogram matching algorithm is designed to find a grayscale transformation function, that is, to establish a linear relationship between the mean and variance of the two images. That is, the histogram matching algorithm is used to make the grayscale distribution of the grayscale image and the second image similar, so that the grayscale image and the second image are similar.

[0072] On this basis, the pixels corresponding to the grayscale image and the pixels corresponding to the second image may be mapped to obtain a mapping relationship. Figure 7 A flowchart for determining a mapping relationship is schematically shown in FIG. Figure 7 As shown in , it mainly includes the following steps:

[0073] In step S710, the first histogram data of the grayscale image is equalized by a preset number of intervals, and the second histogram data of the second image is equalized by a preset number of intervals.

[0074] In this step, the histogram data can be equalized. Histogram equalization can enhance the contrast of the image. The histogram equalization process is to change the grayscale histogram of the original image from a relatively concentrated grayscale interval to a uniform distribution in the entire grayscale range. Histogram equalization is used to perform nonlinear stretching on the image and redistribute the image pixel values ​​so that the number of pixels in a certain grayscale range is roughly the same.

[0075] Specifically, a preset number of intervals can be set for equalization. The interval can be represented by bin to represent the equalization number. The preset number n can be 32 or other numbers. The interval refers to the equalization number, which is used to calculate the color histogram. The color space needs to be divided into multiple small color intervals, and the color histogram is obtained by calculating the pixels of the color in each small interval. For the same histogram data, the number of pixels contained in each interval in the preset number is basically the same to achieve data equalization.

[0076] refer to Figure 8 As shown in , the horizontal axis is the maximum value of the pixels in each interval, and the vertical axis is the number of pixels contained in each interval. Since the number of pixels contained in each interval is basically the same, the equalization effect is better. For example, the second interval is 20, which means that the number of pixels with pixel values ​​of 19 to 20 are divided into the second interval.

[0077] Based on this, a preset number of intervals can be set to perform equalization processing on the first histogram data, and a preset number of intervals can be set to perform equalization processing on the second histogram data. The number of pixels contained in each interval of the first histogram data and the second histogram data can be the same or different, and the maximum value of the pixels in each interval can be the same or different, depending on actual needs. The maximum value of the pixels in each interval can be determined according to the equalization number. For example, the maximum pixel value in the first interval of the first histogram data is 21, and the maximum pixel value in the first interval of the second histogram data is 18.

[0078] It should be noted that the number of intervals can be adaptively adjusted for different scenes to achieve better results. The more intervals there are, the stronger the histogram's color resolution is, but it may increase the burden on the computer.

[0079] In step S720, the preset number of mapping relationships is determined according to the maximum pixel values ​​of the first histogram data of the grayscale image and the second histogram data of the second image in the same interval.

[0080] In this step, the maximum pixel value corresponding to each interval in the first histogram data and the second histogram data can be mapped to determine the mapping relationship. That is, the mapping relationship can be determined based on the correspondence between the maximum pixel value of each interval in the first histogram data and the maximum pixel value of each interval in the second histogram data. For example, assuming that the maximum pixel value of the first interval of the first histogram data is 21, and the maximum pixel value in the first interval of the second histogram data is 18, then the mapping relationship between the first interval of the first histogram data and the second histogram data is 18 to 21. Therefore, a preset number of mapping relationships can be obtained based on a preset number of intervals.

[0081] In order to ensure the continuity of the interval, the maximum pixel value of the first interval is equal to the minimum pixel value of the second interval. Based on this, when determining the mapping relationship, it can also be understood that: the mapping relationship is determined according to the minimum pixel value and the maximum pixel value of each interval. Specifically, the first mapping relationship can be determined by the minimum pixel value of each interval corresponding to the first histogram data and the second histogram data, and the second mapping relationship can be determined according to the maximum pixel value of each interval corresponding to the first histogram data and the second histogram data. The mapping relationship corresponding to the pixels of each interval can be further determined according to the first mapping relationship and the second mapping relationship. Among them, the minimum pixel value refers to the starting point of each interval, and the maximum pixel value refers to the end point of each interval. For adjacent intervals, the maximum pixel value of the previous interval adjacent to the current interval is equal to the minimum pixel value of the current interval. Since the first mapping relationship corresponding to the starting point of each interval is the second mapping relationship corresponding to the end point of the previous interval, in order to avoid repetition, the first mapping relationship can be ignored, and the mapping relationship between the first histogram data and the second histogram data can be determined according to the maximum pixel value of the same interval.

[0082] In the disclosed embodiment, local color mapping is performed separately for the image region where each category is located according to the grayscale image corresponding to the first image and the second image, so that local mapping can be achieved for the image by region, and targeted calculation can be performed for each region, avoiding the problem of large errors in the overall calculation caused by different imaging rules, and improving the accuracy of color mapping. Color mapping is performed on the first image and the second image by region, so that color mapping can be performed on different types of images, avoiding the limitation of color mapping only for images of the same type, and solving the problem of color anomalies in related technologies in an appropriate manner, thereby improving the accuracy of color mapping and the accuracy of feature point matching, which is beneficial to the implementation of the fusion algorithm and improves the image fusion effect.

[0083] In addition, since there may be uncalculated pixel values ​​in two adjacent mapping relationships, the method further includes: if there are uncalculated pixel values ​​in two adjacent mapping relationships, the mapping relationship is filled in by linear interpolation. Two adjacent mapping relationships, for example, the mapping relationship of the first interval and the mapping relationship of the second interval. Uncalculated pixel values ​​refer to pixel values ​​whose mapping relationships are not determined, which can be specifically determined based on the starting point and end point of the first histogram data and the second histogram data. That is, the first mapping relationship can be calculated based on the starting point of the first histogram data and the second histogram data, and the second mapping relationship can be calculated based on the end point of the first histogram data and the second histogram data to obtain the mapping relationship, and the remaining pixels outside the mapping relationship range are determined as uncalculated pixel values.

[0084] After determining the uncalculated pixel value, the mapping relationship of the uncalculated pixel value can be filled in according to the linear interpolation method. The uncalculated pixel value can be linearly interpolated according to the determined mapping relationship. For example, if the uncalculated pixel value is the pixel value corresponding to the starting point, linear interpolation is performed according to the second mapping relationship. If the uncalculated pixel value is the pixel value corresponding to the end point, linear interpolation is performed according to the first mapping relationship. Figure 8 As shown in , for example, the fifth interval contains pixels with values ​​from 25 to 28, and the mapping relationship therein is filled using a linear interpolation method.

[0085] For example, if the first interval of the first histogram data contains pixel values ​​of 1-10, and the first interval of the second histogram data contains pixel values ​​of 1-20. It can be calculated that the first mapping relationship is 1 to 1, and the second mapping relationship is 10-20. For pixels whose uncalculated pixel values ​​are 2-9, the mapping relationship of the uncalculated pixel values ​​can be linearly interpolated according to the second mapping relationship. The corresponding relationship during linear interpolation (i.e., the linear interpolation coefficient) is determined according to the corresponding relationship in the second mapping relationship. For example, the mapping relationship of pixel value 2 is 2-4, and the mapping relationship of pixel value 3 is 3-6. In the embodiment of the present disclosure, by performing linear interpolation to fill the mapping relationship of the uncalculated pixel values, the integrity can be improved and data equalization can be achieved.

[0086] Furthermore, since only pixels whose grayscale values ​​satisfy the pixel conditions are counted when performing histogram statistics, in order to improve accuracy, the method further includes: for pixels whose grayscale values ​​do not satisfy the pixel conditions in the grayscale image and the second image, mapping the target interval according to the linear interpolation coefficient of the adjacent interval of the target interval where the pixel is located. The target interval can be an overexposed area or an underexposed area, for example, the first interval represents an underexposed area, and the last interval represents an overexposed area. Due to underexposure or overexposure problems, the image quality of the target area is poor. In order to avoid its influence on the overall image, the linear interpolation coefficient can be determined according to the mapping relationship of the adjacent areas, and then the mapping relationship of the target area is determined according to the linear interpolation coefficient.

[0087] For example, taking the first interval as an example, the linear interpolation coefficient determined by the mapping relationship of the second interval is used to re-map the first interval, and the result is guaranteed to be greater than or equal to 0. The last interval is also recalculated using the second to last interval, and the result is guaranteed to be greater than or equal to 0, so as to improve the effectiveness and accuracy. In the embodiment of the present disclosure, separate processing is performed on the overexposed area and the underexposed area to improve the accuracy.

[0088] In step S330, the second image is mapped according to the mapping relationship to obtain a mapping result, and the mapping result is guided and optimized to determine a mapping image corresponding to the second image.

[0089] In the embodiment of the present disclosure, after obtaining the mapping relationship between the pixels of the first image and the pixels of the second image, the second image can be mapped according to the mapping relationship to obtain a mapping result, that is, the mapped second image, so that the color of the second image is close to that of the single-channel grayscale image. Further, the mapping result can be guided and optimized to determine the mapping image corresponding to the second image. Specifically, the second image can be used as a guide image, and the obtained mapping result can be guided and filtered to determine the mapping image. That is, the original near-infrared light image is used as a guide image, and the mapped second image is guided and filtered to improve the effect. The guided filtering uses a guide image to filter the mapped second image, that is, the initial image (input image), so that the final output image is similar to the initial image, but the texture part is similar to the guide image (the original second image). The guided filtering can not only achieve edge smoothing of bilateral filtering, but also has good performance in detecting edge attachments. In the embodiment of the present disclosure, by guiding the second image after mapping, the inaccuracy caused by the parallax in the binocular shooting process and the error in the separation result is avoided, and the color abnormality problem is also avoided, which can improve the quality of the mapped image.

[0090] Fig. 9 The overall flow chart of color mapping is shown schematically in Fig. 9 As shown in , it mainly includes the following steps:

[0091] In step S910, an RGB image is acquired.

[0092] In step S920, the RGB image is panorama segmented to obtain multiple categories.

[0093] In step S930 , a near infrared light image NIR is acquired.

[0094] In step S940, a local mapping relationship is calculated based on the RGB image and the near-infrared image.

[0095] In step S950 , guided filtering is performed on the mapped near-infrared light image according to the near-infrared light image.

[0096] In step S960, a mapping image is obtained.

[0097] The technical solution provided in the embodiment of the present disclosure performs local color mapping on the image area where each category is located separately according to the grayscale image corresponding to the first image and the second image, and can realize local mapping of the image by region, which can avoid the problem of large errors caused by different imaging rules and improve the accuracy of color mapping. Color mapping is performed on the first image and the second image by region, and color mapping can be performed on different types of images, solving the problem of color anomalies in related technologies, thereby improving the accuracy of color mapping, improving the accuracy of feature point matching, and thus facilitating the implementation of the fusion algorithm.

[0098] The present disclosure provides an image processing device, referring to Fig.10 As shown in , the image processing device 1000 may include:

[0099] An image segmentation module 1001 is used to perform panoramic image segmentation on the first image to obtain multiple image regions, and classify the multiple image regions to obtain multiple categories;

[0100] A mapping relationship determination module 1002 is used to perform local color mapping on the image region where each category in the first image is located according to the grayscale image corresponding to the first image and the second image, and determine the mapping relationship between the pixels of the first image and the pixels of the second image;

[0101] The image mapping module 1003 is used to map the second image according to the mapping relationship to obtain a mapping result, and to guide and optimize the mapping result to determine a mapping image corresponding to the second image.

[0102] In an exemplary embodiment of the present disclosure, a mapping relationship determination module includes: a histogram statistics module, which is used to select a target area from multiple image areas, perform histogram statistics on a grayscale image corresponding to the first image in the target area to obtain first histogram data, and perform histogram statistics on the second image in the target area to obtain second histogram data; a pixel determination module, which is used to determine pixels corresponding to the grayscale image based on the first histogram data, and determine pixels corresponding to the second image based on the second histogram data; a mapping relationship calculation module, which is used to calculate the mapping relationship between pixels corresponding to the grayscale image and pixels corresponding to the second image based on a histogram matching algorithm.

[0103] In an exemplary embodiment of the present disclosure, the histogram statistics module includes: a statistical control module, which is used to perform histogram statistics on pixels whose grayscale values ​​contained in the grayscale image meet the pixel conditions to obtain first histogram data, and to perform histogram statistics on pixels whose grayscale values ​​contained in the second image meet the pixel conditions to obtain second histogram data.

[0104] In an exemplary embodiment of the present disclosure, the mapping relationship calculation module includes: an equalization module, which is used to equalize the first histogram data of the grayscale image through a preset number of intervals, and to equalize the second histogram data of the second image through the preset number of intervals; a calculation control module, which is used to determine the preset number of mapping relationships based on the maximum pixel value of the first histogram data of the grayscale image and the second histogram data of the second image in the same interval.

[0105] In an exemplary embodiment of the present disclosure, the device further includes: a filling module, configured to fill the mapping relationship of the uncalculated pixel values ​​by using a linear interpolation method if there are uncalculated pixel values ​​in two adjacent mapping relationships.

[0106] In an exemplary embodiment of the present disclosure, the device also includes: a supplementary mapping module, which is used to map the target interval according to the linear interpolation coefficients of the adjacent intervals of the target interval where the pixels are located, for the pixels whose grayscale values ​​in the grayscale image and the second image do not meet the pixel conditions.

[0107] In an exemplary embodiment of the present disclosure, the image mapping module includes: a guided filtering module, configured to use the second image as a guided image and perform guided filtering on the mapping result to determine the mapping image.

[0108] It should be noted that the specific details of each module in the above-mentioned image processing device have been described in detail in the corresponding image processing method, so they will not be repeated here.

[0109] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0110] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0111] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0112] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

[0113] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Performing panoramic image segmentation on the first image to obtain a plurality of image regions, and classifying the plurality of image regions to obtain a plurality of categories; According to the grayscale image corresponding to the first image and the second image, performing local color mapping separately on the image region where each category in the first image is located, and determining a mapping relationship between pixels of the first image and pixels of the second image; Mapping the second image according to the mapping relationship to obtain a mapping result, and guiding and optimizing the mapping result to determine a mapping image corresponding to the second image; The step of performing local color mapping on the image region where each category in the first image is located separately to determine the mapping relationship between pixels of the first image and pixels of the second image includes: Selecting a target area from a plurality of image areas, performing histogram statistics on a grayscale image corresponding to the first image in the target area to obtain first histogram data, and performing histogram statistics on the second image in the target area to obtain second histogram data; Determining pixels corresponding to the grayscale image according to the first histogram data, and determining pixels corresponding to the second image according to the second histogram data; A mapping relationship between pixels corresponding to the grayscale image and pixels corresponding to the second image is calculated according to a histogram matching algorithm.

2. The image processing method according to claim 1, characterized in that: The step of performing histogram statistics on the grayscale image corresponding to the first image in the target area to obtain first histogram data, and performing histogram statistics on the second image in the target area to obtain second histogram data, comprises: Histogram statistics are performed on pixels whose grayscale values ​​satisfy the pixel conditions contained in the grayscale image to obtain first histogram data, and histogram statistics are performed on pixels whose grayscale values ​​satisfy the pixel conditions contained in the second image to obtain second histogram data.

3. The image processing method according to claim 1, characterized in that: The calculating, according to the histogram matching algorithm, a mapping relationship between pixels corresponding to the grayscale image and pixels corresponding to the second image comprises: Performing equalization processing on the first histogram data of the grayscale image through a preset number of intervals, and performing equalization processing on the second histogram data of the second image through the preset number of intervals; The preset number of mapping relationships is determined according to the maximum pixel value of the equalized first histogram data of the grayscale image and the equalized second histogram data of the second image in the same interval.

4. The image processing method according to claim 1, characterized in that: The method further comprises: If there are uncalculated pixel values ​​in two adjacent mapping relationships, the mapping relationships of the uncalculated pixel values ​​are filled in by linear interpolation.

5. The image processing method according to claim 1, characterized in that: The method further comprises: For pixels whose grayscale values ​​in the grayscale image and the second image do not satisfy the pixel condition, the target interval is mapped according to the linear interpolation coefficient of the adjacent interval of the target interval where the pixel is located.

6. The image processing method according to claim 1, characterized in that: The guiding and optimizing the mapping result to determine the mapping image corresponding to the second image comprises: The second image is used as a guide image, and guided filtering is performed on the mapping result to determine the mapping image.

7. An image processing device, characterized in that: include: An image segmentation module, configured to perform panoramic image segmentation on the first image to obtain a plurality of image regions, and classify the plurality of image regions to obtain a plurality of categories; a mapping relationship determination module, configured to perform local color mapping on an image region where each category in the first image is located, according to a grayscale image corresponding to the first image and a second image, to determine a mapping relationship between pixels of the first image and pixels of the second image; An image mapping module, configured to map the second image according to the mapping relationship to obtain a mapping result, and to guide and optimize the mapping result to determine a mapping image corresponding to the second image; The step of performing local color mapping on the image region where each category in the first image is located separately to determine the mapping relationship between pixels of the first image and pixels of the second image includes: Selecting a target area from a plurality of image areas, performing histogram statistics on a grayscale image corresponding to the first image in the target area to obtain first histogram data, and performing histogram statistics on the second image in the target area to obtain second histogram data; Determining pixels corresponding to the grayscale image according to the first histogram data, and determining pixels corresponding to the second image according to the second histogram data; A mapping relationship between pixels corresponding to the grayscale image and pixels corresponding to the second image is calculated according to a histogram matching algorithm.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to perform the image processing method described in any one of claims 1-6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 6 is implemented.

Images