An image processing method, apparatus and electronic device

Abstract

A method, apparatus, and electronic device for image processing are disclosed, relating to the field of image processing technology. The method includes: fusing at least two first original images to obtain a first fused image; determining a first compensation value for a preset channel in the first fused image based on image data of the first fused image; performing compensation processing on at least two second original images based on the first compensation value to obtain at least two compensated second original images; and fusing the at least two compensated second original images to obtain a second fused image. The scheme disclosed herein performs compensation processing on each image to be fused before fusing multiple images, enabling the color of the obtained fused image to be close to the color of the real image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus and electronic device. Background Technology

[0002] Currently, multi-frame fusion sensors are widely used in the field of autonomous driving. These sensors acquire multiple original images simultaneously and fuse them to obtain a fused image, which the vehicle perception system then uses for processing. Because the colors of the images captured by the multi-frame fusion sensor vary depending on the light source illuminating the real scene, existing solutions use automatic white balance algorithms to adjust the gain of the R, G, and B channels of the fused image, correcting the color cast of the entire image to make it closer to the real image.

[0003] When multi-frame fusion sensors fuse images, the R, G, and B channel values ​​of the regions with brightness exceeding a preset threshold (i.e., brighter areas in the fused image) are truncated to the same value. This results in these regions appearing gray. After white balance correction, the fused regions, originally gray, develop a color cast due to the new correction gain. This makes it difficult to reconcile the color representation of the fused regions in the white-balanced image with the original gray regions, leading to a significant difference between the gray areas and the true image color. This color cast is common around vehicle headlights at night, where some areas are brighter, resulting in a large color cast in the generated fused image and potentially causing false detections by vehicle perception algorithms. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide an image processing method, apparatus, and electronic device. Because white balance adjustment is performed on each of the multiple frames to be fused before fusion, the color of the resulting fused image is close to the color of the real image.

[0005] The embodiments of this disclosure provide an image processing method, including: first, fusing at least two first original images to obtain a first fused image; second, determining a first compensation value for a preset channel in the first fused image based on image data of the first fused image; then, performing compensation processing on the preset channels of at least two second original images based on the first compensation value to obtain at least two compensated second original images; and finally, fusing at least two compensated second original images to obtain a second fused image.

[0006] According to another aspect of the present disclosure, an image processing apparatus is provided, comprising: a fusion module for fusing at least two first original images to obtain a first fused image; a determination module for determining compensation values ​​for preset channels in the first fused image based on image data of the first fused image; a compensation module for performing compensation processing on preset channels of at least two second original images based on the first compensation values ​​to obtain at least two compensated second original images; and a fusion module further for fusing at least two compensated second original images to obtain a second fused image.

[0007] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing any of the methods described above.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement any of the methods described above.

[0009] Based on the image processing method, apparatus, electronic device, and medium provided in the above embodiments of this disclosure, a first compensation value for a preset channel in the first fused image is calculated based on the image data of the first fused image. After compensating the preset channels of each second original image (i.e., the image to be fused) according to the first compensation value, the second original images are then fused. Since the first compensation value for the preset channels in the first fused image has been calculated based on the image data of the first fused image before fusing multiple frames of second original images, and compensation processing (such as white balance adjustment) has been performed on each second original image based on the first compensation value, the fused areas in the second original images with brightness greater than a preset threshold still appear as gray dots after compensation. Therefore, the color of the fused area of ​​the second fused image obtained by fusing based on the compensated second original images is close to the color of the real image; that is, the fused area of ​​the second fused image obtained by fusing based on the compensated second original images still appears gray, while the commonly used original scheme will appear colored (color cast). Since the fused area of ​​the second fused image obtained by fusing based on the compensated second original images still appears gray, it is closer to human visual perception and less likely to cause false detections of reference color indicators by the algorithm. Attached Figure Description

[0010] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 This is the system diagram to which this disclosure applies.

[0012] Figure 2 This is a flowchart of an exemplary embodiment of the present disclosure of an image processing method.

[0013] Figure 3 This is a flowchart of an image processing method provided in another exemplary embodiment of this disclosure.

[0014] Figure 4 This is a flowchart of an image processing method provided in another exemplary embodiment of this disclosure.

[0015] Figure 5 This is a flowchart of an image processing method provided in another exemplary embodiment of this disclosure.

[0016] Figure 6 This is a flowchart of an image processing method provided in another exemplary embodiment of this disclosure.

[0017] Figure 7 This is a flowchart of an image processing method provided in another exemplary embodiment of this disclosure.

[0018] Figure 8 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of the present disclosure.

[0019] Figure 9 This is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. Detailed Implementation

[0020] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0021] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0022] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0023] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0024] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0025] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0026] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0027] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0028] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0029] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0030] Application Overview

[0031] Currently, multi-frame fusion sensors (such as on-chip multi-frame fusion high dynamic range sensors) are widely used in the field of autonomous driving. Multi-frame fusion sensors can acquire multiple frames of original images at once, fuse these frames to obtain a fused image, and then send the fused image to an image processor.

[0032] To make the fused image closer to the real image, such as making the colors of the fused image closer to the colors of the real image, the image processor usually uses a white balance algorithm to perform white balance correction on the fused image sent by the multi-frame fusion sensor, so that the colors of the fused image after correction are close to the colors of the real image.

[0033] However, existing multi-frame fusion sensors, when fusing multiple original images, truncate the values ​​of the red (red, R), green (green, G), and blue (blue, B) channels in the image region where the brightness is greater than a preset threshold in the fused region image between two frames to the same value. Therefore, the color of the fused region in the fused image obtained by the multi-frame fusion sensor will appear gray.

[0034] This results in the image processor, after receiving fused images from a multi-frame fusion sensor, applying a white balance algorithm to correct the white balance of these images, causing the fused regions of the resulting image to appear as other colors instead of gray. Therefore, existing image processors, after applying white balance algorithms to correct fused images from multi-frame fusion sensors, produce fused images where the colors of the fused regions differ significantly from the colors of the actual images.

[0035] For example, a multi-frame fusion sensor can acquire multiple original images with different brightness levels (i.e., different exposure times) at once. After the multi-frame fusion sensor acquires multiple original images with different brightness levels, when fusing these images, the R, G, and B channel values ​​in the image regions where the brightness is greater than a preset threshold in the fusion region between two frames are truncated to the same value. That is, the R, G, and B channel values ​​in the image regions where the brightness is greater than the preset threshold in the fusion region between two frames are set to the same value, such as 200. Therefore, the color of the fusion region in the fusion image will appear gray. If the multiple original images correspond to a dynamic scene, and the fusion region in the fusion image is in a moving area, then the entire motion blur in that moving area will appear gray.

[0036] Because the colors of the multi-frame fusion sensor images (i.e., the colors corresponding to the fused image) vary depending on the light source illuminating the real scene, an image processor is needed to adjust the white balance of the fused image obtained from the multi-frame fusion sensor. This involves adjusting the gain of the red, green, and blue channels corresponding to the three primary colors in the fused image to correct the color cast of the entire fused image. However, after the image processor acquires the fused image from the multi-frame fusion sensor, when it adjusts the white balance, the fused area, which was originally gray, becomes discolored due to the new correction gain applied to it. This discoloration often occurs around vehicle headlights at night, causing vehicle perception algorithms to misdetect vehicles.

[0037] To address the issue that existing image processors, after applying white balance correction algorithms to fused images from multi-frame fusion sensors, result in significant color differences between the fused region and the actual image, this disclosure provides an image processing method. This method calculates a first compensation value for a preset channel in the first fused image based on its image data. Then, it compensates the preset channels of each second original image (the image to be fused) based on this first compensation value before fusing the second original images. Since the first compensation value for the preset channel in the first fused image is calculated based on the image data before fusing the multiple frames of second original images, and compensation processing (such as white balance adjustment) is performed on the preset channels of each second original image based on this first compensation value, the fused region in the second original image with a brightness greater than a preset threshold still appears as a gray dot after compensation. Consequently, the color of the fused region in the second fused image obtained by fusing based on the compensated second original image is close to the color of the actual image. That is, the fused region of the second fused image obtained by fusing based on the compensated second original image still appears gray, while the fused region in commonly used original schemes appears colored (color cast). Since the fused region of the second fused image obtained by fusing based on the compensated second original image is still gray, the gray color is not abrupt and is closer to human visual perception, and it is less likely to cause the algorithm to misdetect the reference color index.

[0038] Exemplary System

[0039] Figure 1 An exemplary system architecture 100 is shown for an apparatus that can be applied to an image processing method according to embodiments of the present disclosure.

[0040] like Figure 1As shown, the system architecture 100 may include an electronic device 101, a network 102, and a server 103. The electronic device 101 may include a multi-frame fusion sensor 1011 and an image processor 1012.

[0041] The electronic device 101 includes a multi-frame fusion sensor 1011, which can be used to fuse at least two first original images to obtain a first fused image. The electronic device 101 also includes an image processor 1012, which can be used to determine a first compensation value for a preset channel in the first fused image based on the image data of the first fused image.

[0042] The electronic device 101 includes a multi-frame fusion sensor 1011, which can also be used to perform compensation processing on at least two frames of the second original image based on a first compensation value to obtain at least two compensated second original images. The electronic device 101 also includes a multi-frame fusion sensor 1012, which can also be used to fuse at least two compensated second original images to obtain a second fused image.

[0043] Network 102 serves as a medium for providing a communication link between electronic device 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0044] Users can use electronic device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications, such as image processing applications and video surveillance applications, can be installed on electronic device 101.

[0045] Electronic device 101 can be various electronic devices, including but not limited to mobile terminals such as vehicle terminals, mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. The specific form of electronic device 101 is not limited in this embodiment.

[0046] In some examples, the electronic device 101 includes an image processor 1012 that can determine a first compensation value for a preset channel in the first fused image based on the image data of the first fused image. The electronic device 101 can also send the image data of the first fused image to a server 103.

[0047] Server 103 can be a server providing various services. For example, server 103 can determine a first compensation value for a preset channel in the first fused image based on the image data of the first fused image uploaded by electronic device 101. After server 103 determines the first compensation value for the preset channel in the first fused image, server 101 can send the first compensation value for the preset channel in the first fused image to electronic device 101. Thus, the multi-frame fusion sensor 1012 included in electronic device 101 can perform compensation processing on at least two frames of second original images based on the first compensation value to obtain at least two frames of compensated second original images, and fuse the at least two frames of compensated second original images to obtain a second fused image.

[0048] It should be noted that the image processing method provided in the embodiments of this disclosure can be executed by the server 103 or by the electronic device 101. Accordingly, the image processing device can be located in the server 103 or in the electronic device 101.

[0049] It should be understood that Figure 1 The number of electronic devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of electronic devices, networks, and servers can be included. If the initial raw image data does not need to be acquired remotely, the above system architecture may exclude the network and include only servers or electronic devices.

[0050] Exemplary methods

[0051] The image processing method provided in the embodiments of this disclosure is described below.

[0052] Figure 2 An image processing method provided in this disclosure embodiment can be applied to, for example... Figure 1 The electronic device shown may include a multi-frame fusion sensor and an image processor. For example... Figure 2 As shown, the method includes steps 201-204.

[0053] Step 201: Fuse at least two frames of the first original image to obtain the first fused image.

[0054] A multi-frame fusion sensor can fuse at least two first original images to obtain a first fused image.

[0055] Before fusing at least two first original images, the multi-frame fusion sensor can acquire at least two first original images, so that the multi-frame fusion sensor can fuse at least two first original images.

[0056] The first original image can be a raw (RAW) image.

[0057] In some examples, the brightness of at least two first original images can be different, meaning that the multi-frame fusion sensor can acquire at least two first original images at different exposure times.

[0058] For example, at least two first original images can include a long original image, a second shortest original image, and a shortest original image. The long original image is the original image with the longest exposure time among the at least two first original images, the shortest original image is the original image with the shortest exposure time among the at least two first original images, and the second shortest original image is the original image with an exposure time shorter than the exposure time corresponding to the long original image and longer than the exposure time corresponding to the second shortest original image among the at least two first original images. Since the dynamic range of original images with different exposure times is different, fusing the long original image, the second shortest original image, and the shortest original image can expand the dynamic range of the entire image.

[0059] Step 202: Based on the image data of the first fused image, determine the first compensation value of the preset channel in the first fused image.

[0060] After fusing at least two first original images to obtain a first fused image, the multi-frame fusion sensor can send the image data of the first fused image to an image processor. The image processor can determine a first compensation value for a preset channel in the first fused image based on the image data. Therefore, the multi-frame fusion sensor can perform compensation processing on at least two second original images based on the first compensation value, such as white balance adjustment on the at least two second original images.

[0061] In some examples, the image data of the first fused image can be the original (RAW) data corresponding to the first fused image. After obtaining the first fused image, the multi-frame fusion sensor can acquire the image data of the first fused image, that is, the original (RAW) data corresponding to the first fused image, so that the multi-frame fusion sensor can send the image data of the first fused image to the image processor.

[0062] In some examples, the first compensation value may include a third compensation value for the first channel and a fourth compensation value for the second channel in the first fused image. The first channel can be the red channel, green channel, or blue channel in the first fused image. The second channel is different from the first channel. For example, if the first channel is the red channel and the second channel is the blue channel, then the green channel in the first fused image can be called the third channel. In this embodiment, the first channel is the red channel, the second channel is the blue channel, and the third channel is the green channel in the first fused image, as an example for illustration.

[0063] In some examples, determining a first compensation value for a preset channel in the first fused image based on image data of the first fused image may include: the image processor first determining image gray areas in the first fused image; then, the image processor determining a third compensation value for the first channel of the image gray area based on data from a first channel and a third channel of multiple gray points in the image gray area; the image processor determining a fourth compensation value for the second channel of the image gray area based on data from a second channel and a third channel of multiple gray points in the image gray area; finally, the image processor determining the first compensation value for the preset channel in the first fused image based on the third and fourth compensation values.

[0064] The image gray area, also known as the white balance gray area, can be any region in the first fused image whose brightness is greater than a preset threshold; that is, the image gray area is the brighter region in the first fused image. The preset threshold can be determined according to actual conditions, and is not limited in this embodiment.

[0065] In some examples, the gray area in the first fused image can be one or more, that is, the brighter area in the first fused image can be one or more.

[0066] In some examples, an image gray area may include multiple gray points, meaning the image data corresponding to the image gray area may include image data corresponding to multiple gray points. The data for each gray point includes the first channel, the second channel, and the third channel data, which are the first, second, and third channel data of the image data corresponding to each gray point.

[0067] In some examples, the image processor determines a third compensation value for the first channel of the image gray area based on data from the first and third channels of multiple gray points in the image gray area. This may include: the image processor calculating the ratio of the data from the first and third channels of each gray point to obtain a first ratio for each gray point. Then, the image processor determines the third compensation value for the first channel of the image gray area based on the first ratio of each gray point.

[0068] The image processor described above determines a fourth compensation value for the second channel of the image gray area based on the data from the second and third channels of multiple gray points in the image gray area. This can include: the image processor calculating the ratio of the data from the second channel to the data from the third channel of each gray point to obtain a second ratio for each gray point. Then, the image processor determines the fourth compensation value for the second channel of the image gray area based on the second ratio of each gray point.

[0069] In some examples, when the first compensation value includes a third compensation value for the first channel and a fourth compensation value for the second channel in the first fused image, after the image processor determines the third compensation value, it can send the first compensation value to the multi-frame fusion sensor. The multi-frame fusion sensor can write the third and fourth compensation values ​​into a first register and a second register, respectively. Thus, the multi-frame fusion sensor can perform compensation processing on the first channel of each second original image through the first register and on the second channel of each second original image through the second register.

[0070] Step 203: Perform compensation processing on the preset channels of at least two frames of the second original image based on the first compensation value to obtain at least two compensated second original images.

[0071] After the image sensor determines the first compensation value of a preset channel in the first fused image based on the image data of the first fused image, the image processor can send the first compensation value to the multi-frame fusion sensor. The multi-frame fusion sensor can perform compensation processing (such as white balance adjustment) on at least two frames of the second original images based on the first compensation value to obtain at least two compensated second original images. Since the first compensation value of the preset channel in the first fused image has been calculated based on the image data of the first fused image before fusing the multiple frames of the second original images, and compensation processing (such as white balance adjustment) has been performed on each second original image based on the first compensation value, the color of the fusion area with brightness greater than a preset threshold in the second original image still appears as gray dots after compensation. Then, the multi-frame fusion sensor can fuse the at least two compensated second original images to obtain the second fused image.

[0072] In some examples, at least two second original images can be the multiple original images acquired by the multi-frame fusion sensor after acquiring at least two first original images. That is, at least two second original images can be the images to be fused in the next frame. In other words, the multi-frame fusion sensor can determine a first compensation value for a preset channel in the first fused image based on the first fused image corresponding to the at least two first original images. This first compensation value can then be used to compensate the at least two second original images acquired in the next frame. Afterward, the multi-frame fusion sensor can fuse the at least two compensated second original images to obtain the second fused image.

[0073] In other examples, at least two second original images can also be at least two first original images acquired by a multi-frame fusion sensor. That is, the multi-frame fusion sensor can determine a first compensation value for a preset channel in the first fusion image based on the first fusion image corresponding to the at least two first original image data. The at least two first original images can be compensated using this first compensation value. Afterward, the multi-frame fusion sensor can fuse the compensated at least two first original images to obtain a new first fusion image.

[0074] The preset channels of at least two frames of the second original image are the same as the preset channels in the first fused image, which will not be described again here.

[0075] In some examples, when the first compensation value includes the third compensation value of the first channel in the first fused image and the fourth compensation value of the second channel in the first fused image, and the multi-frame fusion sensor writes the third compensation value and the fourth compensation value into the first register and the second register respectively, the above-mentioned compensation processing of the preset channels of at least two frames of second original images based on the first compensation value to obtain at least two compensated second original images may include: the multi-frame fusion sensor performs compensation processing on the first channel of each second original image through the first register and performs compensation processing on the second channel of each second original image through the second register.

[0076] The first register can be the register for the first channel corresponding to each second original image in the multi-frame fusion sensor. That is, the data of the first channel corresponding to each second original image can be adjusted through the first register. It should be noted that when the third compensation value is the compensation value of the first channel in the first fused image, the first channel corresponding to the second original image is the same as the first channel in the first fused image. That is, when the first channel in the first fused image is the red channel, the first channel corresponding to the second original image is also the red channel.

[0077] The second register can be the register for the second channel corresponding to each second original image in the multi-frame fusion sensor. That is, the data of the second preset channel corresponding to each second original image can be adjusted through the second register. It should be noted that when the fourth compensation value is the fourth compensation value of the second channel in the first fused image, the second channel corresponding to the second original image is the same as the second channel in the first fused image. That is, when the second channel in the second fused image is the blue channel, the second channel corresponding to the second original image is also the blue channel.

[0078] Step 204: Fuse at least two frames of the compensated second original image to obtain the second fused image.

[0079] After the multi-frame fusion sensor performs compensation processing on the preset channels of at least two frames of the second original image based on the first compensation value to obtain at least two compensated second original images, the multi-frame fusion sensor can fuse the at least two compensated second original images to obtain the second fused image.

[0080] In some examples, after the multi-frame fusion sensor of an electronic device obtains the second fused image, the multi-frame fusion sensor can send the second fused image to the image processor, so that the image processor can process the second fused image. For example, the image processor can adjust the white balance of the second fused image using a white balance algorithm (also known as white balance correction), so that the image processed by the image processor can be displayed.

[0081] The solution provided in this embodiment calculates a first compensation value for a preset channel in the first fused image based on the first fused image. After compensating each second original image (i.e., the image to be fused) based on this first compensation value, the second original images are then fused. Since the first compensation value for the preset channel in the first fused image has been calculated based on the image data of the first fused image before fusing multiple frames of the second original images, and compensation processing (such as white balance adjustment) has been performed on each second original image based on the first compensation value, the fused areas in the second original images where the brightness is greater than a preset threshold still appear as gray dots after compensation. Therefore, the color of the fused area in the second fused image obtained by fusing based on the compensated second original images is close to the color of the real image; that is, the fused area in the second fused image obtained by fusing based on the compensated second original images still appears gray.

[0082] In other words, the solution provided by the embodiments of this disclosure combines the white balance algorithm of the backend (such as an image processor) with white balance compensation of the single-frame image before fusion by the multi-frame fusion processor. This eliminates the color cast problem in the fusion region of the fused image from the source (i.e., the single-frame image before fusion), and better balances the white balance performance of the overall image with the color performance of the fusion region. Therefore, when the brightness of some image areas around vehicle lights is high at night, it can avoid a large color cast in the generated fused image, improving the accuracy of vehicle perception algorithm detection.

[0083] In some embodiments, such as Figure 3 As shown above, in the above Figure 2 Based on the illustrated embodiment, step 203 may include steps 301-304.

[0084] Step 301: Determine the image gray area in the first fused image.

[0085] After the image processor acquires the first fused image sent by the multi-frame fusion sensor, the image processor of the electronic device can determine the image gray areas in the first fused image. An image gray area can include multiple gray points. The image data corresponding to the image gray area can include image data corresponding to multiple gray points. The data of the first channel, the second channel, and the third channel of each gray point constitute the first channel, second channel, and third channel data of the image data corresponding to each gray point.

[0086] In some examples, determining the image gray areas in the first fused image may include: determining the average ratio of the data of the first channel and the data of the third channel for each gray point in multiple regions of the first fused image, and the average ratio of the data of the second channel and the data of the third preset channel for each gray point. The first channel may be the red channel corresponding to a gray point, the second channel may be the blue channel corresponding to a gray point, and the third channel may be the green channel corresponding to a gray point.

[0087] Next, the average ratio of the data from the first channel and the second preset channel for each gray point, and the average ratio of the data from the third channel and the second channel for each gray point, are determined, and their weights are calculated based on their distances from the Planck curve in the white balance statistical coordinate system. Based on the weights corresponding to the average ratios of the data from the first and second channels for each gray point, and the average weights corresponding to the average ratios of the data from the third and second channels for each gray point, image gray areas in the first fused image are determined. Areas with weights greater than a preset threshold are considered image gray areas in the first fused image.

[0088] For example, taking the first channel as the red channel (R) corresponding to a gray point, the second channel as the blue channel (B) corresponding to a gray point, and the third channel as the green channel (G) corresponding to a gray area as an example, determining the image gray area in the first fused image can include: dividing the first fused image into M x N regions, such as a typical 16x16 block configuration. Then, calculating the average R / G and average B / G values ​​for each region. Next, placing the statistical results for each region (i.e., the average R / G and average B / G values) into a white balance statistical coordinate system, and assigning a weight value to each point based on the distance between the statistical result of each region and the Planck curve; the point with the larger weight is the image gray area in the first fused image.

[0089] The white balance statistical coordinate system and Planck curve can be determined based on the assumption of a "grayscale world" using a color temperature calibration method. Specifically, this involves capturing images using a multi-frame fusion sensor under different high and low color temperature environments, statistically analyzing the R / G and B / G distributions of the gray areas in the captured images, and determining the Planck curve based on these distributions. A white balance statistical coordinate system is then created with R / G as the x-axis and B / G as the y-axis. Next, the statistical values ​​at different color temperatures are fitted to a Planck curve, assigning different weights to points based on their distance from the curve; closer points have higher weights, and points with higher weights represent the gray areas in the image.

[0090] It should be noted that the gray area in the first fused image can be one or multiple.

[0091] Step 302: Based on the data of the first channel and the data of the third channel of multiple gray points in the image gray area, determine the third compensation value of the first channel of the image gray area.

[0092] After the image processor determines the image gray area in the first fused image, the image processor can determine the third compensation value of the first channel of the image gray area based on the data of the first channel and the data of the third channel of multiple gray points in the image gray area.

[0093] It should be noted that when there are multiple image gray areas in the first fused image, that is, when the first fused image includes multiple image gray areas, the above-mentioned determination of the third compensation value of the first channel of the image gray area based on the data of the first channel and the data of the third channel of multiple gray points in the image gray area may include determining the third compensation value of the first channel of each image gray area based on the data of the first channel and the data of the third channel of multiple gray points in each image gray area.

[0094] Step 303: Based on the data of the second channel and the data of the third channel of multiple gray points in the image gray area, determine the fourth compensation value of the second channel of the image gray area.

[0095] After the image processor determines the image gray area in the first fused image, the image processor can determine the fourth compensation value of the second channel of the image gray area based on the data of the second channel and the data of the third channel based on multiple gray points in the image gray area.

[0096] It should be noted that when there are multiple image gray areas in the first fused image, that is, when the first fused image includes multiple image gray areas, the above-mentioned determination of the fourth compensation value of the second channel of the image gray area based on the data of the second channel and the data of the third channel of multiple gray points in the image gray area may include determining the fourth compensation value of the second channel of the image gray area based on the data of the second channel and the data of the third channel of multiple gray points in each image gray area.

[0097] It should be noted that the execution order of steps 302 and 303 in this embodiment is not limited. Step 302 can be executed first, followed by step 303, or step 303 can be executed first, followed by step 302, or both steps 302 and 303 can be executed simultaneously. This embodiment illustrates the simultaneous execution of steps 302 and 303 as an example.

[0098] Step 304: Based on the third compensation value and the fourth compensation value, determine the first compensation value of the preset channel in the first fused image.

[0099] After the image processor determines the third compensation value and the fourth compensation value, the image processor can determine the first compensation value of the preset channel in the first fused image based on the third compensation value and the fourth compensation value.

[0100] In some examples, the first compensation value may include a third compensation value and a fourth compensation value, that is, the first compensation value of a preset channel in the first fused image, including the third compensation value of the first channel of the image gray area and the fourth compensation value of the second channel of the image gray area.

[0101] It should be noted that when there are multiple image gray areas in the first fused image, that is, when the first fused image includes multiple image gray areas, the third compensation value of the first channel of the image gray area is multiple, and the fourth compensation value of the second channel of the image gray area is multiple. The above-mentioned determination of the first compensation value of the preset channel in the first fused image based on the third compensation value and the fourth compensation value may include weighted averaging of multiple third compensation values ​​to obtain the third compensation value of the first channel in the first fused image, and weighted averaging of multiple fourth compensation values ​​to obtain the fourth compensation value of the second channel in the first fused image.

[0102] The solution provided in this embodiment can accurately determine the first compensation value of a preset channel in the first fused image by using the data of each gray point in the gray area of ​​the first fused image. Therefore, after compensating the preset channels in each second original image (i.e., the image to be fused) according to this accurate first compensation value, the compensated color of the fused region in the second original image where the brightness is greater than a preset threshold still appears as gray points. This eliminates the color cast problem in the fused region of the fused image and improves the accuracy of vehicle perception algorithm detection.

[0103] In some embodiments, such as Figure 4 As shown above, in the above Figure 3 Based on the illustrated embodiment, step 302 may include steps 401-402, and step 303 may include steps 403-404.

[0104] Step 401: Calculate the ratio of the data from the first channel to the data from the third channel of each gray point to obtain the first ratio of each gray point.

[0105] After the image processor determines the gray areas in the first fused image, it can calculate the ratio of the data from the first channel to the data from the third channel of each gray point, thus obtaining the first ratio for each gray point. That is, there can be multiple first ratios.

[0106] In this embodiment, the first channel is the red channel corresponding to a gray point, the second channel is the blue channel corresponding to a gray point, and the third channel is the green channel corresponding to a gray point, as an example for illustration. That is, the data of the first channel is the data corresponding to the red channel in the image data corresponding to the gray point, the data of the second channel is the data corresponding to the blue channel in the image data corresponding to the gray point, and the data of the third channel is the data corresponding to the green channel in the image data corresponding to the gray point.

[0107] It should be noted that when there are multiple image gray areas in the first fused image, that is, when the first fused image includes multiple image gray areas, the above calculation of the ratio of the data of the first channel and the data of the third channel of each gray point to obtain the first ratio of each gray point may include calculating the ratio of the data of the first channel and the data of the third channel of each gray point in each image gray area to obtain the first ratio of each gray point in each image gray area.

[0108] Step 402: Determine the third compensation value of the first channel of the image gray area based on the first ratio of each gray point.

[0109] After the image processor calculates the ratio of the data from the first channel to the data from the third channel of each gray point and obtains the first ratio of each gray point, the image processor can determine the third compensation value of the first channel of the image gray area based on the first ratio of each gray point.

[0110] In some examples, determining a third compensation value for a first channel of an image gray area based on a first ratio of each gray point may include: averaging the first ratios of each gray point to obtain a first average value; and then determining the third compensation value for the first channel of the image gray area based on the first average value.

[0111] In other examples, determining the third compensation value for the first channel of the image gray area based on the first ratio of each gray point may include: obtaining a second compensation value corresponding to a third original image, which may be a previous historical frame image of the first original image. The second compensation value corresponding to the third original image may include a fifth compensation value for the first channel in the third original image. Then, the first ratios of each gray point are averaged to obtain a first average value. Next, the first average value is divided by the fifth compensation value to obtain a third ratio. Finally, based on the third ratio, the third compensation value for the first channel of the image gray area is determined.

[0112] It should be noted that when there are multiple image gray areas in the first fused image, i.e., the first fused image includes multiple image gray areas, the above-mentioned determination of the third compensation value of the first channel of the image gray area based on the first ratio of each gray point can include averaging the first ratio of each gray point in each image gray area to obtain the first average value of each image gray area. Based on the first average value of each image gray area, the third compensation value of the first channel of each image gray area is determined.

[0113] Step 403: Calculate the ratio of the data from the second channel to the data from the third channel of each gray point to obtain the second ratio of each gray point.

[0114] After the image processor determines the gray areas in the first fused image, it can obtain a second ratio for each gray point based on the ratio of the data from the second channel to the data from the third channel. That is, there can be multiple second ratios.

[0115] In this embodiment, the second channel is the blue channel corresponding to a gray point, the second pre-channel is the blue channel corresponding to a gray point, and the third channel is the green channel corresponding to a gray point, as an example for illustration. That is, the data of the first channel is the data corresponding to the red channel in the image data corresponding to the gray point, the data of the second channel is the data corresponding to the blue channel in the image data corresponding to the gray point, and the data of the third channel is the data corresponding to the green channel in the image data corresponding to the gray point.

[0116] It should be noted that when there are multiple image gray areas in the first fused image, that is, when the first fused image includes multiple image gray areas, the above calculation of the ratio of the data of the second channel and the data of the third channel of each gray point to obtain the second ratio of each gray point may include calculating the ratio of the data of the second channel and the data of the third channel of each gray point in each image gray area to obtain the second ratio of each gray point in each image gray area.

[0117] Step 404: Determine the fourth compensation value of the second channel of the image gray area based on the second ratio of each gray point.

[0118] After the image processor calculates the ratio of the data of the second channel to the data of the third channel of each gray point and obtains the second ratio of each gray point, the image processor can determine the fourth compensation value of the second channel of the gray area of ​​the image based on the second ratio of each gray point.

[0119] In some examples, determining the fourth compensation value of the second channel of the image gray area based on the second ratio of each gray point may include: averaging the second ratios of each gray point to obtain a second average value; and determining the fourth compensation value of the second channel of the image gray area based on the second average value.

[0120] In other examples, determining the third compensation value of the first channel of the image gray area based on the first ratio of each gray point may further include: obtaining the second compensation value corresponding to the third original image, which may be a previous historical frame image of the first original image. The second compensation value corresponding to the third original image may also include the sixth compensation value of the second channel in the third original image. Then, the second ratios of each gray point are averaged to obtain a second average value. Next, the second average value is divided by the fifth compensation value to obtain the third ratio. Finally, based on the third ratio, the fourth compensation value of the second channel of the image gray area is determined.

[0121] It should be noted that when there are multiple image gray areas in the first fused image, i.e., the first fused image includes multiple image gray areas, the above-mentioned determination of the third compensation value of the second channel of the image gray area based on the second ratio of each gray point can include averaging the second ratio of each gray point in each image gray area to obtain the second average value of each image gray area. Based on the second average value of each image gray area, the fourth compensation value of the second channel of each image gray area is determined.

[0122] It should be noted that the execution order of steps 401-402 and 403-404 in this embodiment is not limited. Steps 401-402 can be executed first, followed by steps 403-404; steps 403-404 can be executed first, followed by steps 401-402; or steps 401-402 and 403-404 can be executed simultaneously. This embodiment illustrates the simultaneous execution of steps 401-402 and 403-404 as an example.

[0123] The solution provided in this embodiment can accurately determine the compensation value of the first channel of the image gray area by using the data of the first channel and the third channel of each gray point in the image gray area of ​​the first fused image. Similarly, it can accurately determine the compensation value of the second channel of the image gray area by using the data of the second channel and the third channel of each gray point in the image gray area of ​​the first fused image. Therefore, based on these accurate compensation values ​​for the first and second channels of the image gray area, the compensation values ​​for the preset channels in the first fused image are more accurate. Thus, after compensating the preset channels in each of the second original images (i.e., the images to be fused) according to the compensation values ​​of the preset channels in the first fused image, the color cast problem in the fusion region of the fused image can be accurately eliminated, further improving the accuracy of vehicle perception algorithm detection.

[0124] In some embodiments, such as Figure 5 As shown above, in the above Figure 4 Based on the illustrated embodiment, step 402 may include steps 501-502, and step 404 may include steps 503-504.

[0125] Step 501: Calculate the average of the first ratios of each gray point to obtain the first average value.

[0126] After the image processor calculates the ratio of the data from the first channel to the data from the third channel of each gray point to obtain the first ratio of each gray point, the image processor can average the first ratio of each gray point to obtain the first average value.

[0127] It should be noted that when there are multiple image gray areas in the first fused image, that is, when the first fused image includes multiple image gray areas, the above-mentioned averaging of the first ratio of each gray point to obtain the first average value may include averaging the first ratio of each gray point in each image gray area to obtain the first average value of each image gray area, thereby determining the third compensation value of the first channel of each image gray area based on the first average value of each image gray area.

[0128] Step 502: Based on the first average value, determine the third compensation value of the first channel of the image gray area.

[0129] After the image processor averages the first ratios of each gray point to obtain a first average value, it can determine a third compensation value for the first channel of the image gray area based on the first average value. For example, the image processor can determine the third compensation value for the first channel of the image gray area as the reciprocal of the first average value.

[0130] It should be noted that when there are multiple image gray areas in the first fused image, i.e., the first fused image includes multiple image gray areas, the above-mentioned averaging of the first ratio of each gray point to obtain the first average value can include averaging the first ratio of each gray point in each image gray area to obtain the first average value for each image gray area. Then, a weighted average can be performed on the first average value of each image gray area to obtain a weighted average first average value. Based on the weighted average first average value, the third compensation value of the first channel of the image gray area is determined. For example, the image processor can determine the third compensation value of the first channel of the image gray area as the reciprocal of the weighted average first average value.

[0131] Step 503: Average the second ratio of each gray point to obtain the second average value.

[0132] After the image processor calculates the ratio of the data from the second channel to the data from the third channel of each gray point and obtains the second ratio of each gray point, the image processor can average the second ratio of each gray point to obtain the second average value.

[0133] It should be noted that when there are multiple image gray areas in the first fused image, that is, when the first fused image includes multiple image gray areas, the above-mentioned averaging of the second ratio of each gray point to obtain the second average value may include averaging the second ratio of each gray point in each image gray area to obtain the second average value of each image gray area, thereby determining the fourth compensation value of the second channel of each image gray area based on the second average value of each image gray area.

[0134] Step 504: Based on the second average value, determine the fourth compensation value of the second channel of the image gray area.

[0135] After the image processor averages the second ratio of each gray point to obtain a second average value, it can determine a third compensation value for the second channel of the image gray area based on this second average value. For example, the image processor can determine the third compensation value for the second channel of the image gray area as the reciprocal of the second average value.

[0136] It should be noted that when there are multiple image gray areas in the first fused image, i.e., the first fused image includes multiple image gray areas, the above-mentioned averaging of the second ratios of each gray point to obtain the second average value can include averaging the second ratios of each gray point in each image gray area to obtain the second average value for each image gray area. Then, a weighted average can be performed on the second average value of each image gray area to obtain a weighted average second average value. Based on the weighted average second average value, the third compensation value for the second channel of the image gray area is determined. For example, the image processor can determine the fourth compensation value for the second channel of the image gray area as the reciprocal of the weighted average second average value.

[0137] It should be noted that the execution order of steps 501-502 and 503-504 in this embodiment is not limited. Steps 501-502 can be executed first, followed by steps 503-504; or steps 503-504 can be executed first, followed by steps 501-502; or steps 501-502 and 503-504 can be executed simultaneously. This embodiment illustrates the simultaneous execution of steps 501-502 and 503-504 as an example.

[0138] The scheme provided in this embodiment can determine the compensation value of the first channel of the image gray area by averaging the first ratios of the data from the first channel and the third channel of each gray point in the image gray area of ​​the first fused image, and can determine the compensation value of the second channel of the image gray area by averaging the second ratios of the data from the second channel and the third channel of each gray point in the image gray area of ​​the first fused image. Therefore, based on the compensation values ​​of the first and second channels of the image gray area, the first compensation value of a preset channel in the first fused image can be determined. By compensating each second original image (i.e., the image to be fused) according to this first compensation value before fusion, the color cast problem in the fusion region of the fused image can be accurately eliminated, further improving the accuracy of vehicle perception algorithm detection.

[0139] In some embodiments, such as Figure 6 As shown above, in the above Figure 4 Based on the illustrated embodiment, step 402 may include steps 601-604, and step 404 may include steps 605-608.

[0140] Step 601: Obtain the fifth compensation value of the first channel of the third original image, where the third original image is the previous historical frame image of the first original image.

[0141] After the image processor calculates the ratio of the data from the first channel to the data from the third channel of each gray point to obtain the first ratio for each gray point, the image processor can obtain the second compensation value corresponding to the third original image. The third original image can be the previous historical frame image of the first original image. The second compensation value corresponding to the third original image can include the fifth compensation value of the first channel in the third original image.

[0142] The second compensation value corresponding to the third original image may include the fifth compensation value of the first channel and the sixth compensation value of the second channel in the third original image. In some examples, there may be one or more third original images. That is, the fifth compensation value of the first channel in the third original image may be a compensation value for compensating the first channel corresponding to at least two frames of the third original image. The sixth compensation value of the second channel in the third original image may be a compensation value for compensating the second preset channel of at least two frames of the third original image. In this embodiment, the first channel in the third original image is the red channel and the second channel in the third original image is the blue channel for illustrative purposes.

[0143] Step 602: Calculate the average of the first ratios of each gray point to obtain the first average value.

[0144] After the image processor calculates the ratio of the data from the first channel to the data from the third channel of each gray point to obtain the first ratio of each gray point, the image processor can average the first ratio of each gray point to obtain the first average value.

[0145] It should be noted that when there are multiple image gray areas in the first fused image, that is, when the first fused image includes multiple image gray areas, the above-mentioned averaging of the first ratio of each gray point to obtain the first average value can include averaging the first ratio of each gray point in each image gray area to obtain the first average value of each image gray area. Then, a weighted average is performed on the first average value of each image gray area to obtain the weighted average first average value.

[0146] Step 603: Divide the first average value by the fifth compensation value to obtain the third ratio.

[0147] After the image processor averages the first ratio of each gray point to obtain the first average value, the image processor can divide the first average value by the fifth compensation value to obtain the third ratio.

[0148] It should be noted that when there are multiple image gray areas in the first fused image, i.e., the first fused image includes multiple image gray areas, the above-mentioned method of obtaining the first average value based on the first ratio of each gray point can include averaging the first ratio of each gray point in each image gray area to obtain the first average value for each image gray area. Then, a weighted average is performed on the first average value of each image gray area to obtain a weighted average first average value. Finally, the weighted average first average value can be divided by the fifth compensation value to obtain the third ratio.

[0149] Step 604: Based on the third ratio, determine the third compensation value of the first channel of the image gray area.

[0150] After the image processor obtains the third ratio, it can determine the third compensation value for the first channel of the image gray area based on the third ratio. For example, the image processor can determine the third compensation value for the first channel of the image gray area as the reciprocal of the third ratio.

[0151] When the image processor obtains the third ratio, it has already removed the fifth compensation value of the first channel in the third original image. That is, when the image processor obtains the third ratio, it can eliminate the influence of the fifth compensation value of the first channel corresponding to the previous historical frame image of the first original image. Therefore, the third compensation value of the first channel of the image gray area determined by the image processor based on the third ratio can eliminate the influence of the fifth compensation value of the first channel corresponding to the previous historical frame image of the first original image.

[0152] Step 605: Obtain the sixth compensation value of the second channel of the third original image.

[0153] After the image processor calculates the ratio of the data from the second channel to the data from the third channel of each gray point, obtaining the second ratio for each gray point, the image processor can obtain the second compensation value corresponding to the third original image. The second compensation value corresponding to the third original image may include the sixth compensation value of the second channel in the third original image.

[0154] Step 606: Average the second ratio of each gray point to obtain the second average value.

[0155] After the image processor calculates the ratio of the data from the second channel to the data from the third channel of each gray point and obtains the second ratio of each gray point, the image processor can average the second ratio of each gray point to obtain the second average value.

[0156] It should be noted that when there are multiple image gray areas in the first fused image, that is, when the first fused image includes multiple image gray areas, the above-mentioned averaging of the second ratio of each gray point to obtain the second average value can include averaging the second ratio of each gray point in each image gray area to obtain the second average value of each image gray area. Then, a weighted average is performed on the second average values ​​of each image gray area to obtain the weighted average second average value.

[0157] Step 607: Divide the second average value by the sixth compensation value to obtain the fourth ratio.

[0158] After the image processor averages the second ratio of each gray point to obtain the second average value, the image processor can divide the second average value by the sixth compensation value to obtain the fourth ratio.

[0159] It should be noted that when there are multiple image gray areas in the first fused image, i.e., the first fused image includes multiple image gray areas, the above-mentioned method of obtaining the second average value based on the second ratio of each gray point can include averaging the second ratio of each gray point in each image gray area to obtain the second average value for each image gray area. Then, a weighted average is performed on the second average values ​​of each image gray area to obtain a weighted average second average value. Finally, the weighted average second average value can be divided by the sixth compensation value to obtain the fourth ratio.

[0160] Step 608: Based on the fourth ratio, determine the fourth compensation value of the second channel of the image gray area.

[0161] After the image processor obtains the fourth ratio, it can determine the fourth compensation value for the second channel of the image gray area based on the fourth ratio. For example, the image processor can determine the fourth compensation value for the second channel of the image gray area as the reciprocal of the fourth ratio.

[0162] When the image processor obtains the fourth ratio, it has already removed the sixth compensation value of the second channel in the third original image. That is, when the image processor obtains the fourth ratio, it can eliminate the influence of the sixth compensation value of the second channel corresponding to the previous historical frame image of the first original image. Therefore, the fourth compensation value of the second channel of the image gray area determined by the image processor based on the fourth ratio can eliminate the influence of the sixth compensation value of the second channel corresponding to the previous historical frame image of the first original image.

[0163] It should be noted that the execution order of steps 601-604 and 605-608 in this embodiment is not limited. Steps 601-604 can be executed first, followed by steps 605-608; steps 605-608 can be executed first, followed by steps 601-604; or steps 601-604 and 605-608 can be executed simultaneously. This embodiment illustrates the simultaneous execution of steps 601-604 and 605-608 as an example.

[0164] The solution provided in this embodiment can remove the fifth compensation value of the first channel in the third original image when calculating the third compensation value of the first channel and the fourth compensation value of the second channel in the image gray area. Therefore, when determining the first compensation value of a preset channel in the first fused image based on the third compensation value of the first channel and the fourth compensation value of the second channel in the image gray area, the influence of the fifth compensation value corresponding to the previous historical frame image of the first original image can be eliminated. Thus, the first compensation value determined by the image processor based on the third compensation value of the first channel and the fourth compensation value of the second channel in the image gray area can eliminate the influence of the compensation value corresponding to the previous historical frame image of the first original image. After compensating each second original image (i.e., the image to be fused) according to this first compensation value before fusion, the influence of the previous historical frame image of the first original image can be eliminated. This can more accurately eliminate the color cast problem in the fusion region of the fused image, further improving the accuracy of vehicle perception algorithm detection.

[0165] In some embodiments, such as Figure 7 As shown above, in the above Figure 3 Based on the illustrated embodiment, step 304 may be followed by step 701, and step 203 may include step 702.

[0166] Step 701: Write the third compensation value and the fourth compensation value into the first register and the second register, respectively.

[0167] After the image sensor determines the first compensation value of a preset channel in the first fused image based on the image data of the first fused image, the first compensation value including the third compensation value of the first channel in the first fused image and the fourth compensation value of the second channel in the first fused image, the image processor can send the third compensation value and the fourth compensation value to the multi-frame fusion sensor included in the electronic device.

[0168] The multi-frame fusion sensor can write the third compensation value and the fourth compensation value into the first register and the second register respectively, so that the multi-frame fusion sensor can perform compensation processing on the preset channels of at least two frames of the second original image based on the first register and the second register.

[0169] The first register is the register for the first channel corresponding to each second original image in the multi-frame fusion sensor of the electronic device, and the second register is the register for the second channel corresponding to each original image in the multi-frame fusion sensor of the electronic device. That is, there can be multiple first registers and multiple second registers.

[0170] Step 702: Compensate the first channel of each second original image through the first register, and compensate the second channel of each second original image through the second register to obtain at least two frames of compensated second original images.

[0171] After the multi-frame fusion sensor writes the third compensation value and the fourth compensation value into the first register and the second register respectively, the multi-frame fusion sensor can perform compensation processing on the preset channel (i.e., the first channel) in each second original image through the first register, and perform compensation processing on the preset channel (i.e., the second channel) in each second original image through the second register to obtain at least two compensated second original images.

[0172] After the multi-frame fusion sensor obtains at least two compensated second original images, it can fuse at least two compensated second original images to obtain a second fused image.

[0173] The solution provided in this embodiment writes the third compensation value of the first channel in the first fused image into a first register and the fourth compensation value of the second channel into a second register. This allows compensation processing of the first channel of each second original image through the first register and compensation processing of the second channel of each second original image through the second register, resulting in at least two compensated second original images. These at least two compensated second original images can then be fused to obtain a second fused image. Because the first and second channels of each second original image are compensated before fusing multiple frames of second original images, the fused areas in the second original images with brightness exceeding a preset threshold still appear as gray dots after compensation. Therefore, the color of the fused area in the second fused image obtained by fusing based on the compensated second original images is close to the color of the real image; that is, the fused area in the second fused image obtained by fusing based on the compensated second original images still appears gray.

[0174] Exemplary device

[0175] Figure 8An image processing apparatus provided in the embodiments of this disclosure, such as Figure 8 As shown, the image processing device 8 includes a fusion module 801, a determination module 802, and a compensation module 803.

[0176] The fusion module 801 is used to fuse at least two frames of the first original image to obtain a first fused image.

[0177] The determining module 802 is used to determine the first compensation value of a preset channel in the first fused image based on the image data of the first fused image.

[0178] The compensation module 803 is used to perform compensation processing on a preset channel of at least two frames of the second original image based on a first compensation value, so as to obtain at least two frames of compensated second original image.

[0179] The fusion module 801 is also used to fuse at least two frames of compensated second original images to obtain a second fused image.

[0180] In some examples, the determining module 802 includes: a first determining unit, a first calculating unit, a second determining unit, a second calculating unit, a third determining unit, and a fourth determining unit. The first determining unit is used to determine image gray areas in the first fused image. The first calculating unit is used to calculate the ratio of the first channel data and the second channel data of each gray point based on the data of the first channel and the third channel data of multiple gray points in the image gray area, obtaining a first ratio value for each gray point. The second determining unit is used to determine a third compensation value for the first channel of the image gray area based on the first ratio value of each gray point. The second calculating unit is used to calculate the ratio of the second channel data and the third channel data of each gray point based on the data of the second channel and the third channel data of multiple gray points in the image gray area, obtaining a second ratio value for each gray point. The third determining unit is used to determine a fourth compensation value for the second channel of the image gray area based on the second ratio value of each gray point. The fourth determining unit is used to determine a first compensation value based on the third compensation value and the fourth compensation value.

[0181] In some examples, the second determining unit includes a first determining subunit, a second determining subunit, a third determining subunit, and a fourth determining subunit. The first determining subunit is used to average the first ratio of each gray point to obtain a first average value. The second determining subunit is used to determine a third compensation value for a first channel of the image gray area based on the first average value.

[0182] The third determining unit includes a third determining subunit and a fourth determining subunit. The third determining subunit is used to average the second ratio of each gray point to obtain a second average value. The second determining subunit is used to determine the fourth compensation value of the second channel of the image gray area based on the second average value.

[0183] In some examples, the second determining unit includes a fifth determining subunit, a sixth determining subunit, a seventh determining subunit, and an eighth determining subunit. The fifth determining subunit is used to obtain the fifth compensation value of the first channel of the third original image, where the third original image is the previous historical frame image of the first original image. The sixth determining subunit is used to average the first ratio of each gray point to obtain a first average value. The seventh determining subunit is used to divide the first average value by the fifth compensation value to obtain a third ratio. The eighth determining subunit, based on the third ratio, determines the third compensation value of the first channel of the image gray area.

[0184] The third determining unit includes a ninth determining subunit, a tenth determining subunit, an eleventh determining subunit, and a twelfth determining subunit. The ninth determining subunit is used to obtain the sixth compensation value of the second channel of the third original image. The tenth determining subunit is used to average the second ratio of each gray point to obtain a second average value. The eleventh determining subunit is used to divide the second average value by the sixth compensation value to obtain a fourth ratio. The twelfth determining subunit is used to determine the fourth compensation value of the second channel of the image gray area based on the fourth ratio.

[0185] In some examples, the compensation module 803 includes a first compensation unit and a second compensation unit. The first adjustment unit is used to write the third compensation value and the fourth compensation value into a first register and a second register, respectively. The second compensation unit is used to perform compensation processing on the first channel of each second original image through the first register and compensation processing on the second channel of each second original image through the second register, to obtain at least two frames of compensated second original images.

[0186] Exemplary electronic devices

[0187] Figure 9 A structural diagram of an electronic device provided in an embodiment of this disclosure, such as... Figure 9 As shown, the electronic device 900 includes one or more processors 901 and memory 902.

[0188] The processor 901 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 900 to perform desired functions.

[0189] The memory 902 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 901 may execute the program instructions to implement the instruction generation methods of the various embodiments of this disclosure described above and / or other desired functions.

[0190] In one example, the electronic device 900 may also include an input device 903 and an output device 904, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0191] Of course, for the sake of simplicity, Figure 9 Only some of the components of the electronic device 9 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 9 may include any other suitable components depending on the specific application.

[0192] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0193] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0194] Exemplary computer program products and computer-readable storage media

[0195] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0196] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0197] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0198] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0199] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0200] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0201] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. An image processing method, comprising: At least two frames of the first original image are fused to obtain the first fused image; Based on the image data of the gray area in the first fused image, a first compensation value for a preset channel in the first fused image is determined; wherein, the gray area is the region in the first fused image whose brightness is greater than a preset threshold. Based on the first compensation value, at least two frames of the second original image are compensated for a preset channel to obtain at least two compensated second original images. The second fused image is obtained by fusing the at least two compensated second original images.

2. The method according to claim 1, wherein, The step of determining a first compensation value for a preset channel in the first fused image based on image data of the gray areas in the first fused image includes: Based on the data of the first channel and the data of the third channel of multiple gray points in the image gray area, a third compensation value of the first channel of the image gray area is determined; Based on the data of the second channel and the data of the third channel of multiple gray points in the image gray area, a fourth compensation value of the second channel of the image gray area is determined; Based on the third compensation value and the fourth compensation value, the first compensation value of the preset channel in the first fused image is determined.

3. The method according to claim 2, wherein, The determination of the third compensation value for the first channel of the image gray area based on the data of the first channel and the data of the third channel of multiple gray points in the image gray area includes: Calculate the ratio of the data from the first channel to the data from the third channel of each gray point to obtain the first ratio of each gray point; Based on the first ratio of each gray point, determine the third compensation value of the first channel of the image gray area; The determination of the fourth compensation value for the second channel of the image gray area based on the data of the second channel of multiple gray points in the image gray area and the data of the third channel includes: Calculate the ratio of the data from the second channel to the data from the third channel of each gray point to obtain the second ratio of each gray point; Based on the second ratio of each gray point, a fourth compensation value for the second channel of the image gray area is determined.

4. The method according to claim 3, wherein, Determining the third compensation value of the first channel of the image gray area based on the first ratio of each of the gray points includes: The first average value is obtained by averaging the first ratios of the gray points. Based on the first average value, a third compensation value for the first channel of the image gray area is determined; Determining the fourth compensation value of the second channel of the image gray area based on the second ratio of each of the gray points includes: The second average value is obtained by averaging the second ratios of all the gray points. Based on the second average value, a fourth compensation value for the second channel of the image gray area is determined.

5. The method according to claim 3, wherein, Determining the third compensation value of the first channel of the image gray area based on the first ratio of each of the gray points includes: Obtain the fifth compensation value of the first channel of the third original image, wherein the third original image is the previous historical frame image of the first original image; The first average value is obtained by averaging the first ratios of the gray points. Divide the first average value by the fifth compensation value to obtain the third ratio; Based on the third ratio, a third compensation value for the first channel of the image gray area is determined; Determining the fourth compensation value of the second channel of the image gray area based on the second ratio of each of the gray points includes: Obtain the sixth compensation value of the second channel of the third original image; The second average value is obtained by averaging the second ratios of all the gray points. Divide the second average value by the sixth compensation value to obtain the fourth ratio; Based on the fourth ratio, the fourth compensation value of the second channel of the image gray area is determined.

6. The method according to claim 2, wherein, The method further includes: Write the third compensation value and the fourth compensation value into the first register and the second register, respectively; The step of compensating at least two frames of the second original image based on the first compensation value for a preset channel to obtain at least two compensated second original images includes: The first channel of each of the second original images is compensated using the first register, and the second channel of each of the second original images is compensated using the second register, to obtain at least two frames of compensated second original images.

7. An image processing apparatus, comprising: The fusion module is used to fuse at least two frames of the first original image to obtain a first fused image; The determining module is used to determine a first compensation value for a preset channel in the first fused image based on image data of the image gray area in the first fused image; wherein, the image gray area is a region in the first fused image whose brightness is greater than a preset threshold; The compensation module is used to perform compensation processing on a preset channel of at least two frames of the second original image based on the first compensation value, so as to obtain at least two frames of compensated second original image; The fusion module is also used to fuse the at least two compensated second original images to obtain a second fused image.

8. The apparatus according to claim 7, wherein, The determining module includes: The first calculation unit is used to calculate the ratio of the data of the first channel and the data of the second channel of each gray point based on the data of the first channel and the data of the third channel of multiple gray points in the gray area of ​​the image, so as to obtain the first ratio of each gray point; The second determining unit is used to determine the third compensation value of the first channel of the image gray area based on the first ratio of each of the gray points. The second calculation unit is used to calculate the ratio of the second channel data to the third channel data of each gray point based on the second channel data and the third channel data of multiple gray points in the gray area of ​​the image, and to obtain the second ratio of each gray point. The third determining unit is used to determine the fourth compensation value of the second channel of the image gray area based on the second ratio of each gray point; The fourth determining unit is used to determine the first compensation value based on the third compensation value and the fourth compensation value.

9. A computer-readable storage medium storing a computer program for performing the image processing method according to any one of claims 1-6.

10. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image processing method according to any one of claims 1-6.

Images