Image processing method and device and storage medium

Abstract

The invention relates to an image processing method and device and a storage medium. The method comprises the steps that M images are obtained, the M images are images collected in different polarization states, the M images comprise a target object and other objects, and M is an integer larger than 2; a first image and a second image are determined according to the M images, the first image is used for indicating intensity information of the target object and other objects, and the second image is used for indicating polarization information of the target object and other objects; the first image and the second image are fused according to a first fusion method, a fused image is determined, and the first fusion method is determined according to the material information of the target object or according to the intensity difference between the target object and other objects and the polarization degree difference between the target object and other objects. According to the embodiment of the invention, the quality of the obtained fused image can be further improved, and the adaptability of the fused image and a traditional task model is improved, so that the fused image is adaptive to multiple task models.

Description

Image processing method, device and storage medium Technical Field

[0001] The present application relates to the field of optical perception technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] Optical perception technology is essential for autonomous and assisted driving systems. Currently, visible light cameras are the primary in-vehicle optical perception devices. However, in automotive scenarios, traditional optical imaging technology is susceptible to factors such as lighting variations, shadows, and the presence of heterogeneous objects. This can lead to decreased image quality, usability, and model prediction accuracy.

[0003] Compared to traditional intensity images, polarization images generated using polarization imaging techniques have the advantages of being less affected by illumination variations and providing clearer contrast between different targets. However, due to the differences in features between polarization images and traditional intensity images, and the fact that current methods typically design and train neural network models based on traditional intensity images, direct application of these models to polarization images results in poor adaptability and fails to fully realize the advantages of polarization imaging. Therefore, a new image processing method is urgently needed to improve the compatibility between polarization images and models.

[0004] Summary of the Invention

[0005] In view of this, an image processing method, device and storage medium are proposed.

[0006] In a first aspect, an embodiment of the present application provides an image processing method. The method includes:

[0007] Acquire M images, where the M images are images acquired in different polarization states, the M images include the target object and other objects, and M is an integer greater than 2;

[0008] determining a first image and a second image based on the M images, where the first image is used to indicate intensity information of the target object and other objects, and the second image is used to indicate polarization information of the target object and other objects;

[0009] The first image and the second image are fused according to a first fusion method to determine a fused image, wherein the first fusion method is determined based on material information of the target object, or based on an intensity difference between the target object and other objects, or a polarization difference between the target object and other objects.

[0010] The M images can be acquired at the same time or at different times. Different polarization states can indicate different polarization angles at the time of acquisition. The fused images can be used to perform the target task.

[0011] According to an embodiment of the present application, by acquiring images captured under different polarization states, a first image and a second image are determined for indicating intensity information and polarization information, respectively. The advantages of polarization images being less affected by lighting changes and having obvious contrast between different objects can be utilized, and the difference between polarization images and traditional intensity images is taken into account to achieve more targeted image processing. By fusing the first image and the second image according to a first fusion method, since the first fusion method can be determined based on the material information of the target object, or based on the intensity difference between the target object and other objects and the polarization difference between the target object and other objects, the quality of the fused image can be further improved, and the decoupling of polarization image processing from subsequent tasks can be achieved. The adaptability of the fused image to the traditional task model can be improved to adapt to a variety of task models.

[0012] According to the first aspect, in a first possible implementation manner of the image processing method, the material information of the target object may include reflectivity, roughness, refractive index, and glossiness of the target object.

[0013] According to an embodiment of the present application, by determining the reflectivity, roughness, refractive index and glossiness of the target object, a fusion method of the first image and the second image can be designed more specifically to obtain an image that is more suitable for subsequent tasks and improve the accuracy of the target task results.

[0014] According to the first aspect or the first possible implementation of the first aspect, in a second possible implementation of the image processing method, the material information of the target object can be obtained based on the intensity difference between the target object and other objects and the polarization difference between the target object and other objects.

[0015] According to the embodiments of the present application, by utilizing the intensity difference between the target object and other objects and the polarization difference between the target object and other objects, the material information of the target object can be obtained more specifically, so that the obtained material information is more consistent with the actual material condition of the target object, and the process of determining the material information is more real-time.

[0016] According to the first aspect or the first or second possible implementation manner of the first aspect, in a third possible implementation manner of the image processing method, the first fusion method may include one of the following:

[0017] Taking the difference between the first image and the second image;

[0018] summing the first image and the second image;

[0019] Multiply the first image by the second image.

[0020] According to the embodiments of the present application, through multiple first fusion methods, different first fusion methods can be flexibly selected to deal with different task scenarios in a more targeted manner, so that the fused image can be flexibly adapted to various task models to obtain more accurate task results.

[0021] According to the first aspect or the first or second or third possible implementation manner of the first aspect, in a fourth possible implementation manner of the image processing method, when the first preset condition and the second preset condition are met, the first fusion method may be to calculate the difference between the first image and the second image;

[0022] Among them, the first preset condition may include that the reflectivity of the target object is greater than the reflectivity of other objects, and the intensity difference between the target object and the other objects is greater than a first preset threshold; the second preset condition may include that the polarization degree of the target object is less than the polarization degree of other objects, and the polarization degree difference between the target object and the other objects is greater than a second preset threshold.

[0023] According to an embodiment of the present application, by taking the difference between the first image and the second image when the reflectivity of the target object is greater than the reflectivity of other objects and the polarization degree of the target object is less than the polarization degree of other objects, it is possible to achieve more targeted image fusion, adapt to subsequent tasks, and enable the fused image to more clearly distinguish the target object, thereby improving the accuracy of the subsequent task execution results.

[0024] According to the first aspect or the first or second or third or fourth possible implementation manner of the first aspect, in a fifth possible implementation manner of the image processing method, when the third preset condition and the fourth preset condition are met, the first fusion method may be summing the first image and the second image;

[0025] Among them, the third preset condition may include that the intensity of the target object and the intensity of other objects are both less than a third preset threshold, and the fourth preset condition may include that the polarization degree of the target object is greater than the polarization degree of other objects, and the difference in polarization degree between the target object and other objects is greater than the fourth preset threshold.

[0026] According to an embodiment of the present application, by summing the first image and the second image when the reflectivity of the target object and the reflectivity of other objects are both smaller and the polarization degree of the target object is greater than the polarization degree of other objects, it is possible to achieve more targeted image fusion, adapt to subsequent tasks, and enable the fused image to more clearly distinguish the target object, thereby improving the accuracy of the subsequent task execution results.

[0027] According to the first aspect or the first or second or third or fourth or fifth possible implementation manner of the first aspect, in a sixth possible implementation manner of the image processing method, when the fifth preset condition and the sixth preset condition are met, the first fusion method may be multiplying the first image and the second image;

[0028] Among them, the fifth preset condition may include that the intensity difference between the target object and other objects is less than the fifth preset threshold, and the sixth preset condition may include that the polarization degree of the target object is greater than the polarization degree of other objects, and the polarization degree difference between the target object and other objects is greater than the sixth preset threshold.

[0029] According to an embodiment of the present application, by multiplying the first image and the second image when the difference between the reflectivity of the target object and the reflectivity of other objects is small and the polarization degree of the target object is greater than the polarization degree of other objects, more targeted image fusion can be achieved to adapt to subsequent tasks, and the fused image can more clearly distinguish the target object, thereby improving the accuracy of the subsequent task execution results.

[0030] According to the first aspect or the first or second or third or fourth or fifth or sixth possible implementation manner of the first aspect, in a seventh possible implementation manner of the image processing method, the first fusion method may include one of the following:

[0031] f=a1*I-b1*P,

[0032] f=a2*I+b2*P,

[0033] f=I*c*P;

[0034] Among them, f can represent the first fusion method, I can represent the first image, P can represent the second image, a1 and a2 can respectively represent the weights corresponding to the first image, and b1, b2 and c can respectively represent the weights corresponding to the second image.

[0035] According to an embodiment of the present application, by setting corresponding weights for the first image and the second image, the preference for intensity information and polarization information can be adjusted more flexibly according to task requirements, so as to respond to different task scenarios more specifically and improve the adaptability of the fused image.

[0036] According to the first aspect or the first or second or third or fourth or fifth or sixth or seventh possible implementation of the first aspect, in an eighth possible implementation of the image processing method, the first image can be determined based on the average value of the intensities of the pixels of the M images, and the second image can be determined by calculating the ratio of the intensities of the pixels of the polarized part and the intensities of the overall pixels in the M images.

[0037] According to the embodiment of the present application, the characteristics of the polarization image can be utilized to integrate the intensity information indicated by the M images to determine the overall intensity information, and to integrate the polarization information indicated by the M images to determine the overall polarization information.

[0038] In a second aspect, an embodiment of the present application provides an image processing device. The device includes:

[0039] an acquisition module, configured to acquire M images, where the M images are images acquired in different polarization states, the M images include a target object and other objects, and M is an integer greater than 2;

[0040] a determination module, configured to determine a first image and a second image based on the M images, wherein the first image is used to indicate intensity information of the target object and other objects, and the second image is used to indicate polarization information of the target object and other objects;

[0041] A fusion module is used to fuse the first image and the second image according to a first fusion method to determine a fused image. The first fusion method is determined based on material information of the target object, or based on the intensity difference between the target object and other objects, and the polarization difference between the target object and other objects.

[0042] The M images can be acquired at the same time or at different times. Different polarization states can indicate different polarization angles at the time of acquisition. The fused images can be used to perform the target task.

[0043] According to the second aspect, in a first possible implementation manner of the image processing device, the material information of the target object may include reflectivity, roughness, refractive index, and glossiness of the target object.

[0044] According to the second aspect or the first possible implementation of the second aspect, in the second possible implementation of the image processing device, the material information of the target object can be obtained based on the intensity difference between the target object and other objects and the polarization difference between the target object and other objects.

[0045] According to the second aspect or the first or second possible implementation manner of the second aspect, in a third possible implementation manner of the image processing device, the first fusion method may include one of the following:

[0046] Taking the difference between the first image and the second image;

[0047] summing the first image and the second image;

[0048] Multiply the first image by the second image.

[0049] According to the second aspect or the first or second or third possible implementation manner of the second aspect, in a fourth possible implementation manner of the image processing device, when the first preset condition and the second preset condition are met, the first fusion method may be to calculate the difference between the first image and the second image;

[0050] Among them, the first preset condition may include that the reflectivity of the target object is greater than the reflectivity of other objects, and the intensity difference between the target object and the other objects is greater than a first preset threshold; the second preset condition may include that the polarization degree of the target object is less than the polarization degree of other objects, and the polarization degree difference between the target object and the other objects is greater than a second preset threshold.

[0051] According to the second aspect or the first or second or third or fourth possible implementation manner of the second aspect, in a fifth possible implementation manner of the image processing device, when the third preset condition and the fourth preset condition are met, the first fusion method may be summing the first image and the second image;

[0052] Among them, the third preset condition may include that the intensity of the target object and the intensity of other objects are both less than a third preset threshold, and the fourth preset condition may include that the polarization degree of the target object is greater than the polarization degree of other objects, and the difference in polarization degree between the target object and other objects is greater than the fourth preset threshold.

[0053] According to the second aspect or the first or second or third or fourth or fifth possible implementation manner of the second aspect, in a sixth possible implementation manner of the image processing device, when the fifth preset condition and the sixth preset condition are met, the first fusion method may be multiplying the first image and the second image;

[0054] Among them, the fifth preset condition may include that the intensity difference between the target object and other objects is less than the fifth preset threshold, and the sixth preset condition may include that the polarization degree of the target object is greater than the polarization degree of other objects, and the polarization degree difference between the target object and other objects is greater than the sixth preset threshold.

[0055] According to the second aspect or the first or second or third or fourth or fifth or sixth possible implementation manner of the second aspect, in a seventh possible implementation manner of the image processing device, the first fusion method may include one of the following:

[0056] f=a1*I-b1*P,

[0057] f=a2*I+b2*P,

[0058] f=I*c*P;

[0059] Among them, f can represent the first fusion method, I can represent the first image, P can represent the second image, a1 and a2 can respectively represent the weights corresponding to the first image, and b1, b2 and c can respectively represent the weights corresponding to the second image.

[0060] According to the second aspect or the first or second or third or fourth or fifth or sixth or seventh possible implementation of the second aspect, in an eighth possible implementation of the image processing device, the first image can be determined based on the average value of the intensities of the pixels of the M images, and the second image can be determined by calculating the ratio of the intensities of the pixels of the polarized part and the intensities of the overall pixels in the M images.

[0061] In a third aspect, an embodiment of the present application provides an image processing device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-mentioned first aspect or one or more of the multiple possible implementation methods of the first aspect when executing the instructions.

[0062] In a fourth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the image processing method of the above-mentioned first aspect or one or more of the multiple possible implementation methods of the first aspect is implemented.

[0063] In a fifth aspect, an embodiment of the present application provides a terminal device that can execute the image processing method of the above-mentioned first aspect or one or more of the multiple possible implementation methods of the first aspect.

[0064] In the sixth aspect, an embodiment of the present application provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in an electronic device, the processor in the electronic device executes the image processing method of the above-mentioned first aspect or one or more of the multiple possible implementations of the first aspect.

[0065] These and other aspects of the present application will become more readily apparent from the following description of the embodiment(s). BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the application and, together with the description, serve to explain the principles of the application.

[0067] FIG1 is a schematic diagram showing an application scenario according to an embodiment of the present application.

[0068] FIG2 shows a flowchart of an image processing method according to an embodiment of the present application.

[0069] FIG3 is a schematic diagram showing images collected under different polarization states according to an embodiment of the present application.

[0070] FIG. 4 shows a schematic diagram of obtaining material information according to an embodiment of the present application.

[0071] FIG5(a), FIG5(b) and FIG5(c) are schematic diagrams showing image fusion according to an embodiment of the present application.

[0072] FIG6(a), FIG6(b) and FIG6(c) are schematic diagrams showing image fusion according to an embodiment of the present application.

[0073] FIG7( a ), FIG7 ( b ) and FIG7 ( c ) are schematic diagrams showing image fusion according to an embodiment of the present application.

[0074] FIG8( a ) and FIG8 ( b ) are schematic diagrams showing the lane detection effect according to an embodiment of the present application.

[0075] FIG9 shows a structural diagram of an image processing apparatus according to an embodiment of the present application.

[0076] FIG10 shows a structural diagram of an electronic device 1000 according to an embodiment of the present application. DETAILED DESCRIPTION

[0077] Various exemplary embodiments, features, and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0078] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0079] In addition, numerous specific details are provided in the detailed description below to better illustrate the present application. Those skilled in the art will appreciate that the present application can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present application.

[0080] Optical perception technology is essential for autonomous driving and assisted driving systems. Currently, the main on-board optical perception device is a visible light camera. However, in on-board scenarios, traditional optical imaging technology is susceptible to factors such as illumination changes, shadow occlusion, and the presence of heterogeneous objects with the same spectrum, which can lead to a decrease in image quality, usability, and the accuracy of model prediction results. Compared with traditional intensity images, polarization images obtained using polarization imaging technology have the advantages of being less affected by illumination changes and having a clear contrast between different targets. Due to the differences in features between polarization images and traditional intensity images, and the fact that current methods are usually based on designing and training neural network models based on traditional intensity images, if the model is directly applied to polarization images, the adaptability is poor and the advantages of polarization imaging cannot be reflected. Therefore, a new image processing method is urgently needed to improve the adaptability between polarization images and models.

[0081] In order to solve the above technical problems, the present application provides an image processing method. The image processing method of an embodiment of the present application determines a first image and a second image for indicating intensity information and polarization information respectively by acquiring images collected under different polarization states. The method can utilize the advantages of polarization images that are less affected by lighting changes and have obvious contrast between different objects, and takes into account the difference between polarization images and traditional intensity images. The first image and the second image are fused according to a first fusion method. Since the first fusion method can be determined based on the material information of the target object, or based on the intensity difference between the target object and other objects and the polarization difference between the target object and other objects, the quality of the fused image can be further improved, and the decoupling of polarization image processing and subsequent tasks can be achieved. The adaptability of the fused image to the traditional task model can be improved to adapt to multiple task models.

[0082] Figure 1 shows a schematic diagram of an application scenario according to an embodiment of the present application. As shown in Figure 1, the image processing system of the embodiment of the present application can be deployed on a server or terminal device for processing images in an in-vehicle scenario. For example, in a scenario of automatic driving or assisted driving, the image processing system of the embodiment of the present application can be applied to obtain images under multiple polarization states collected by an optical sensor (for example, an in-vehicle sensor), and the images are processed and fused to obtain a fused image. The fused image can be used to perform corresponding tasks. For example, the fused image can be input into a corresponding software module (such as a neural network model) to perform subsequent tasks (such as traffic light head detection, lane line detection, etc.) to obtain the target task result. The image processing system can also be connected to hardware modules such as a novel optical imaging sensor and an ISP (image signal processor) to perform subsequent tasks.

[0083] In the above scenario, the optical sensor can be installed on a vehicle (for example, a collection vehicle), which is one or more cameras, such as color, grayscale, infrared, multispectral cameras, etc. Polarization imaging technology can be used through one or more cameras to obtain images in multiple polarization states.

[0084] The server involved in this application can be located in the cloud or locally, and can be a physical device or a virtual device, such as a virtual machine, container, etc., and has a wireless communication function, wherein the wireless communication function can be set in the chip (system) or other parts or components of the server. It can refer to a device with a wireless connection function, and the wireless connection function means that it can be connected to other servers or terminal devices through wireless connection methods such as Wi-Fi and Bluetooth. The server of this application can also have the function of communicating via a wired connection.

[0085] The terminal device involved in this application may refer to a device with a wireless connection function. The wireless connection function means that it can be connected to other terminal devices or servers through wireless connection methods such as Wi-Fi and Bluetooth. The terminal device of this application may also have the function of communicating through a wired connection. The terminal device of this application may be a touch screen, a non-touch screen, or a screenless device. The touch screen device can be controlled by clicking, sliding, etc. on the display screen with a finger or a stylus. The non-touch screen device can be connected to an input device such as a mouse, keyboard, or touch panel to control the terminal device through the input device. For example, a device without a screen can be a Bluetooth speaker without a screen. For example, the terminal device of this application can be a smart phone, a netbook, a tablet computer, a laptop, a wearable electronic device (such as a smart bracelet, a smart watch, etc.), a TV, a virtual reality device, an audio device, an electronic ink, etc. The terminal device of the embodiment of this application may also be a vehicle-mounted terminal device. Among them, the processor can be built into the vehicle computer on the vehicle as an on-board computing unit, so that the image processing process of the embodiment of this application can be realized in real time on the vehicle side.

[0086] The image processing system of the embodiment of the present application can also be applied to other scenarios besides vehicle-mounted scenarios, as long as it involves processing polarization images, and the present application does not impose any restrictions on this.

[0087] The image processing method of the embodiment of the present application is described in detail below with reference to Figures 2 to 8.

[0088] FIG2 is a flowchart of an image processing method according to an embodiment of the present application. The method can be used in the above-mentioned image processing system. As shown in FIG2 , the method includes:

[0089] Step S201, obtaining M images.

[0090] The M images may be images captured in different polarization states, for example, images captured by one or more of the aforementioned optical sensors. For example, in vehicle-mounted scenarios such as autonomous driving or assisted driving, the optical sensor may be installed on the vehicle. The M images may include the target object and other objects, where M is an integer greater than 2.

[0091] Images in different polarization states can be obtained through a variety of polarization imaging methods. The M images can be collected at the same time or at different times; they can be collected by a single optical sensor or by multiple optical sensors set at different positions. For example, images can be collected through a multi-camera synchronous imaging method (i.e., polarizers with different directions are set on M cameras, and the M cameras are set at different positions to collect images simultaneously), a single-camera imaging method (i.e., a rotatable polarizer is set on one camera, and the orientation of the polarizer is adjusted by rotation to collect images in different polarization states), a pixel-level polarization coating camera imaging method (i.e., different polarization states are set to collect images by using the polarization coating on the camera itself).

[0092] Different polarization states may refer to different polarization angles (i.e., polarization directions). Referring to FIG3 , a schematic diagram of images captured under different polarization states according to an embodiment of the present application is shown. For example, taking images captured under four polarization states as an example (i.e., M is 4), these four images may correspond to four different polarization angles of 0°, 45°, 90°, and 135°, respectively, and may correspond to the four images I0 (upper left portion), I45 (upper right portion), I90 (lower left portion), and I135 (lower right portion) in FIG3 .

[0093] Among them, the target object can be determined based on the task to be performed subsequently, and can include one object element or multiple object elements in the image, and other objects can include all or part of the object elements in the image other than the target object. For example, in the lane line detection task, the target object can include the lane line (such as the road center line, lane dividing line, lane edge line, etc. on the lane), and correspondingly, other objects can include objects other than the target object, such as the road environment background, etc. For another example, in the traffic light head detection task in the road scene, the target object can include the lamp head edge line, and other objects can include objects other than the lamp head, such as the environment background, etc. For another example, in the road manhole cover detection task, the target object can include the manhole cover, and other objects can include other objects on the road except the manhole cover.

[0094] Next, the characteristics of the polarization image can be utilized to determine the intensity and polarization information indicated in the M images respectively.

[0095] Step S202: Determine a first image and a second image according to the M images.

[0096] A first image and a second image can be determined according to the M images respectively.

[0097] The first image can be used to indicate intensity information of the target object and other objects. The intensity information can be light intensity, for example, the grayscale value of an image pixel in a grayscale image, or the intensity of an image pixel in a color image (which can be reflected as the pixel value of each pixel in the first image).

[0098] Alternatively, the first image may be determined based on an average value of the intensities of the pixels of the M images. Thus, the intensity information indicated by the M images can be integrated to determine the overall intensity information.

[0099] Taking M as 4 as an example, a calculation method for determining the first image can be referred to formula (1):

[0100]

[0101] Where I corresponds to the first image and represents the pixel intensity of the first image. I0, I45, I90, and I135 can represent the pixel intensities of the images collected under different polarization states, respectively. Since light attenuates when passing through a polarizer, it is attenuated by half on average. Therefore, the average value can be divided by 2 (for example, the denominator in formula (1) is originally 4, and divided by 2 to get 2) to obtain the pixel value of the first image.

[0102] The second image may be used to indicate polarization information of the target object and other objects. The polarization information may be, for example, a degree of polarization (which may be reflected by a pixel value of each pixel of the second image).

[0103] Alternatively, the second image may be determined by calculating the ratio of the intensity of the pixels of the polarized portion to the intensity of the overall pixels in the M images. Thus, the overall polarization information can be determined by integrating the polarization information indicated by the M images.

[0104] Taking M as 4 as an example, a calculation method for determining the second image can be referred to formula (2):

[0105]

[0106] Wherein, P may correspond to the second image and represent the polarization degree of the second image. I0, I45, I90, and I135 may represent the pixel intensities of the images acquired under different polarization states, respectively. I may represent the pixel intensity of the first image and may be obtained by formula (1). The pixel value of each pixel point in the second image can be determined by mapping the ratio corresponding to P (i.e., the polarization degree) to a range consistent with the pixel intensity indicated by I (e.g., 0-255).

[0107] After obtaining the first image and the second image, the first image and the second image can be fused in combination with the material characteristics of the target object in the scene, or the difference information between the target object and other objects to determine the fused image and realize the processing of the polarization image, as described below.

[0108] Step S203: Fusing the first image and the second image according to the first fusion method to determine a fused image.

[0109] The first fusion method may be determined based on material information of the target object, or based on an intensity difference between the target object and other objects, or a polarization difference between the target object and other objects.

[0110] According to an embodiment of the present application, by acquiring images captured under different polarization states, a first image and a second image are determined for indicating intensity information and polarization information, respectively. The advantages of polarization images being less affected by lighting changes and having obvious contrast between different objects can be utilized, and the difference between polarization images and traditional intensity images is taken into account to achieve more targeted image processing. By fusing the first image and the second image according to a first fusion method, since the first fusion method can be determined based on the material information of the target object, or based on the intensity difference between the target object and other objects and the polarization difference between the target object and other objects, the quality of the fused image can be further improved, and the decoupling of polarization image processing from subsequent tasks can be achieved. The adaptability of the fused image to the traditional task model can be improved to adapt to a variety of task models.

[0111] Optionally, the material information of the target object may include reflectivity, roughness, refractive index and glossiness of the target object, or may include one or more of them.

[0112] Reflectivity can be the ratio of the intensity of reflected light to the intensity of incident light. Surfaces of different materials can have different reflectivities. Roughness can be the roughness of an object's surface. The smaller the surface roughness, the smoother the surface of the corresponding material. The refractive index can be expressed as the ratio of the speed of light in a vacuum to its phase velocity after entering the medium corresponding to the object. The refractive index of the target object is related to the degree of polarization. Glossiness can indicate the mirror reflection ability of the object's surface to light. Mirror reflection can indicate a reflection characteristic with direction selection. The lower the mirror reflectivity of the material surface, the lower the glossiness of the corresponding material surface.

[0113] According to an embodiment of the present application, by determining the reflectivity, roughness, refractive index and glossiness of the target object, a fusion method of the first image and the second image can be designed more specifically to obtain an image that is more suitable for subsequent tasks and improve the accuracy of the target task results.

[0114] When the first fusion method is based on the material information of the target object, to improve the efficiency of obtaining material information, the target object's material information can be obtained a priori. For example, a sensor such as a gloss meter can be used to pre-collect information such as the reflectivity, roughness, refractive index, and glossiness of the target object's surface material as a priori input for the fusion process. One or more of reflectivity, roughness, refractive index, and glossiness can be collected based on task requirements. The a priori input can be input by the user or collected and input by an on-board sensor (such as a gloss meter).

[0115] In order to improve the real-time and pertinence of material information determination, optionally, the material information of the target object can also be obtained based on the intensity difference between the target object and other objects and the polarization degree difference between the target object and other objects.

[0116] The intensity of the target object can be determined based on the pixel values ​​corresponding to the target object region in the first image (e.g., the average value of the pixel values ​​within the region); the intensity of other objects can be determined based on the pixel values ​​corresponding to the regions of other objects in the first image other than the target object (e.g., the average value of the pixel values ​​within the region). The polarization degree of the target object can be determined based on the pixel values ​​corresponding to the target object region in the second image (e.g., the average value of the pixel values ​​within the region); the polarization degree of other objects can be determined based on the pixel values ​​corresponding to the regions of other objects in the second image other than the target object (e.g., the average value of the pixel values ​​within the region).

[0117] For example, the intensity difference between the target object and other objects (e.g., the difference between the two) can be determined based on the intensity information indicated by the first image; the polarization difference between the target object and other objects (e.g., the difference between the two) can also be determined based on the polarization information indicated by the second image. In one determination method, if the environment of the other object is specific (e.g., the other object is an asphalt road surface), the reflectivity and roughness of the target object can be relatively determined based on the intensity difference; and the refractive index and glossiness of the target object can be relatively determined based on the polarization difference.

[0118] According to the embodiments of the present application, by utilizing the intensity difference between the target object and other objects and the polarization difference between the target object and other objects, the material information of the target object can be obtained more specifically, so that the obtained material information is more consistent with the actual material condition of the target object, and the process of determining the material information is more real-time.

[0119] Referring to Figure 4 , a schematic diagram illustrating obtaining material information according to an embodiment of the present application is shown. As shown in Figure 4 , the target object may be the object indicated in the image, and the road surface background in the image may represent other objects corresponding to the target object. In scenarios where the material information of the target object is obtained a priori, the user may input the material information of the target object, or the vehicle-mounted sensor may perform a priori measurement of the object indicated in the image to acquire the material information.

[0120] When determining the material information of the target object by the intensity difference between the target object and other objects and the polarization difference between the target object and other objects, the intensity difference and polarization difference between the object indicated in the figure and the road background can be determined in the above manner to determine the material information of the target object.

[0121] Since different task scenarios involve different target objects, and the material properties of these target objects are not exactly the same, different first fusion methods can be designed to specifically address different task scenarios. The following describes in detail how to determine the first fusion method.

[0122] Optionally, the first fusion method may include one of the following:

[0123] Taking the difference between the first image and the second image;

[0124] summing the first image and the second image;

[0125] multiplying the first image by the second image;

[0126] The above may be to calculate the difference, sum or product of the pixel values ​​of the first image and the pixel values ​​of the second image.

[0127] According to the embodiments of the present application, through multiple first fusion methods, different first fusion methods can be flexibly selected to deal with different task scenarios in a more targeted manner, so that the fused image can be flexibly adapted to various task models to obtain more accurate task results.

[0128] In particular, different weights may be set for the first image and the second image during the process of finding the difference, sum, and product, so as to flexibly adjust the preference for intensity information and polarization information according to task requirements.

[0129] For example, a calculation method for calculating the difference between the first image and the second image can be referred to formula (3):

[0130] f=a1*I-b1*P Formula (3)

[0131] Wherein, f may represent the first fusion method. I may represent the first image (e.g., the pixel value of each pixel in the first image), and P may represent the second image (e.g., the pixel value of each pixel in the second image). I and P can be obtained according to the above process. a1 may represent the weight corresponding to the first image, and b1 may represent the weight corresponding to the second image. The values ​​of a1 and b1 can be pre-set as needed.

[0132] A calculation method for summing the first image and the second image can be found in formula (4):

[0133] f=a²*I+b²*P Formula (4)

[0134] Wherein, f may represent the first fusion method. I may represent the first image (e.g., the pixel value of each pixel in the first image), and P may represent the second image (e.g., the pixel value of each pixel in the second image). I and P can be obtained according to the above process. a2 may represent the weight corresponding to the first image, and b2 may represent the weight corresponding to the second image. The values ​​of a2 and b2 can be pre-set as needed.

[0135] One way to calculate the product of the first image and the second image can be seen in formula (5):

[0136] f=I*c*P Formula (5)

[0137] Wherein, f may represent the first fusion method. I may represent the first image (e.g., the pixel value of each pixel point in the first image), and P may represent the second image (e.g., the pixel value of each pixel point in the second image). I and P may be obtained according to the above process. c may represent the weight corresponding to the second image (or the weight corresponding to the first image, or the product of the weight corresponding to the first image and the weight corresponding to the second image). The value of c may be pre-set as needed.

[0138] According to an embodiment of the present application, by setting corresponding weights for the first image and the second image, the preference for intensity information and polarization information can be adjusted more flexibly according to task requirements, so as to respond to different task scenarios more specifically and improve the adaptability of the fused image.

[0139] The following is an exemplary introduction to the three cases of determining the first fusion method:

[0140] 5(a), 5(b), and 5(c) illustrate a schematic diagram of image fusion according to an embodiment of the present application. The diagram illustrates how a first fusion method is determined in a lane detection scenario, where FIG5(a) represents the first image, FIG5(b) represents the second image, and FIG5(c) represents the fused image.

[0141] In the lane detection task scenario, the target object can be the lane line, and the other object can be the road area outside the lane line. As can be seen from Figure 5(a), in this scenario, the reflectivity corresponding to the lane line is greater than the reflectivity corresponding to other road areas (as reflected by the pixel values ​​of the lane line area in the first image being greater than the pixel values ​​of other road areas); as can be seen from Figure 5(b), in this scenario, the polarization degree corresponding to the lane line is less than the polarization degree corresponding to other road areas (as reflected by the pixel values ​​of the lane line area in the second image being less than the pixel values ​​of other road areas).

[0142] Therefore, based on the difference characteristics between the target object and other objects in the above-mentioned intensity and polarization dimensions, optionally, when the first preset condition and the second preset condition are met, the first fusion method can be to calculate the difference between the first image and the second image. One method of calculating the difference can also be referred to the above-mentioned formula (3).

[0143] The first preset condition may include that the reflectivity of the target object is greater than the reflectivity of other objects, and the intensity difference between the target object and other objects is greater than a first preset threshold (corresponding to the feature in FIG5(a) that the reflectivity corresponding to the lane marking is greater than the reflectivity corresponding to other road areas). The second preset condition may include that the polarization degree of the target object is less than the polarization degree of other objects, and the polarization degree difference between the target object and other objects is greater than a second preset threshold (corresponding to the feature in FIG5(b) that the polarization degree corresponding to the lane marking is less than the polarization degree corresponding to other road areas).

[0144] For example, since intensity information can reflect reflectivity (a larger intensity value indicates a greater reflectivity), whether the first preset condition is satisfied can be determined based on the intensity difference between the target object and other objects determined above. Whether the second preset condition is satisfied can also be determined based on the polarization difference between the target object and other objects determined above.

[0145] Optionally, when the material information of the target object is determined, the first preset condition may also be that the reflectivity of the target object is greater than a predetermined threshold and / or the roughness of the target object is less than a predetermined threshold; the second preset condition may also be that the glossiness of the target object is less than a predetermined threshold. This application does not impose any restrictions on the first and second preset conditions, as long as the target object can be more clearly distinguished in the difference image.

[0146] As shown in FIG5(c), the fused image determined by the above fusion method can more clearly distinguish object elements such as lane lines in the image.

[0147] According to an embodiment of the present application, by taking the difference between the first image and the second image when the reflectivity of the target object is greater than the reflectivity of other objects and the polarization degree of the target object is less than the polarization degree of other objects, it is possible to achieve more targeted image fusion, adapt to subsequent tasks, and enable the fused image to more clearly distinguish the target object, thereby improving the accuracy of the subsequent task execution results.

[0148] 6(a), 6(b), and 6(c) illustrate schematic diagrams of image fusion according to an embodiment of the present application. These diagrams illustrate how a first fusion method is determined in a traffic light head detection scenario, where FIG6(a) represents the first image, FIG6(b) represents the second image, and FIG6(c) represents the fused image.

[0149] In the scenario of a traffic light head detection task, the target object can be the edge of the traffic light head, and the other object can be the background area outside the traffic light head. As can be seen from Figure 6(a), in this scenario, the reflectivity corresponding to the light head edge is lower than the reflectivity corresponding to the background area (reflected by the smaller pixel values ​​of the light head edge and the background area in the first image). As can be seen from Figure 6(b), in this scenario, the polarization degree corresponding to the light head edge is greater than the polarization degree corresponding to the background area (reflected by the larger pixel values ​​of the light head edge and the background area in the second image).

[0150] Therefore, based on the difference characteristics between the target object and other objects in the above-mentioned intensity and polarization dimensions, optionally, when the third and fourth preset conditions are met, the first fusion method can be to sum the first image and the second image. One method of summing can also be referred to the above-mentioned formula (4).

[0151] The third preset condition may include that the intensity of the target object and the intensity of other objects are both less than a third preset threshold (corresponding to the feature in FIG6(a) that the reflectivity corresponding to the edge of the lamp head and the reflectivity corresponding to the background area are both lower). The fourth preset condition may include that the polarization degree of the target object is greater than the polarization degrees of other objects, and the difference in polarization degrees between the target object and the other objects is greater than a fourth preset threshold (corresponding to the feature in FIG6(b) that the polarization degree corresponding to the edge of the lamp head is greater than the polarization degree corresponding to the background area).

[0152] For example, since intensity information can reflect reflectivity (a larger intensity value can indicate a greater reflectivity), one approach can determine whether a third preset condition is satisfied based on the intensity difference between the target object and the other objects determined above, and the average pixel value of the first image (for example, the third preset condition is considered to be satisfied when the average pixel value of the first image is less than a preset threshold, and the intensity difference between the target object and the other objects determined above is less than another preset threshold (these two preset thresholds can be determined based on the third preset threshold, respectively)). Whether a fourth preset condition is satisfied can also be determined based on the polarization difference between the target object and the other objects determined above.

[0153] Optionally, when the material information of the target object is determined, the third preset condition may also be that the reflectivity of the target object is less than a predetermined threshold and / or the roughness of the target object is greater than a predetermined threshold; the fourth preset condition may also be that the glossiness of the target object is greater than a predetermined threshold. This application does not impose any restrictions on the third and fourth preset conditions, as long as the summed image can more clearly distinguish the target object.

[0154] As shown in FIG6( c ), the fused image determined by the above fusion method can more clearly distinguish object elements such as the traffic light head in the image.

[0155] According to an embodiment of the present application, by summing the first image and the second image when the reflectivity of the target object and the reflectivity of other objects are both smaller and the polarization degree of the target object is greater than the polarization degree of other objects, it is possible to achieve more targeted image fusion, adapt to subsequent tasks, and enable the fused image to more clearly distinguish the target object, thereby improving the accuracy of the subsequent task execution results.

[0156] 7(a), 7(b), and 7(c) illustrate a schematic diagram of image fusion according to an embodiment of the present application. The diagram illustrates how a first fusion method is determined in a road manhole cover detection scenario, where FIG7(a) represents the first image, FIG7(b) represents the second image, and FIG7(c) represents the fused image.

[0157] In the manhole cover detection scenario, the target object can be a manhole cover, and the other objects can be other road areas. As shown in Figure 7(a), in this scenario, the reflectivity of the manhole cover is similar to that of other road areas (as evidenced by the similar pixel values ​​of the manhole cover and other road areas in the first image). Figure 7(b) also shows that the polarization degree of the manhole cover is greater than that of other road areas (as evidenced by the larger pixel values ​​of the manhole cover and other road areas in the second image).

[0158] Therefore, based on the difference characteristics between the target object and other objects in the above-mentioned intensity and polarization dimensions, optionally, when the fifth and sixth preset conditions are met, the first fusion method is to multiply the first image and the second image. One method for calculating the product can also be seen in the above-mentioned formula (5).

[0159] The fifth preset condition may include that the intensity difference between the target object and other objects is less than a fifth preset threshold (corresponding to the characteristic that the reflectivity corresponding to the manhole cover is similar to the reflectivity corresponding to other road surface areas in FIG7(a)). The sixth preset condition may include that the degree of polarization of the target object is greater than the degrees of polarization of other objects, and the difference in the degree of polarization between the target object and other objects is greater than a sixth preset threshold (corresponding to the characteristic that the degree of polarization corresponding to the manhole cover is greater than the degrees of polarization corresponding to other road surface areas in FIG7(b)).

[0160] For example, since intensity information can reflect reflectivity (a larger intensity value indicates a greater reflectivity), one approach is to determine whether the fifth preset condition is satisfied based on the intensity difference between the target object and the other objects. Alternatively, the sixth preset condition may be determined based on the polarization difference between the target object and the other objects.

[0161] Optionally, when the material information of the target object is determined, the fifth preset condition may also be that the reflectivity of the target object is less than a predetermined threshold and / or the roughness of the target object is greater than a predetermined threshold; the sixth preset condition may also be that the glossiness of the target object is greater than a predetermined threshold. This application does not impose any restrictions on the fifth and sixth preset conditions, as long as the target object can be more clearly distinguished in the image after the product is calculated.

[0162] Referring to FIG. 7( c ), the fused image determined by the above fusion method can more clearly distinguish object elements such as manhole covers in the image.

[0163] The specific values ​​of the above thresholds can be set according to actual needs.

[0164] According to an embodiment of the present application, by multiplying the first image and the second image when the difference between the reflectivity of the target object and the reflectivity of other objects is small and the polarization degree of the target object is greater than the polarization degree of other objects, more targeted image fusion can be achieved to adapt to subsequent tasks, and the fused image can more clearly distinguish the target object, thereby improving the accuracy of the subsequent task execution results.

[0165] The three scenarios described above are merely exemplary, and more scenarios are possible. Furthermore, the above only illustrates some exemplary methods for determining the first fusion method, and more methods are possible. The first fusion method can be flexibly determined based on different task scenarios, and this application does not impose any restrictions on this.

[0166] The fused image can be used to perform corresponding image tasks, which can be object detection tasks, semantic segmentation tasks, classification tasks, etc. After the fused image is determined through the embodiments of the present application, it can be input into the corresponding task model to obtain more accurate task execution results.

[0167] Figures 8(a) and 8(b) are schematic diagrams showing the lane line detection effect according to an embodiment of the present application. Taking the lane line detection task as an example, Figure 8(a) may correspond to the visual detection result obtained by inputting the lane line detection model when the image is not processed using the method of the embodiment of the present application; Figure 8(b) may correspond to the visual result obtained by inputting the lane line detection model when the image is processed using the method of the embodiment of the present application. The straight lines shown in the figure may represent the detected lane lines. It can be seen that due to interference such as glare and occlusion, the image is processed using the method of the embodiment of the present application and then input into the lane line detection model for detection, which can effectively prevent missed detection and false detection of lane lines, making the lane line detection results more accurate.

[0168] FIG9 shows a structural diagram of an image processing device according to an embodiment of the present application. The device can be used in the above-mentioned image processing system. As shown in FIG9 , the device includes:

[0169] An acquisition module 901 is configured to acquire M images, where the M images are images acquired in different polarization states, the M images include a target object and other objects, and M is an integer greater than 2;

[0170] A determination module 902 is configured to determine a first image and a second image based on the M images, wherein the first image is used to indicate intensity information of the target object and other objects, and the second image is used to indicate polarization information of the target object and other objects;

[0171] The fusion module 903 is used to fuse the first image and the second image according to a first fusion method to determine a fused image. The first fusion method is determined based on material information of the target object, or based on the intensity difference between the target object and other objects and the polarization difference between the target object and other objects.

[0172] The M images can be acquired at the same time or at different times. Different polarization states can indicate different polarization angles at the time of acquisition. The fused images can be used to perform the target task.

[0173] According to an embodiment of the present application, by acquiring images captured under different polarization states, a first image and a second image are determined for indicating intensity information and polarization information, respectively. The advantages of polarization images being less affected by lighting changes and having obvious contrast between different objects can be utilized, and the difference between polarization images and traditional intensity images is taken into account to achieve more targeted image processing. By fusing the first image and the second image according to a first fusion method, since the first fusion method can be determined based on the material information of the target object, or based on the intensity difference between the target object and other objects and the polarization difference between the target object and other objects, the quality of the fused image can be further improved, and the decoupling of polarization image processing from subsequent tasks can be achieved. The adaptability of the fused image to the traditional task model can be improved to adapt to a variety of task models.

[0174] Optionally, the material information of the target object may include reflectivity, roughness, refractive index, and glossiness of the target object.

[0175] According to an embodiment of the present application, by determining the reflectivity, roughness, refractive index and glossiness of the target object, a fusion method of the first image and the second image can be designed more specifically to obtain an image that is more suitable for subsequent tasks and improve the accuracy of the target task results.

[0176] Optionally, the material information of the target object may be obtained according to an intensity difference between the target object and other objects and a polarization degree difference between the target object and other objects.

[0177] According to the embodiments of the present application, by utilizing the intensity difference between the target object and other objects and the polarization difference between the target object and other objects, the material information of the target object can be obtained more specifically, so that the obtained material information is more consistent with the actual material condition of the target object, and the process of determining the material information is more real-time.

[0178] Optionally, the first image may be determined based on an average value of pixel intensities of the M images, and the second image may be determined by calculating a ratio of pixel intensities of the polarized portion to overall pixel intensities in the M images.

[0179] According to the embodiment of the present application, the characteristics of the polarization image can be utilized to integrate the intensity information indicated by the M images to determine the overall intensity information, and to integrate the polarization information indicated by the M images to determine the overall polarization information.

[0180] Optionally, the first fusion method may include one of the following:

[0181] Taking the difference between the first image and the second image;

[0182] summing the first image and the second image;

[0183] Multiply the first image by the second image.

[0184] According to the embodiments of the present application, through multiple first fusion methods, different first fusion methods can be flexibly selected to deal with different task scenarios in a more targeted manner, so that the fused image can be flexibly adapted to various task models to obtain more accurate task results.

[0185] Optionally, the first fusion method may include one of the following:

[0186] f=a1*I-b1*P,

[0187] f=a2*I+b2*P,

[0188] f=I*c*P;

[0189] Among them, f can represent the first fusion method, I can represent the first image, P can represent the second image, a1 and a2 can respectively represent the weights corresponding to the first image, and b1, b2 and c can respectively represent the weights corresponding to the second image.

[0190] According to an embodiment of the present application, by setting corresponding weights for the first image and the second image, the preference for intensity information and polarization information can be adjusted more flexibly according to task requirements, so as to respond to different task scenarios more specifically and improve the adaptability of the fused image.

[0191] Optionally, when the first preset condition and the second preset condition are met, the first fusion method may be to calculate the difference between the first image and the second image;

[0192] Among them, the first preset condition may include that the reflectivity of the target object is greater than the reflectivity of other objects, and the intensity difference between the target object and the other objects is greater than a first preset threshold; the second preset condition may include that the polarization degree of the target object is less than the polarization degree of other objects, and the polarization degree difference between the target object and the other objects is greater than a second preset threshold.

[0193] According to an embodiment of the present application, by taking the difference between the first image and the second image when the reflectivity of the target object is greater than the reflectivity of other objects and the polarization degree of the target object is less than the polarization degree of other objects, it is possible to achieve more targeted image fusion, adapt to subsequent tasks, and enable the fused image to more clearly distinguish the target object, thereby improving the accuracy of the subsequent task execution results.

[0194] Optionally, when the third preset condition and the fourth preset condition are met, the first fusion method may be to sum the first image and the second image;

[0195] Among them, the third preset condition may include that the intensity of the target object and the intensity of other objects are both less than a third preset threshold, and the fourth preset condition may include that the polarization degree of the target object is greater than the polarization degree of other objects, and the difference in polarization degree between the target object and other objects is greater than the fourth preset threshold.

[0196] According to an embodiment of the present application, by summing the first image and the second image when the reflectivity of the target object and the reflectivity of other objects are both smaller and the polarization degree of the target object is greater than the polarization degree of other objects, it is possible to achieve more targeted image fusion, adapt to subsequent tasks, and enable the fused image to more clearly distinguish the target object, thereby improving the accuracy of the subsequent task execution results.

[0197] Optionally, when the fifth preset condition and the sixth preset condition are met, the first fusion method may be to multiply the first image and the second image;

[0198] Among them, the fifth preset condition may include that the intensity difference between the target object and other objects is less than the fifth preset threshold, and the sixth preset condition may include that the polarization degree of the target object is greater than the polarization degree of other objects, and the polarization degree difference between the target object and other objects is greater than the sixth preset threshold.

[0199] According to an embodiment of the present application, by multiplying the first image and the second image when the difference between the reflectivity of the target object and the reflectivity of other objects is small and the polarization degree of the target object is greater than the polarization degree of other objects, more targeted image fusion can be achieved to adapt to subsequent tasks, and the fused image can more clearly distinguish the target object, thereby improving the accuracy of the subsequent task execution results.

[0200] An embodiment of the present application provides an image processing device, comprising: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-mentioned image processing method when executing the instructions.

[0201] An embodiment of the present application provides a terminal device that can execute the above-mentioned image processing method.

[0202] An embodiment of the present application provides a non-volatile computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the above-mentioned image processing method is implemented.

[0203] An embodiment of the present application provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned image processing method.

[0204] Figure 10 shows a block diagram of an electronic device 1000 according to an embodiment of the present application. As shown in Figure 10 , the electronic device 1000 may be the aforementioned image processing system. The electronic device 1000 includes at least one processor 1801, at least one memory 1802, and at least one communication interface 1803. Furthermore, the electronic device may also include common components such as an antenna, which will not be described in detail here.

[0205] The following is a detailed introduction to the various components of the electronic device 1000 with reference to FIG10 .

[0206] Processor 1801 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the above program. Processor 1801 may include one or more processing units. For example, processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0207] The communication interface 1803 is used to communicate with other electronic devices or communication networks, such as Ethernet, Radio Access Network (RAN), core network, Wireless Local Area Networks (WLAN), etc.

[0208] The memory 1802 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0209] The memory 1802 is used to store application code for executing the above solution, and the execution is controlled by the processor 1801. The processor 1801 is used to execute the application code stored in the memory 1802.

[0210] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0211] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof.

[0212] The computer-readable program instructions or codes described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0213] The computer program instructions for performing the operations of the present application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, by utilizing the state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions, thereby implementing various aspects of the present application.

[0214] Various aspects of the present application are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0215] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that, when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowcharts and / or block diagrams. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowcharts and / or block diagrams.

[0216] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0217] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, systems, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and the part for the module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be performed substantially in parallel, and they can sometimes also be performed in the opposite order, depending on the function involved.

[0218] It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by hardware (such as a circuit or ASIC (Application Specific Integrated Circuit)) that performs the corresponding function or action, or can be implemented by a combination of hardware and software, such as firmware.

[0219] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0220] The embodiments of the present application have been described above. The above description is illustrative and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire M images, where the M images are images collected in different polarization states, the M images include a target object and other objects, and M is an integer greater than 2; determining a first image and a second image according to the M images, wherein the first image is used to indicate intensity information of the target object and the other objects, and the second image is used to indicate polarization information of the target object and the other objects; The first image and the second image are fused according to a first fusion method to determine a fused image, wherein the first fusion method is determined based on material information of the target object, or based on an intensity difference between the target object and the other objects and a polarization difference between the target object and the other objects.

2. The method according to claim 1, characterized in that The material information of the target object includes reflectivity, roughness, refractive index and glossiness of the target object.

3. The method according to claim 1 or 2, characterized in that The material information of the target object is obtained according to the intensity difference between the target object and the other objects and the polarization degree difference between the target object and the other objects.

4. The method according to any one of claims 1 to 3, characterized in that The first fusion method includes one of the following: Taking a difference between the first image and the second image; summing the first image and the second image; Multiply the first image by the second image.

5. The method according to any one of claims 1 to 4, characterized in that When a first preset condition and a second preset condition are met, the first fusion method is to calculate the difference between the first image and the second image; Among them, the first preset condition includes that the reflectivity of the target object is greater than the reflectivity of the other objects, and the intensity difference between the target object and the other objects is greater than a first preset threshold; the second preset condition includes that the polarization degree of the target object is less than the polarization degree of the other objects, and the polarization degree difference between the target object and the other objects is greater than a second preset threshold.

6. The method according to any one of claims 1 to 5, characterized in that When the third preset condition and the fourth preset condition are met, the first fusion method is to sum the first image and the second image; Among them, the third preset condition includes that the intensity of the target object and the intensity of the other objects are both less than a third preset threshold, and the fourth preset condition includes that the polarization degree of the target object is greater than the polarization degree of the other objects, and the difference in polarization degree between the target object and the other objects is greater than the fourth preset threshold.

7. The method according to any one of claims 1 to 6, characterized in that When the fifth preset condition and the sixth preset condition are met, the first fusion method is to multiply the first image and the second image; Among them, the fifth preset condition includes that the intensity difference between the target object and the other objects is less than the fifth preset threshold, and the sixth preset condition includes that the polarization degree of the target object is greater than the polarization degree of the other objects, and the polarization degree difference between the target object and the other objects is greater than the sixth preset threshold.

8. The method according to any one of claims 1 to 7, characterized in that The first fusion method includes one of the following: f=a1*I-b1*P, f=a2*I+b2*P, f=I*c*P; Among them, f represents the first fusion method, I represents the first image, P represents the second image, a1 and a2 respectively represent the weights corresponding to the first image, and b1, b2 and c respectively represent the weights corresponding to the second image.

9. The method according to any one of claims 1 to 8, characterized in that The first image is determined according to an average value of pixel intensities of the M images, and the second image is determined by calculating a ratio of pixel intensities of polarized parts to overall pixel intensities in the M images.

10. An image processing device, characterized in that: The device comprises: an acquisition module, configured to acquire M images, wherein the M images are images acquired in different polarization states, the M images include a target object and other objects, and M is an integer greater than 2; a determining module, configured to determine a first image and a second image based on the M images, wherein the first image is used to indicate intensity information of the target object and the other objects, and the second image is used to indicate polarization information of the target object and the other objects; A fusion module is used to fuse the first image and the second image according to a first fusion method to determine a fused image, wherein the first fusion method is determined based on material information of the target object, or based on an intensity difference between the target object and the other objects and a polarization difference between the target object and the other objects.

11. The device according to claim 10, characterized in that The material information of the target object includes reflectivity, roughness, refractive index and glossiness of the target object.

12. The device according to claim 10 or 11, characterized in that The material information of the target object is obtained according to the intensity difference between the target object and the other objects and the polarization degree difference between the target object and the other objects.

13. The device according to any one of claims 10 to 12, characterized in that: The first fusion method includes one of the following: Taking a difference between the first image and the second image; summing the first image and the second image; Multiply the first image by the second image.

14. The device according to any one of claims 10 to 13, characterized in that When a first preset condition and a second preset condition are met, the first fusion method is to calculate the difference between the first image and the second image; Among them, the first preset condition includes that the reflectivity of the target object is greater than the reflectivity of the other objects, and the intensity difference between the target object and the other objects is greater than a first preset threshold; the second preset condition includes that the polarization degree of the target object is less than the polarization degree of the other objects, and the polarization degree difference between the target object and the other objects is greater than a second preset threshold.

15. The device according to any one of claims 10 to 14, characterized in that When the third preset condition and the fourth preset condition are met, the first fusion method is to sum the first image and the second image; Among them, the third preset condition includes that the intensity of the target object and the intensity of the other objects are both less than a third preset threshold, and the fourth preset condition includes that the polarization degree of the target object is greater than the polarization degree of the other objects, and the difference in polarization degree between the target object and the other objects is greater than the fourth preset threshold.

16. The device according to any one of claims 10 to 15, characterized in that When the fifth preset condition and the sixth preset condition are met, the first fusion method is to multiply the first image and the second image; Among them, the fifth preset condition includes that the intensity difference between the target object and the other objects is less than the fifth preset threshold, and the sixth preset condition includes that the polarization degree of the target object is greater than the polarization degree of the other objects, and the polarization degree difference between the target object and the other objects is greater than the sixth preset threshold.

17. The device according to any one of claims 10 to 16, characterized in that The first fusion method includes one of the following: f=a1*I-b1*P, f=a2*I+b2*P, f=I*c*P; Among them, f represents the first fusion method, I represents the first image, P represents the second image, a1 and a2 respectively represent the weights corresponding to the first image, and b1, b2 and c respectively represent the weights corresponding to the second image.

18. The device according to any one of claims 10 to 17, characterized in that The first image is determined according to an average value of pixel intensities of the M images, and the second image is determined by calculating a ratio of pixel intensities of polarized parts to overall pixel intensities in the M images.

19. An image processing device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 9 when executing the instructions.

20. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

21. A computer program product comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, wherein when the computer-readable code is executed in an electronic device, a processor in the electronic device executes the method according to any one of claims 1 to 9.

Images