Image processing method and device and vehicle

Abstract

The invention provides an image processing method and device and a vehicle, which can be applied to the fields of automatic driving, auxiliary driving or intelligent driving and the like and are used for assisting an image acquisition system to output an image with relatively high image quality. The image processing method comprises the steps that after a first image is obtained, the image quality of an area where at least one physical object contained in the first image is located is determined, and when the image quality of the area where the first physical object is located in the first image belongs to a set image quality range, the gain value of the area where the first physical object is located is determined, according to the gain value of the area where the first physical object is located, a target gain value corresponding to the first image is determined, and the target gain value is used for determining the exposure compensation amount needed by the second image. According to the method, after the image area needing image quality optimization in the first image is selected, the exposure compensation amount needed by the image acquisition system for acquiring the second image is calculated based on the image quality of the image area, so that the image acquisition system can output the image with better quality.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to an image processing method, device and vehicle. Background Art

[0002] With the development of autonomous driving technology, on-board sensors (such as image acquisition systems or radars) have become crucial in the system architecture of vehicle autonomous driving. In intelligent driving scenarios, accurate perception of the vehicle's surrounding environment through on-board sensors is a prerequisite for ensuring safe driving on the road.

[0003] Taking the image acquisition system as an on-board camera system as an example, the on-board camera system may encounter a variety of environmental scenes (such as night road scenes or tunnel scenes, etc.) during operation. Among them, when capturing the picture content of each environmental scene, the on-board camera system can use its own capabilities to adjust the picture content of the environmental scene, so as to output images with natural brightness and clarity as much as possible. However, since the picture content of some environmental scenes is relatively complex, it is difficult for the on-board camera system to ensure that these complex picture contents have high image quality. Based on this, the on-board camera system may encounter the following problems: Problem 1, it is difficult for the on-board camera system to output images with high brightness and clarity; Problem 2, the on-board camera system does not know the user's ultimate needs, so it is impossible to output images that meet the user's needs.

[0004] To address the above issues, there are currently several solutions available.

[0005] Solution 1: The image processing device uses high-end processing chips, zone light control, high-dynamic range (HDR) multi-frame synthesis and other technologies to brighten the dark parts of the image and darken the bright parts, thereby obtaining a relatively high-quality image. However, the processing chip required for this solution is relatively expensive, which increases the image processing cost. In addition, this solution is difficult to obtain clear images in scenes with both bright and dim environments (such as tunnel scenes).

[0006] Solution 2: The developer first identifies multiple environmental scenes that require image optimization (such as tunnel scenes or scenes with low visibility, etc.), and configures the corresponding image processing flow for each environmental scene that requires image optimization. After that, when the image processing device identifies an environmental scene that requires image optimization through scene recognition technology, the corresponding image processing flow is used to process the screen content of the environmental scene that requires image optimization. However, due to the ever-changing environmental scenes, this solution is difficult to cover all environmental scenes, so it is not very universal.

[0007] However, the above-mentioned traditional solutions cannot ensure that the image acquisition system outputs images with high image quality. Summary of the invention

[0008] The present application provides an image processing method, device and vehicle to improve the quality of images output by an image acquisition system.

[0009] In the first aspect, the present application provides an image processing method, which can be applied to fields such as advanced driving assistant system (ADAS), automatic driving system or intelligent driving system. Optionally, the method can be performed by an image processing device or a component (such as a chip, a chip system or a circuit, etc.) that can support the image processing device to implement the functions required by the method. Exemplarily, the image processing device can be any device with image processing capabilities in a vehicle (such as a computing device or a control unit, etc.), such as a vehicle control unit (VCU), a vehicle dynamics control (VDC) system or an intelligent driving computing center such as an MDC (mobiledata center), etc., or it can also be any other device with the functions required to implement the method (such as a vehicle server or roadside equipment, etc.), or it can also be a functional element (such as a plug-in, component or chip, etc.) set in any of the above devices or any of the above devices. Optionally, taking the image processing device executing the image processing method as an example, in the method, the image processing device may first acquire a first image, wherein the first image is acquired by an image acquisition system located on a vehicle, and then the image processing device may determine the image quality of an area where at least one physical object contained in the first image is located; when the image quality of the area where the first physical object is located in the first image belongs to a set image quality range, the image processing device may determine a gain value of the area where the first physical object is located, and may determine a target gain value corresponding to the first image based on the gain value of the area where the first physical object is located, wherein the target gain value is used to determine the exposure compensation amount required for the image acquisition system to acquire the second image.

[0010] In this method, when the image quality of the area where the first physical object contained in the first image is located belongs to the set image quality range, the image processing device can determine that the image quality of the area where the first physical object is located needs to be optimized, so that the image area in the first image that needs image quality optimization (or can be called the image area to be optimized, such as the area where the first physical object is located) can be intelligently selected, that is, the image area whose image quality needs to be optimized is automatically screened out, so that the image quality optimization is more targeted. Then, the image processing device can calculate the parameters used to help the image acquisition system optimize the image quality of the above-mentioned image area to be optimized (such as the target gain value or the exposure compensation amount required to obtain the next image (such as the second image)), so as to assist the image acquisition system to effectively improve the quality of the output image, so that the overall performance of the image acquisition system is effectively improved. Since the present application can realize dynamic optimization of the image area, the user's visual experience can be improved. In addition, the method does not require manual targeted processing of various scenes, so that the image optimization efficiency is high and the versatility is strong.

[0011] In one possible design, the image processing device determines the image quality of the area where at least one physical object contained in the first image is located, including: after acquiring the first image, the image processing device performs object detection on the first image to obtain the confidence of each physical object in the at least one physical object contained in the first image, wherein the confidence of each physical object can be used to indicate the image quality of the area where the physical object is located. Exemplarily, the image processing device can use the existing relatively mature YOLOV5 target detection algorithm to perform object detection on the first image, can more accurately identify one or more physical objects contained in the first image, and can accurately obtain the confidence corresponding to the one or more physical objects.

[0012] In the above design, by using the confidence of the physical object detected in the first image to characterize the image quality of the area where the physical object is located, the image quality of the image area can be quantified more intuitively and clearly, thereby facilitating the effective distinction between areas with high image quality and areas with low image quality.

[0013] In one possible design, a numerical range is formed by setting two different thresholds (such as a first threshold and a second threshold, the second threshold being greater than the first threshold) as a set image quality range. The set image quality range can be used to distinguish between areas with high image quality and areas with low image quality, which helps to accurately screen areas where image quality optimization is required, thereby achieving dynamic optimization of image areas.

[0014] In one possible design, an image processing device determines a gain value of an area where a first physical object is located, including: the image processing device first determines the total number of pixels in the area where the first physical object is located, and then the image processing device determines the overall grayscale value corresponding to the area where the first physical object is located based on the total number of pixels in the area where the first physical object is located and the grayscale values ​​of multiple pixels contained in the area where the first physical object is located; and then, the image processing device determines the gain value of the area where the first physical object is located based on the overall grayscale value corresponding to the area where the first physical object is located.

[0015] In the above design, by comprehensively considering the multiple pixels included in the area where the first physical object is located, the calculation accuracy of the gain value of the area where the first physical object is located can be effectively improved, thereby effectively improving the optimization accuracy of the image quality of the area where the first physical object is located, and further making the image quality of the area where the first physical object is located meet the actual needs of users.

[0016] In one possible design, the image processing device determines the total number of pixels in the area where the first physical object is located, including: the image processing device performs object detection on the first image to obtain multiple key position coordinates in the area where the first physical object is located (such as the position coordinates of the center point of the area where the first physical object is located or the position coordinates of other multiple key points in the area where the first physical object is located (such as multiple corners of the area (such as key points on two corners (such as the upper left corner and the lower right corner))), etc.), and then the image processing device can determine the total number of pixels in the area where the first physical object is located based on the multiple key position coordinates in the area where the first physical object is located.

[0017] In the above design, by using multiple key position coordinates in the area where the first physical object is located (i.e., the area where image quality optimization is required) detected in the first image, the total number of pixels in the area where the first physical object is located can be accurately calculated.

[0018] In one possible design, the image processing device determines a target gain value corresponding to the first image based on a gain value of an area where the first physical object is located, including: when the number of first physical objects is one, the image processing device uses the gain value of the area where the first physical object is located as the target gain value corresponding to the first image, or when the number of first physical objects is multiple, the image processing device may use an average value or a weighted average value of the gain values ​​of the areas where multiple first physical objects are located as the target gain value corresponding to the first image.

[0019] In the above design, different calculation methods can be flexibly selected to calculate the target gain value corresponding to the first image according to the number of the first physical objects, so that the calculation of the target gain value can be more targeted and the calculation result can be more accurate.

[0020] In one possible design, the average value of the gain values ​​of the area where the multiple first physical objects are located can be determined based on the gain values ​​of the area where the multiple first physical objects are located and the number of the multiple first physical objects, and the weighted average value of the gain values ​​of the area where the multiple first physical objects are located can be determined based on the gain values ​​of the area where the multiple first physical objects are located and the total number of pixels in the area where the multiple first physical objects are located.

[0021] In the above design, by using the average value or weighted average value of the gain values ​​of the areas where multiple first physical objects are located as the target gain value, the comprehensive optimization degree of the areas where image quality optimization is required can be accurately characterized, and the image quality of the areas where multiple first physical objects are located can be optimized based on the target gain value, which can effectively improve the display effect of the next image acquired by the image acquisition system. For example, the picture brightness and picture clarity of the next image are increased to a certain extent compared with the picture brightness and picture clarity of the current image.

[0022] In a possible design, the exposure compensation amount required for the image acquisition system to acquire the second image may be determined according to the target gain value and the exposure compensation amount required for the image acquisition system to acquire the first image.

[0023] In the above design, by integrating the target gain value (i.e., the parameter used to optimize the image quality of certain image areas) based on the exposure compensation amount required by the image acquisition system to acquire the first image, the exposure compensation amount required by the image acquisition system to acquire the next image can be accurately determined. Afterwards, the image acquisition system can process the next environmental information acquired accordingly based on the exposure compensation amount, and can acquire the next image with increased picture clarity, picture brightness, etc., so that the image acquisition system can output images with better user experience.

[0024] In a second aspect, the present application provides a possible image processing device. Exemplarily, the image processing device is an image processing device having the functions required to implement the method of the first aspect, or may be an element (such as a plug-in, component or chip, etc.) provided in the image processing device having the functions required to implement the method of the first aspect, or may be any other device (such as a terminal device, etc.) having the functions required to implement the method of the first aspect, or may be an element provided in any other device having the functions required to implement the method of the first aspect.

[0025] Taking the image processing device as a chip set in the image processing equipment as an example, when the image processing device is a chip, the image processing device includes a transceiver and a processor, but does not include a memory. Among them, the transceiver exists in the form of an input and output interface, and the input and output interface is used for the chip to realize the transceiver of the image processing device. The input and output interface may include an input interface and / or an output interface. The processor is used to read and execute the corresponding computer program or instruction, so that the corresponding function of the image processing device is realized.

[0026] When the image processing device is used to implement any possible method of the first aspect, the beneficial effects can be referred to the corresponding description in the first aspect, which will not be described in detail here. The image processing device has the function of implementing the method of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the image processing device includes an acquisition module and a processing module. The acquisition module is used to acquire a first image. The first image is acquired by an image acquisition system located on a vehicle. The processing module is used to determine the image quality of the area where at least one physical object contained in the first image is located. The processing module is also used to determine the gain value of the area where the first physical object is located when the image quality of the area where the first physical object is located in the first image belongs to the set image quality range. The processing module is also used to determine the target gain value corresponding to the first image according to the gain value of the area where the first physical object is located. The target gain value is used to determine the exposure compensation amount required for the image acquisition system to acquire the second image. These modules can perform the corresponding functions in any possible design of the first aspect, and refer to the detailed description in the method example for details, which will not be described in detail here.

[0027] In a third aspect, the present application provides a possible image processing device, which includes a transceiver and a processor. Optionally, the image processing device also includes a memory. The memory is used to store computer programs or instructions, and the processor is coupled to the memory and the transceiver. When the processor executes the computer program or instructions, the image processing device executes the method in any possible design of the first aspect above. In one example, the transceiver can be a transceiver device with integrated data transceiver function. In another example, the transceiver can also be composed of a transmitter and a receiver, wherein the transmitter is used to send data and the receiver is used to receive data.

[0028] In a fourth aspect, the present application provides a possible image processing system, which includes an image acquisition system and an image processing device, wherein the image processing device can be used to execute the method in any possible design of the first aspect above, and the image acquisition system can be used to obtain images corresponding to the surrounding environment information of the vehicle. Exemplarily, the number of image acquisition systems included in the image processing system can be one or more, and the number of image processing devices can be one or more, and the present application does not limit this. The image processing system can be deployed on a vehicle. Accordingly, the present application also provides a vehicle including the above-mentioned image processing system.

[0029] In a fifth aspect, the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are run on a computer, the computer executes the method in any possible design of the first aspect above.

[0030] In a sixth aspect, the present application provides a computer-readable storage medium, which stores a computer program or instruction. When the computer program or instruction is executed by a computer, the computer executes the method in any possible design of the first aspect above.

[0031] In a seventh aspect, the present application further provides a chip, which is coupled to a memory, and is used to read a computer program stored in the memory and execute a method in any possible design of the first aspect above.

[0032] In an eighth aspect, the present application further provides a chip system, the chip system including a processor for supporting a computer device to implement the method in any possible design of the first aspect. In one possible design, the chip system also includes a memory for storing programs and data necessary for the computer device. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0033] Based on the implementations provided in the above aspects, the present application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of a system architecture provided by an embodiment of the present application is exemplarily shown;

[0035] Figure 2 A schematic diagram exemplarily illustrates a flow chart of an image processing method provided in an embodiment of the present application;

[0036] Figure 3 A schematic diagram of a first image provided by an embodiment of the present application is exemplarily shown;

[0037] Figure 4 A schematic diagram showing the structure of an image processing device provided by an embodiment of the present application is exemplified;

[0038] Figure 5 The following is a schematic diagram showing the structure of another image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0040] The following is an introduction to the system architecture applicable to the image processing method provided by the present application. It should be noted that these introductions are for the convenience of understanding by those skilled in the art and do not limit the scope of protection claimed by the present application.

[0041] Figure 1 The following is a schematic diagram of a system architecture applicable to the embodiment of the present application. Figure 1 As shown, the system architecture may include an image acquisition system (such as a camera system) 100 and an image processing device 200 .

[0042] The image acquisition system 100 may be arranged on a vehicle to collect or capture the information of the vehicle's surrounding environment to generate a corresponding image; the image processing device 200 may be arranged on a vehicle, or may be arranged on other devices outside the vehicle (such as a vehicle server or a roadside device, etc.), to process the first image output by the image acquisition system 100 accordingly, and obtain parameters for optimizing the image quality of the image area to be optimized in the first image (such as a target gain value or a camera parameter such as an exposure compensation amount required for obtaining the next image), and then the image processing device 200 may feed back the parameters to the image acquisition system 100, so that the image acquisition system 100 adjusts the corresponding parameters configured in the image acquisition system 100 according to the parameters, so that the image acquisition system 100 can output an image with higher image quality. The vehicle server may also be referred to as a cloud server, a vehicle cloud server, a vehicle cloud, a vehicle cloud, a cloud, a cloud server, a cloud controller or a vehicle network server, etc. In a possible implementation, the image processing device 200 may be arranged in the image acquisition system 100 as a functional element of the image acquisition system 100. In another implementation, the image processing device 200 may also be separately disposed in a vehicle. In yet another possible implementation, the image processing method provided by the embodiment of the present application and executed by the image processing device 200 may also be integrated into the vehicle-mounted autonomous driving software.

[0043] Exemplarily, the image processing device 200 may include an object detection module, an object classification module, and an image quality optimization module (such as an ISP optimizer). Optionally, the image processing device 200 may also include an acquisition module for acquiring a first image from the image acquisition system 100. In one example, the image acquisition system 100 includes an image acquisition device (or may be referred to as an image acquisition device). In another example, when the image acquisition system includes an image acquisition device and an image signal processor (ISP), the image acquisition device and the image processor may be separate and independent devices, or the image processor may also be integrated in the image acquisition device.

[0044] Among them, the image acquisition device can be used to capture or collect the surrounding environment information of the vehicle to form an original image.

[0045] The image processor is used to process and optimize (or adjust) the original image output by the image acquisition device, and output a first image that better meets user needs.

[0046] The object detection module is used to perform object detection on the first image output by the image acquisition device or the image processor, and obtain the detection result of each physical object in at least one physical object contained in the first image (such as category, bounding box, confidence, position coordinates in the image, size of the area where it is located, or multiple key position coordinates in the area where it is located, etc.). The confidence of each physical object can be used to indicate the image quality of the area where the physical object is located. Optionally, the image processing device 200 can also output the detection result of each physical object in at least one physical object, and the image processing device 200 can send the detection result of each physical object in at least one physical object to other processing devices (such as other vehicle-mounted devices) in the vehicle (such as an autonomous driving vehicle).

[0047] The object classification module is used to classify at least one physical object output by the object detection module, determine which physical objects are located in areas where image quality needs to be optimized, and also determine which physical objects are located in areas where image quality does not need to be optimized.

[0048] The image quality optimization module is used to determine parameters that assist the image acquisition system 100 in optimizing the image quality of the image region to be optimized in the acquired image.

[0049] Optionally, the image processor, the object detection module, the object classification module, and the image quality optimization module can all be implemented by software, or can all be implemented by hardware. Exemplarily, the implementation of the object detection module is described below by taking the object detection module as an example. Similarly, the implementation of the image processor, the object classification module, and the image quality optimization module can refer to the implementation of the object detection module, which will not be repeated here.

[0050] When implemented by software, the object detection module may be an application or code block running on a computing device (or control unit, etc.) with image processing capabilities in the vehicle. The computing device may be at least one of a physical host, a virtual machine, a container, and the like. Furthermore, the computing device may be one or more. For example, the object detection module may be an application running on multiple hosts / virtual machines / containers.

[0051] When implemented by hardware, the object detection module may include at least one computing device, such as a server, etc. Alternatively, the object detection module may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0052] In addition, it should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The functional modules in the embodiments of the present application can be integrated into one module, or each module can exist physically separately, or two or more modules can be integrated into one module. For example, an object detection module, an object classification module, and an image quality optimization module can be integrated into one module, or an object detection module, an object classification module, and an image quality optimization module can be the same module. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0053] It should be noted that Figure 1 Only a schematic system architecture is provided, which is intended to more clearly illustrate the technical solution of the embodiment of the present application, and does not limit the system architecture applicable to the image processing method provided by the present application. Figure 1 The form and quantity of each device in the system architecture shown are for example only and do not constitute a limitation to the present application. Figure 1The names of the various devices in the system architecture shown are only examples, and the names of the various devices in the specific implementation may also be other names, which are not specifically limited in this application. In addition, it is known to those skilled in the art that with the evolution of the system architecture, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0054] As described in the background technology, the quality of the image output by the image acquisition system in the existing autonomous driving vehicle is not high, so that the image processing device has low detection accuracy for the physical objects included in the image, resulting in the vehicle being unable to effectively identify the physical objects (such as pedestrians, obstacles or other vehicles, etc.) that conflict with the vehicle in the vehicle's driving path during the autonomous driving process, thereby causing certain safety risks for the vehicle during the autonomous driving process. In view of this, the embodiment of the present application provides an image processing method for effectively improving the quality of the image output by the image acquisition system, so that the detection accuracy of the physical objects included in the image is effectively improved, thereby effectively improving the safety of the vehicle during the autonomous driving process.

[0055] The following is based on Figure 1 The system architecture shown in the figure provides a detailed introduction to the specific implementation of the image processing method in the embodiment of the present application.

[0056] Figure 2 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application. The method is applicable to Figure 1 The method may be performed by an image processing device or a component (such as a chip, a chip system or a circuit, etc.) that can support the image processing device to implement the functions required by the method. Figure 1 The image processing device 200 shown in the figure may also be a functional element (such as a plug-in, component or chip, etc.) set in the image processing device 200, and the functional element has the function of implementing the image processing method. In one example, the image processing device 200 may be any device in the vehicle with image processing capabilities (such as a computing device or a control unit, etc.), such as a VCU, a VDC system or an MDC in the vehicle. In another example, the image processing device 200 may be other devices located outside the vehicle (such as a vehicle server or a roadside device, etc.), or it may be a functional element with an image processing function set in other devices, and the functional element has the function of implementing the image processing method. In order to facilitate the introduction of the technical solution provided in the embodiments of the present application, the image processing device executing the image processing method is taken as an example for introduction below. Figure 2 As shown, the method includes steps 201-204.

[0057] Step 201: An image processing device acquires a first image.

[0058] In one example, when the image acquisition system includes an image acquisition device but does not include an ISP, the first image may be an original image acquired by the image acquisition device in the image acquisition system located on the vehicle. For example, when vehicle A is driving, after the image acquisition device in the image acquisition system located on vehicle A captures certain environmental information to form an original image (such as image 1), image 1 may be sent as the first image to the image processing device, or image 1 may be cached. When the image acquisition device caches the acquired original image, the image acquisition device may send the cached original image to the image processing device in response to an image request from the image processing device.

[0059] In another example, when the image acquisition system includes an image acquisition device but does not include an ISP, the first image is an image obtained by processing the original image acquired by the image acquisition system on the vehicle through an external ISP. Continuing with vehicle A as an example, when the image acquisition device in the image acquisition system on vehicle A captures a certain piece of environmental information to form an original image (such as image 1), image 1 can be sent to the ISP. After receiving image 1, the ISP processes and optimizes image 1 to obtain a first image (such as an RGB image corresponding to image 1). Afterwards, the ISP can send the first image to the image processing device, or it can cache the first image. When the ISP caches the first image, the ISP can send the cached first image to the image processing device in response to an image request from the image processing device.

[0060] In another example, when the image acquisition system includes an image acquisition device and an ISP, the first image is the original image output by the image acquisition system and processed by the ISP. Continuing with vehicle A as an example, when the image acquisition device in the image acquisition system located on vehicle A captures a certain environmental information to form an original image (such as image 1), it can send image 1 to the ISP in the image acquisition system. After receiving image 1, the ISP can process and optimize image 1 to obtain the first image. Afterwards, the image acquisition system can send the first image to the image processing device, or it can cache the first image. When the image acquisition system caches the first image, the image acquisition system can send the cached first image to the image processing device in response to an image request from the image processing device.

[0061] Exemplarily, the image acquisition device may include, but is not limited to: a visible light camera, a depth camera (ie, a three-dimensional camera), a fisheye camera, a monocular camera, or a depth camera.

[0062] Step 202: The image processing device determines the image quality of a region where at least one physical object contained in the first image is located.

[0063] Exemplarily, physical objects may include but are not limited to pedestrians, vehicles (such as motor vehicles or non-automatic vehicles, etc.) or obstacles (such as stones, cartons, traffic cones, traffic signs set on the roadside, street lights or electric poles, etc.). For example, the above-mentioned motor vehicles may include but are not limited to unmanned vehicles, intelligent vehicles (such as automated guided vehicles (AGV)), electric vehicles, digital vehicles, intelligent manufacturing vehicles or other forms of vehicles, etc.; the above-mentioned non-motor vehicles may include but are not limited to electric bicycles, bicycles, tricycles or rickshaws, etc.

[0064] After the image processing device acquires the first image, it can perform object detection on the first image to obtain detection results such as at least one physical object, the category of each physical object in at least one physical object, and the confidence of each physical object in at least one physical object. Among them, the confidence of each physical object in at least one physical object can be used to indicate the image quality of the area where the physical object is located. Exemplarily, the value of the confidence is between 0 and 1. For example, if the confidence of a physical object is 0, it means that the image quality of the area where the physical object is located is very low, or assuming that the confidence of the physical object is 1, it means that the image quality of the area where the physical object is located is very high.

[0065] Optionally, the image processing device performs object detection on the first image, and can also obtain the position coordinates of each physical object in at least one physical object in the first image, the bounding box of each physical object in at least one physical object (such as a rectangular bounding box, or can be called a rectangular box), the size of the area where each physical object in at least one physical object is located (such as the number of pixels included in the image area occupied by the physical object itself in the first image or the number of pixels included in the bounding box of the physical object (such as the area where the rectangular box is located)), or multiple key position coordinates of the area where each physical object in at least one physical object is located (such as the position coordinates of the center point of the area where each physical object is located or the position coordinates of multiple other key points in the area where each physical object is located (such as the key points at multiple corners of the area (such as two corners (such as the upper left corner and the lower right corner) or three corners or four corners))), etc.). Among them, the position coordinates of the center point of the area where each physical object is located can also be understood as the number of pixels of the central pixel point of the area where the physical object is located in the horizontal or vertical direction of the image, and the position coordinates of the key points at multiple corners in the area where each physical object is located can also be understood as the number of pixels of the pixel points at multiple corners in the area where the physical object is located in the horizontal or vertical direction of the image.

[0066] For example, the image processing device can be based on a deep learning target detection algorithm (such as YOLOV5 (Youonly look once version 5, YOLO series target detection) target detection algorithm), a single-stage target detection algorithm (such as single-stage multi-frame detection (single shot multibox detector, SSD) or a two-stage target detection algorithm (such as faster region convolutional neural networks, Faster ... R-CNN) or other target detection algorithms perform object detection (or can be called target detection) on the first image, identify the category of the physical object included in the first image, and can identify the confidence corresponding to the physical object, the position coordinates of the physical object in the first image, the bounding box of the physical object, multiple key position coordinates in the area where the physical object is located, or the size of the area where the physical object is located. Among them, for the confidence of each physical object, the confidence of the physical object is used to represent the "subjective confidence level" of the image processing device in the detection result of the physical object output by itself. For a mature and reliable image processing device, when the confidence of a certain physical object is low, it can be proved that the image quality of the area where the physical object is located is not enough to support deterministic judgment, so it is necessary to optimize the image quality of the area where the physical object is located. It should be noted that there are generally deep learning-based target detection algorithms or other target detection algorithms in the automatic driving system of the vehicle. When the image processing device located in the vehicle needs a deep learning-based target detection algorithm or other target detection algorithm, the corresponding algorithm result can be directly called, which effectively reduces the operating pressure of the image processing device, and can make the image processing method provided in the embodiment of the present application occupy less additional hardware resources and be easy to implement on low-end and medium-end hardware.

[0067] For example, taking the YOLOV5 target detection algorithm as an example, the image processing device can use the YOLOV5 target detection algorithm to perform object detection on the first image, identify the category of the physical object (such as vehicle B) included in the first image, and can identify the bounding box of the physical object, the position coordinates of the physical object in the first image, the size of the area where the physical object is located, multiple key position coordinates in the area where the physical object is located, or the confidence corresponding to the physical object, etc. In this way, the image processing device can predict the category (or type), size, position coordinates of the physical object in the first image, or multiple key position coordinates in the area where the physical object is located, etc. of the physical object from the first image by using a deep learning target detection algorithm, which helps to realize fast and accurate recognition of physical objects and can improve the robustness of physical object recognition.

[0068] Figure 3A possible first image schematic diagram provided in an embodiment of the present application. Figure 3 In the example, the first image includes 6 physical objects (such as object 1, object 2, object 3, object 4, object 5 and object 6), where object 1, object 2 and object 3 are street lights, object 4 is an electric bicycle, object 5 is a pedestrian, and object 6 is a car. Exemplarily, taking the target detection algorithm acquired by the image processing device as the YOLOV5 target detection algorithm as an example, after the image processing device acquires the first image, the image processing device can use the YOLOV5 target detection algorithm to perform object detection on the first image to obtain the detection results of the 6 objects included in the first image.

[0069] Exemplarily, the detection result of object 1 is: the category of object 1 is street lamp, the rectangular box corresponding to object 1, the size of the area where the rectangular box corresponding to object 1 is located (that is, the total number of multiple pixels included in the area where the rectangular box corresponding to object 1 is located), the coordinates of the pixel points at multiple corners of the area where the rectangular box corresponding to object 1 is located (such as the coordinates of the pixel point at the upper left corner, the coordinates of the pixel point at the lower right corner, etc.), the position coordinates of object 1 in the first image, and the confidence level corresponding to object 1 (for example, the confidence level is c1).

[0070] The detection result of object 2 is: the category of object 2 is street lamp, the rectangular box corresponding to object 2, the size of the area where the rectangular box corresponding to object 2 is located (that is, the total number of multiple pixels included in the area where the rectangular box corresponding to object 2 is located), the coordinates of the pixel points at multiple corners of the area where the rectangular box corresponding to object 2 is located (such as the coordinates of the pixel points at the upper left corner, the coordinates of the pixel points at the lower right corner, etc.), the position coordinates of object 2 in the first image, and the confidence level corresponding to object 2 (for example, the confidence level is c2).

[0071] The detection result of object 3 is: the category of object 3 is street lamp, the rectangular box corresponding to object 3, the size of the area where the rectangular box corresponding to object 3 is located (that is, the total number of multiple pixels included in the area where the rectangular box corresponding to object 3 is located), the coordinates of the pixel points at the multiple corners of the area where the rectangular box corresponding to object 3 is located (such as the coordinates of the pixel points at the upper left corner, the coordinates of the pixel points at the lower right corner, etc.), the position coordinates of object 3 in the first image, and the confidence level corresponding to object 3 (for example, the confidence level is c3).

[0072] The detection result of object 4 is: the category of object 4 is electric bicycle, the rectangular box corresponding to object 4, the size of the area where the rectangular box corresponding to object 4 is located (that is, the total number of multiple pixels included in the area where the rectangular box corresponding to object 4 is located), the coordinates of the pixel points at multiple corners of the area where the rectangular box corresponding to object 4 is located (such as the coordinates of the pixel points at the upper left corner, the coordinates of the pixel points at the lower right corner, etc.), the position coordinates of object 4 in the first image, and the confidence level corresponding to object 4 (for example, the confidence level is c4).

[0073] The detection result of object 5 is: the category of object 5 is pedestrian, the rectangular box corresponding to object 5, the size of the area where the rectangular box corresponding to object 5 is located (that is, the total number of multiple pixels included in the area where the rectangular box corresponding to object 5 is located), the coordinates of the pixel points at the multiple corners of the area where the rectangular box corresponding to object 5 is located (such as the coordinates of the pixel point at the upper left corner, the coordinates of the pixel point at the lower right corner, etc.), the position coordinates of object 5 in the first image, and the confidence level corresponding to object 5 (for example, the confidence level is c5).

[0074] The detection result of object 6 is: the category of object 6 is car, the rectangular box corresponding to object 6, the size of the area where the rectangular box corresponding to object 6 is located (that is, the total number of multiple pixels included in the area where the rectangular box corresponding to object 6 is located), the coordinates of the pixel points at the multiple corners of the area where the rectangular box corresponding to object 6 is located (such as the coordinates of the pixel points at the upper left corner, the coordinates of the pixel points at the lower right corner, etc.), the position coordinates of object 6 in the first image, and the confidence level corresponding to object 6 (for example, the confidence level is c6).

[0075] Afterwards, the image processing device may send the relevant information of one or more physical objects included in the identified first image (such as the category of each physical object in the one or more physical objects, the position coordinates of each physical object in the first image, the size of the area where each physical object is located, multiple key position coordinates in the area where each physical object is located, the bounding box of each physical object and the confidence of each physical object, etc.) to other processing devices in the vehicle (such as an autonomous driving car), so that other processing devices can make corresponding decisions based on the relevant information of the one or more physical objects. For example, taking the other processing device as an autonomous driving algorithm module (such as a decision module) in an autonomous driving car as an example, after obtaining the relevant information of the one or more physical objects, the decision module can make some decisions based on the relevant information of the one or more physical objects, such as deciding the driving behavior of the autonomous driving car. For example, when the decision module determines that there is an obstacle in the first image, it can send an instruction to the control unit in the autonomous driving car, and the instruction is used to instruct the control unit in the autonomous driving car to control the autonomous driving car to avoid the obstacle. When the control unit in the autonomous driving car receives the instruction, it can control the autonomous driving car to avoid the obstacle according to the instruction, so that the driving safety of the autonomous driving car can be effectively ensured. Exemplarily, the autonomous driving algorithm module can be any practical algorithm module that takes the relevant information of the physical object as input.

[0076] In one implementation, after obtaining relevant information of at least one physical object included in the first image (such as the category of each physical object in the at least one physical object, the position coordinates in the first image, the size of the area where it is located, multiple position coordinates in the area, a bounding box or confidence, etc.), the image processing device can send the relevant information of the at least one physical object to a processing unit in the vehicle's automatic driving system, so that the processing unit can control the vehicle's driving behavior based on the relevant information of the at least one physical object.

[0077] In another implementation, after acquiring relevant information of at least one physical object included in the first image, the image processing device can calculate relevant parameters for assisting the image acquisition system to output images with higher image quality (such as the target gain value corresponding to the first image or the exposure compensation amount required to acquire the next image) based on the relevant information of the at least one physical object, that is, execute the process from step 203 to step 204.

[0078] Step 203: When the image quality of the area where the first physical object is located in the first image belongs to the set image quality range, the image processing device determines a gain value of the area where the first physical object is located.

[0079] After the image processing device obtains the confidence of each physical object in at least one physical object included in the first image (which can be used to reflect the image quality of the area where the physical object is located), in order to accurately determine which physical object areas need to have their image quality optimized, that is, to accurately screen which physical object areas need to be optimized, the image processing device can screen which image areas need to have their image quality optimized, such as the area where a physical object or several physical objects are located, by determining whether the image quality of the area where each physical object in at least one physical object identified in the first image is located belongs to the set image quality range. The set image quality range is a numerical range determined by the first threshold and the second threshold, and the first threshold is less than the second threshold.

[0080] For the first physical object included in the first image, the image processing device may compare the image quality of the area where the first physical object is located with the first threshold and the second threshold to determine whether the image quality of the area where the first physical object is located needs to be optimized. The first physical object is any one of the at least one physical object included in the first image. When the image quality of the area where the first physical object is located is greater than the second threshold, the image processing device may determine that the area where the first physical object is located is a high-quality image area, and the image quality of the area where the first physical object is located does not need to be optimized, or it may also determine that the first physical object is a high-quality object (i.e., the image clarity of the area where the first physical object is located is relatively high, the brightness is relatively natural, and the image quality of the area where the first physical object is located does not need to be optimized). When the image quality of the area where the first physical object is located is greater than the first threshold and less than the second threshold, the image processing device may determine that the area where the first physical object is located is a low-quality image area, and the image quality of the area where the first physical object is located needs to be optimized, or it may determine that the first physical object is a low-quality object (i.e., the image clarity of the area where the first physical object is located is relatively low, the brightness is not very natural, and the image quality of the area where the first physical object is located needs to be optimized). When the image quality of the area where the first physical object is located is less than the first threshold, the image processing device may discard the first physical object, that is, there is no need to further process the first physical object. In other words, when the image quality of the area where the first physical object is located is less than the first threshold, the image processing device may regard the area where the first physical object is located as the image background area.

[0081] Exemplarily, the following takes the confidence level of a physical object included in an image as an example to introduce the implementation process of the image processing device determining which physical object regions need to have their image quality optimized.

[0082] For a physical object (such as physical object Ai) in a certain image (such as image 2), when the confidence ci of the physical object Ai is greater than the second threshold (such as a), the image processing device can determine that the physical object Ai is a high-quality object; when the confidence ci of the physical object Ai is greater than the first threshold (such as b) and the confidence ci is less than the second threshold (such as a), the image processing device can determine that the physical object Ai is a low-quality object; when the confidence ci of the physical object Ai is less than the first threshold (such as b), the image processing device can discard the physical object Ai. Wherein, a>b.

[0083] For example, continuing with Figure 3 Take the six physical objects (such as object 1, object 2, object 3, object 4, object 5 and object 6) included in the illustrated first image as an example, where object 1, object 2 and object 3 are street lights, object 4 is an electric bicycle, object 5 is a pedestrian, object 6 is a car, the confidence level c1 corresponding to object 1, the confidence level c2 corresponding to object 2, the confidence level c3 corresponding to object 3, the confidence level c4 corresponding to object 4, the confidence level c5 corresponding to object 5, and the confidence level c6 corresponding to object 6. It is assumed that the confidence level c1 is greater than a, the confidence level c2 is greater than a, the confidence level c3 is less than a and greater than b, the confidence level c4 is greater than a, the confidence level c5 is greater than a, and the confidence level c6 is less than a and greater than b.

[0084] After the image processing device obtains the confidence c1 corresponding to object 1, the confidence c2 corresponding to object 2, the confidence c3 corresponding to object 3, the confidence c4 corresponding to object 4, the confidence c5 corresponding to object 5, and the confidence c6 corresponding to object 6, for object 1, when the confidence c1 corresponding to object 1 is greater than a, the image processing device determines that object 1 is a high-quality object, and can determine that the image quality of the area where object 1 is located does not need to be optimized. For object 2, when the confidence c2 corresponding to object 2 is greater than a, the image processing device determines that object 2 is a high-quality object, and can determine that the image quality of the area where object 2 is located does not need to be optimized. For object 3, when the confidence c3 corresponding to object 3 is less than a and greater than b, the image processing device determines that object 3 is a low-quality object, and can determine that the image quality of the area where object 3 is located needs to be optimized. For object 4, when the confidence c4 corresponding to object 4 is greater than a, the image processing device determines that object 4 is a high-quality object, and can determine that the image quality of the area where object 4 is located does not need to be optimized. For object 5, when the confidence c5 corresponding to object 5 is greater than a, the image processing device determines that object 5 is a high-quality object, and can determine that the image quality of the area where object 5 is located does not need to be optimized. For object 6, when the confidence c6 corresponding to object 6 is less than a and greater than b, the image processing device determines that object 6 is a low-quality object, and can determine that the image quality of the area where object 6 is located needs to be optimized.

[0085] Based on the above content, the following describes the implementation process of the image processing device determining the gain value corresponding to one or more physical object regions after obtaining the region where one or more physical objects requiring image quality optimization are located.

[0086] After the image processing device obtains one or more physical object regions that need image quality optimization, the total number of pixels in the region where the physical object is located can be determined for each physical object region. After that, the image processing device can determine the overall grayscale value corresponding to the region where the physical object is located based on the total number of pixels in the region where the physical object is located and the grayscale values ​​of multiple pixels included in the region where the physical object is located. Then, the image processing device can determine the gain value of the region where the physical object is located based on the overall grayscale value corresponding to the region where the physical object is located. Exemplarily, the overall grayscale value can be one of the following: grayscale mean, median of grayscale values ​​of multiple pixels, weighted grayscale mean or pooling value, etc. For each physical object region, in one example, after the image processing device calculates the grayscale mean corresponding to the region where the physical object is located based on the total number of pixels in the region where the physical object is located and the grayscale values ​​of multiple pixels included in the region where the physical object is located, the grayscale mean can be used as the overall grayscale value corresponding to the region where the physical object is located. In another example, after determining the median of the grayscale values ​​of multiple pixels included in the area where the physical object is located, the image processing device can use the median as the overall grayscale value corresponding to the area where the physical object is located. In another example, after calculating the weighted grayscale mean corresponding to the area where the physical object is located based on the grayscale values ​​of multiple pixels included in the area where the physical object is located, the image processing device can use the weighted grayscale mean as the overall grayscale value corresponding to the area where the physical object is located. In another example, after calculating the pooling value corresponding to the area where the physical object is located based on the grayscale values ​​of multiple pixels included in the area where the physical object is located, the image processing device can use the pooling value as the overall grayscale value corresponding to the area where the physical object is located. It should be understood that the above content is only an exemplary introduction to several implementation methods of image processing devices determining the overall grayscale value corresponding to the area where a physical object is located. The image processing device can also determine the overall grayscale value corresponding to the area where the physical object is located by adopting other algorithms, and the embodiments of the present application will not list them one by one.

[0087] For example, the following uses the overall grayscale value as the grayscale mean value as an example to introduce the image processing device determining the gain value of a region where a physical object is located. The gain value of a region where a physical object where image quality optimization is required can satisfy the following form:

[0088]

[0089] Among them, Gain iUsed to indicate the gain value of a region where a physical object where image quality optimization is required is located; It is used to represent the grayscale mean value of the area where the physical object is located; N 0 It is used to represent a constant, such as 128, which is a constant linked to the color depth of an image acquisition system (such as a camera system). The constant can be set accordingly according to different image acquisition systems and actual needs of users.

[0090] By way of example, the following describes the implementation process of the image processing device determining the total number of pixels in the area where each physical object requiring image quality optimization is located through the following possible implementation methods.

[0091] Implementation method one: For each area where a physical object is located and image quality optimization is required, the image processing device may use the size of the area where the physical object is located (such as the total number of pixels included in the area where the rectangular frame corresponding to the physical object is located, which can also be understood as the area size composed of the length and width of the pixels in the area where the rectangular frame corresponding to the physical object is located) as the total number of pixels in the area where the physical object is located.

[0092] Implementation method two: For each area where a physical object needs to have its image quality optimized, the image processing device can determine the total number of pixels in the area where the physical object is located based on multiple key position coordinate areas of the area where the physical object is located (such as the position coordinates of the central pixel point of the area where the rectangular frame corresponding to the physical object is located and the position coordinates of the key point at a corner (such as the upper left corner or the lower right corner, etc.), the position coordinates of the central pixel point of the area where the rectangular frame corresponding to the physical object is located and the position coordinates of the key points at two corners (such as the upper left corner and the lower right corner, or the upper left corner and the upper right corner, or the lower left corner and the upper right corner, etc.), or the position coordinates of the key points at multiple corners (such as the upper left corner and the lower right corner) of the area where the rectangular frame corresponding to the physical object is located, etc.).

[0093] For example, taking the position coordinates of the key points at the two corners (such as the upper left corner and the lower right corner) of the area where the rectangular frame of a physical object corresponding to the image quality optimization is located as an example, the total number of pixels in the area where the physical object is located can satisfy the following form:

[0094] S i =(x ib -x ia )×(y ib -y ia )

[0095] Among them, S i It is used to indicate the total number of pixels in the area where a physical object needs to be optimized for image quality; iaIt is used to represent the horizontal coordinate of the key point on the upper left corner of the area where the rectangular box corresponding to the physical object is located; ia It is used to represent the vertical coordinate of the key point on the upper left corner of the area where the rectangular box corresponding to the physical object is located; ib It is used to represent the horizontal coordinate of the key point on the lower right corner of the area where the rectangular box corresponding to the physical object is located; ib Used to represent the vertical coordinate of the key point at the lower right corner of the area where the rectangular box corresponding to the physical object is located.

[0096] For example, the grayscale mean corresponding to the area where a physical object where image quality optimization is required is located may satisfy the following form:

[0097]

[0098] in, It is used to represent the grayscale mean value of the area where a physical object needs to be optimized for image quality; g m It is used to represent the gray value of a certain pixel in the area where the physical object is located; S i The total number of pixels used to represent the area where the physical object is located.

[0099] The grayscale value of each pixel in the plurality of pixels included in the area where the physical object is located is determined according to the color component value (such as R value, G value and B value) of the pixel. Exemplarily, the grayscale value of a certain pixel in the plurality of pixels may satisfy the following form:

[0100] g m =0.2126R m +0.7152G m +0.0722B m

[0101] Among them, g m Used to represent the gray value of a pixel; (R m ,G m ,B m ) is used to represent the color information of the pixel.

[0102] Step 204: The image processing device determines a target gain value corresponding to the first image according to the gain value of the area where the first physical object is located.

[0103] After the image processing device obtains the gain values ​​corresponding to the regions where at least one physical object requiring image quality optimization is located in the first image, the image processing device can determine the target gain value corresponding to the first image based on the gain values ​​corresponding to the regions where the at least one physical object is located.

[0104] By way of example, the following takes the first physical object that needs image quality optimization as an example, and introduces the implementation process of the image processing device determining the target gain value corresponding to the first image through the following possible implementation methods. The first physical object is any one of the at least one physical object that needs image quality optimization.

[0105] Method 1: When the number of the first physical object is one, the image processing device may use the gain value of the area where the one first physical object is located as the target gain value corresponding to the first image.

[0106] Method 2: When there are multiple first physical objects, the image processing device may calculate the average value of the gain values ​​of the regions where the multiple first physical objects are located. Then, the image processing device may use the average value as the target gain value corresponding to the first image.

[0107] For example, the average value of the gain values ​​of the regions where the multiple first physical objects are located may satisfy the following form:

[0108]

[0109] Wherein, Gain′ is used to represent the average gain value of the region where the multiple first physical objects are located, n is used to represent the number of the multiple first physical objects, Gain i Used to represent the gain value of the area where a first physical object is located.

[0110] Method 3: When there are multiple first physical objects, the image processing device may calculate a weighted average of the gain values ​​of the regions where the multiple first physical objects are located. Then, the image processing device may use the weighted average as the target gain value corresponding to the first image.

[0111] For example, the weighted average value of the gain values ​​of the regions where the multiple first physical objects are located may satisfy the following form:

[0112]

[0113] Wherein, Gain is used to represent the weighted average of the gain values ​​of the regions where the multiple first physical objects are located, S i Used to indicate the total number of pixels in the area where a first physical object is located. i Used to represent the gain value of the area where the first physical object is located.

[0114] Optionally, when there is no low-quality object in a certain image acquired by the image acquisition system, the image processing device may calculate a target gain value corresponding to the image as 0 for the image.

[0115] In a possible implementation, after determining the target gain value corresponding to the first image, the image processing device may determine the exposure compensation amount (used to represent the compensation amount for the exposure value) required for assisting the image acquisition system to acquire the second image based on the target gain value corresponding to the first image and the exposure compensation amount (used to represent the compensation amount for the exposure value) required for the image acquisition system to acquire the first image. Afterwards, the image processing device may send the exposure compensation amount required for assisting the image acquisition system to acquire the second image to the image acquisition system. The acquisition time of the second image is after the acquisition time of the first image. In other words, the acquisition time of the environmental information corresponding to the second image is after the acquisition time of the environmental information corresponding to the first image. After receiving the exposure compensation amount required for assisting the image acquisition system to acquire the second image, the image acquisition system may intervene in the acquisition process of the next image corresponding to the environmental information according to the exposure compensation amount to obtain the second image.

[0116] Exemplarily, the exposure compensation amount required for assisting the image acquisition system to acquire the second image may satisfy the following form:

[0117] EV new =EV old +k×Gain (or Gain′)

[0118] Among them, EV new Used to indicate the exposure compensation required by the image acquisition system to acquire the second image; EV old It is used to indicate the exposure compensation amount required by the image acquisition system to acquire the first image; k is a parameter freely adjustable by the developer of the image acquisition system, and the parameter k can determine the response speed of the image acquisition system.

[0119] For example, when EV new = 0, the image acquisition system outputs a normal image that it considers appropriate; when EV new >0, the image acquisition system outputs an additional brightened image; when EV new <0, the image acquisition system outputs an additional dimmed image. Optionally, the user can also adjust the EV value according to actual needs.

[0120] In another possible implementation, after determining the target gain value corresponding to the first image, the image processing device may send the target gain value corresponding to the first image to the image acquisition system. After receiving the target gain value corresponding to the first image, the image acquisition system may determine the exposure compensation amount required for assisting the image acquisition system in acquiring the second image based on the target gain value corresponding to the first image and the exposure compensation amount required for the image acquisition system to acquire the first image. Thereafter, the image acquisition system may intervene in the acquisition process of the image corresponding to the next environmental information based on the exposure compensation amount required for assisting the image acquisition system in acquiring the second image, and obtain the second image.

[0121] It can be seen from the above steps 201 to 204 that after acquiring the first image, the image processing device can determine which image areas need to be optimized for image quality, such as the area where a physical object or several physical objects are located, by judging whether the image quality of the area where each physical object in at least one physical object identified in the first image is located belongs to the set image quality range. In this way, one or more image areas in the first image that need image quality optimization can be intelligently selected, so that the image quality optimization is more targeted. Afterwards, the image processing device can calculate the gain value corresponding to each image area that needs to be optimized for image quality according to the total number of multiple pixels included in the image area and the grayscale values ​​of each of the multiple pixels. Then, the image processing device can calculate the target gain value corresponding to the first image according to the gain value corresponding to at least one image area that needs to be optimized for image quality included in the first image. In this way, the method can timely and accurately obtain parameters (such as target gain value or exposure compensation required for acquiring the next image (such as the second image)) used to assist the image acquisition system in optimizing the image quality of the image area to be optimized in the acquired image, and feed it back to the image acquisition system, so that the image acquisition system can accurately optimize the image area that needs to be optimized in a targeted manner, so that the image acquisition system can have the ability to optimize the image quality of the local image area, and then the image acquisition system can output images with better quality (such as images with better picture clarity, picture brightness, etc.), and can realize dynamic optimization of the image area, while improving the user's visual experience. In addition, the method does not require manual targeted processing of various scenes, so that the image optimization efficiency is high and the versatility is strong.

[0122] It should be noted that in the description of the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there may be three relationships, for example, A and / or B, which can represent: the situation where A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are an "or" relationship. "At least one of the following (individuals)" or its similar expression refers to any combination of these items, including any combination of single items (individuals) or plural items (individuals). For example, "at least one of A, B and C" includes A, B, C, AB, AC, BC or ABC. And, unless otherwise specified, the ordinal numbers such as "first", "second", and "third" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects. In addition, the terms "including", "comprising", "having" and their variations appearing in the present application all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0123] In addition, it should be noted that each step involved in the above embodiments can be performed by a corresponding device, or by a chip, processor, or chip system in the device, and the embodiments of the present application do not limit them. The above embodiments are only described by taking the corresponding device as an example.

[0124] It should be noted that in the above embodiments, some steps can be selected for implementation, and the order of the steps in the diagram can be adjusted for implementation, and this application does not limit this. It should be understood that executing some steps in the diagram, adjusting the order of the steps, or combining them for specific implementation all fall within the scope of protection of this application.

[0125] It is understandable that, in order to implement the functions in the above embodiments, the various devices involved in the above embodiments include hardware structures and / or software modules corresponding to the execution of the various functions. It should be easily appreciated by those skilled in the art that, in combination with the units and method steps of the various examples described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0126] It should be noted that the "steps" in the embodiments of the present application are only for illustration, and are a method of expression used to better understand the embodiments, and do not constitute a substantial limitation on the execution of the scheme of the present application. For example, the "steps" can also be understood as "features". In addition, the steps do not constitute any limitation on the execution order of the scheme of the present application. Any changes in the order of steps, step merging, or step splitting made on this basis that do not affect the implementation of the overall scheme, and the resulting new technical solutions are also within the scope of the disclosure of the present application.

[0127] Based on the same concept, the present application also provides an image processing device, which is suitable for Figure 1 In the system architecture shown in FIG. 1 , the image processing device may be as follows: Figure 1 The image processing device 200 shown in the figure may also be a functional element (such as a plug-in, component or chip, etc.) set in the image processing device 200, and the functional element has the function of implementing the image processing method. In one example, the image processing device 200 may be any device (such as a computing device or a control unit, etc.) with image processing capabilities in the vehicle, such as a VCU, a VDC system or an MDC, etc. in the vehicle. In another example, the image processing device 200 may be other devices located outside the vehicle (such as a vehicle server or a roadside device, etc.), or it may be a functional element with image processing functions set in other devices, and the functional element has the function of implementing the image processing method. Optionally, the image processing device may be used to implement the image processing method provided in the above embodiment, or the module (such as a chip) of the image processing device may be used to implement the image processing method provided in the above embodiment, so that the beneficial effects of the above embodiment can also be achieved. Exemplarily, taking the image processing device as a chip set in the image processing device 200 as an example, when the image processing device is a chip, the image processing device includes a transceiver and a processor, but does not include a memory. Among them, the transceiver exists in the form of an input and output interface, and the input and output interface is used for the chip to realize the transceiver of the image processing device 200. The input and output interface may include an input interface and / or an output interface, the input interface can realize the reception of the image processing device 200, and the output interface can be used to realize the transmission of the image processing device 200. The processor is used to read and execute corresponding computer programs or instructions so that the corresponding functions of the image processing device are realized. Optionally, when the chip realizes the corresponding functions of the image processing device in the above-mentioned embodiment, the input and output interface can realize the transceiver operation performed by the image processing device in the above-mentioned embodiment; the processor can realize other operations except the transceiver operation performed by the image processing device in the above-mentioned embodiment. For specific related detailed descriptions, please refer to the above Figure 2 The relevant description of the method embodiment shown will not be introduced in detail here.

[0128] See also Figure 4 The image processing device 400 includes an acquisition module 401 (or may be called a transceiver module, used to send and receive data) and a processing module 402.

[0129] Optionally, the acquisition module 401 may include a receiving module and / or a sending module. The receiving module may be used for the image processing device 400 to receive a signal (information or data, etc.); the sending module may be used for the image processing device 400 to send a signal (information or data, etc.). The sending module may send a signal (information or data, etc.) under the control of the processing module 402, and the receiving module may receive a signal (information or data, etc.) under the control of the processing module 402.

[0130] Optionally, the processing module 402 may include one or more of an object detection module, an object classification module, or an image quality optimization module. For a more detailed description of the object detection module, the object classification module, and the image quality optimization module, please refer to the above Figure 1 The relevant descriptions of the object detection module, object classification module and image quality optimization module in the illustrated system architecture will not be repeated here.

[0131] When the image processing device 400 is used to implement Figure 2 The functions of the image processing device in the method embodiment shown are: an acquisition module 401, which is used to acquire a first image. The first image is acquired by an image acquisition system located on a vehicle. A processing module 402 is used to determine the image quality of an area where at least one physical object contained in the first image is located. The processing module 402 is also used to determine a gain value of the area where the first physical object is located when the image quality of the area where the first physical object is located in the first image belongs to a set image quality range. The processing module 402 is also used to determine a target gain value corresponding to the first image based on the gain value of the area where the first physical object is located. The target gain value is used to determine the exposure compensation amount required for the image acquisition system to acquire the second image.

[0132] For a more detailed description of the acquisition module 401 and the processing module 402, please refer to the above Figure 2 The relevant descriptions about the image processing device in the illustrated method embodiment will not be repeated here one by one.

[0133] It should be understood that the acquisition module 401 in the embodiment of the present application can be implemented by a transceiver or a transceiver-related circuit component, and the processing module 402 can be implemented by a processor or a processor-related circuit component.

[0134] For example, the processing module 402 includes an object detection module, an object classification module, and an image quality optimization module. Figure 2When the related functions of the image processing device in the method embodiment shown are implemented, the object detection module is used to implement the function of the processing module 402 in determining the image quality of the area where at least one physical object contained in the first image is located; the object classification module is used to implement the function of the processing module 402 in determining which physical objects need to have their image quality optimized and which physical objects do not need to have their image quality optimized; the image quality optimization module is used to implement the function of the processing module 402 in determining the gain value of the area where the first physical object is located when the image quality of the area where the first physical object is located in the first image belongs to the set image quality range (that is, the image quality of the area where the first physical object is located needs to be optimized), and can also implement the function of the processing module 402 in determining the target gain value corresponding to the first image based on the gain value of the area where the first physical object is located.

[0135] It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated in a processing unit, or may exist physically separately, or two or more units may be integrated in one unit. For example, the acquisition module and the processing module may be integrated in one module, or the acquisition module and the processing module may be the same module. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0136] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, or a server, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0137] Based on the same concept, the present application also provides an image processing device, which is suitable for Figure 1 The image processing device is used to implement the above Figure 2 The technical solution of the image processing device in the method embodiment shown in the figure can also achieve the above Figure 2The beneficial effects of the image processing device in the method embodiment shown in FIG. Figure 5 , the image processing device 500 includes: a transceiver 501 and a processor 502. Optionally, the image processing device 500 also includes a memory 503. The transceiver 501, the processor 502 and the memory 503 are interconnected. When the image processing device 500 is used to implement the technical solution involved in the image processing device provided in the above embodiment, the transceiver 501 can be used to implement the function of the above acquisition module 401, and the processor 502 is used to implement the function of the above processing module 402.

[0138] Optionally, the transceiver 501, the processor 502 and the memory 503 are interconnected via a bus 504. The bus 504 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0139] The transceiver 501 is used to receive and send data. For example, when the image processing device 500 is an image processing device (such as MDC) having the functions required to implement the method, the transceiver 501 can be used to communicate with an image acquisition system (such as a camera system) or other devices (such as other vehicle-mounted devices or other communication devices, etc.). In one example, the transceiver can be a transceiver device with integrated data transceiver functions. In another example, the transceiver can also be composed of a transmitter and a receiver, wherein the transmitter is used to send data and the receiver is used to receive data.

[0140] Optionally, the transceiver 501 may include a transmitter and / or a receiver. The transmitter is used to send signals, messages, information, or data, etc. The receiver is used to receive signals, messages, information, or data, etc. Exemplarily, the transmitter sends signals, messages, information, or data, etc. under the control of the processor 502. The receiver receives signals, messages, information, or data, etc. under the control of the processor 502.

[0141] The function of the processor 502 can refer to the above Figure 2The description of the corresponding functions involved in the image processing device in the method embodiment shown is not repeated here. Among them, the processor 502 can be a central processing unit (CPU), a network processor (NP) or a combination of CPU and NP, etc. The processor 502 can further include a hardware chip. The above-mentioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above-mentioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. When the processor 502 implements the above-mentioned functions, it can be implemented by hardware, and of course, it can also execute the corresponding software implementation through hardware.

[0142] The memory 503 may include a volatile memory, such as a random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0143] The memory 503 stores executable program codes, and the processor 502 executes the executable program codes to respectively implement the functions of the aforementioned image processing device (such as the acquisition module, the processing module (including one or more of the object detection module, the object classification module, or the image quality optimization module)), thereby implementing the image processing method provided in the embodiment of the present application. That is, the memory 503 stores computer program instructions for executing the image processing method.

[0144] Alternatively, the memory 503 stores executable codes, and the processor 502 executes the executable codes to implement the functions of the aforementioned image processing device (such as an acquisition module, a processing module (including one or more of an object detection module, an object classification module, or an image quality optimization module)), thereby implementing the image processing method provided in the embodiment of the present application. That is, the memory 503 stores computer program instructions for the image processing device to execute the image processing method provided in the embodiment of the present application.

[0145] Based on the same concept, the embodiment of the present application also provides an image processing system, which includes an image acquisition system and an image processing device, wherein the image processing device can be used to implement the above Figure 2 The technical solution involved in the image processing device in the method embodiment shown, the image acquisition system can be used to obtain images corresponding to the surrounding environment information of the vehicle. Exemplarily, the number of image acquisition systems included in the image processing system can be one or more, and the number of image processing devices can be one or more, which is not limited in the embodiment of the present application. The image processing system can be deployed on a vehicle. Accordingly, the embodiment of the present application also provides a vehicle including the above-mentioned image processing system.

[0146] Based on the same concept, an embodiment of the present application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are run on a computer, the computer executes the image processing method provided in the above embodiment.

[0147] Based on the same concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program or instruction. When the computer program or instruction is executed by a computer, the computer executes the image processing method provided in the above embodiment.

[0148] The storage medium may be any available medium that can be accessed by a computer. For example, but not limited to, a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer.

[0149] Based on the same concept, an embodiment of the present application further provides a chip, which is coupled to a memory and is used to read a computer program stored in the memory to implement the image processing method provided in the above embodiment.

[0150] Based on the same concept, the embodiment of the present application also provides a chip system, which includes a processor for supporting a computer device to implement the functions involved in the image processing device in the above embodiment. In one design, the chip system also includes a memory, which is used to store the necessary programs and data for the computer device. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0151] In the method provided in the embodiment of the present application, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state drive (SSD)).

[0152] The steps of the method described in the embodiments of the present application can be directly embedded in the hardware, the software unit executed by the processor, or the combination of the two. The software unit can be stored in RAM, ROM, EEPROM, register, hard disk, removable disk, CD-ROM or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC.

[0153] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0155] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image processing method, characterized in that: include: Acquire a first image, where the first image is acquired by an image acquisition system located on the vehicle; determining an image quality of a region where at least one physical object contained in the first image is located; When the image quality of the area where the first physical object is located in the first image belongs to the set image quality range, determining a gain value of the area where the first physical object is located; determining a target gain value corresponding to the first image according to a gain value of a region where the first physical object is located; The target gain value is used to determine the exposure compensation amount required for the image acquisition system to acquire the second image.

2. The method according to claim 1, characterized in that The determining the image quality of a region where at least one physical object contained in the first image is located comprises: Object detection is performed on the first image to obtain a confidence level of each physical object in the at least one physical object, wherein the confidence level of each physical object indicates an image quality of an area where the physical object is located.

3. The method according to claim 1 or 2, characterized in that The set image quality range is a numerical range determined by a first threshold and a second threshold, and the second threshold is greater than the first threshold.

4. The method according to any one of claims 1 to 3, characterized in that: The determining a gain value of the area where the first physical object is located includes: Determine the total number of pixels in the area where the first physical object is located; Determine an overall grayscale value corresponding to the area where the first physical object is located according to the total number of pixels in the area where the first physical object is located and the grayscale values ​​of a plurality of pixels contained in the area where the first physical object is located; A gain value of the region where the first physical object is located is determined according to an overall grayscale value corresponding to the region where the first physical object is located.

5. The method according to claim 4, characterized in that The determining the total number of pixels in the area where the first physical object is located includes: Performing object detection on the first image to obtain a plurality of key position coordinates of an area where the first physical object is located; The total number of pixels in the area where the first physical object is located is determined according to the multiple key position coordinates.

6. The method according to any one of claims 1 to 5, characterized in that: The determining, according to the gain value of the area where the first physical object is located, a target gain value corresponding to the first image includes: When the number of the first physical object is one, the gain value of the area where the first physical object is located is used as the target gain value; or When there are multiple first physical objects, an average value or a weighted average value of gain values ​​of regions where the multiple first physical objects are located is used as the target gain value.

7. The method according to claim 6, characterized in that The average value of the gain values ​​of the area where the multiple first physical objects are located is determined based on the gain values ​​of the area where the multiple first physical objects are located and the number of the multiple first physical objects, and the weighted average value of the gain values ​​of the area where the multiple first physical objects are located is determined based on the gain values ​​of the area where the multiple first physical objects are located and the total number of pixels in the area where the multiple first physical objects are located.

8. The method according to any one of claims 1 to 7, characterized in that: The exposure compensation amount required by the image acquisition system to acquire the second image is determined according to the target gain value and the exposure compensation amount required by the image acquisition system to acquire the first image.

9. An image processing device, characterized in that: include: An acquisition module, used for acquiring a first image, wherein the first image is acquired by an image acquisition system located on the vehicle; A processing module, used to determine the image quality of an area where at least one physical object contained in the first image is located; when the image quality of the area where the first physical object is located in the first image belongs to a set image quality range, determine the gain value of the area where the first physical object is located; according to the gain value of the area where the first physical object is located, determine the target gain value corresponding to the first image; wherein the target gain value is used to determine the exposure compensation amount required by the image acquisition system to obtain the second image.

10. The device according to claim 9, characterized in that When determining the image quality of the area where at least one physical object contained in the first image is located, the processing module is specifically used to: Object detection is performed on the first image to obtain a confidence level of each physical object in the at least one physical object, wherein the confidence level of each physical object indicates an image quality of an area where the physical object is located.

11. The device according to claim 9 or 10, characterized in that The set image quality range is a numerical range determined by a first threshold and a second threshold, and the second threshold is greater than the first threshold.

12. The device according to any one of claims 9 to 11, characterized in that: When determining the gain value of the area where the first physical object is located, the processing module is specifically used to: Determine the total number of pixels in the area where the first physical object is located; Determine an overall grayscale value corresponding to the area where the first physical object is located according to the total number of pixels in the area where the first physical object is located and the grayscale values ​​of a plurality of pixels contained in the area where the first physical object is located; A gain value of the region where the first physical object is located is determined according to an overall grayscale value corresponding to the region where the first physical object is located.

13. The device according to claim 12, characterized in that When determining the total number of pixels in the area where the first physical object is located, the processing module is specifically used to: Performing object detection on the first image to obtain a plurality of key position coordinates of an area where the first physical object is located; The total number of pixels in the area where the first physical object is located is determined according to the multiple key position coordinates.

14. The device according to any one of claims 9 to 13, characterized in that: When the processing module determines the target gain value corresponding to the first image according to the gain value of the area where the first physical object is located, it is specifically used to: When the number of the first physical object is one, the gain value of the area where the first physical object is located is used as the target gain value; or When there are multiple first physical objects, an average value or a weighted average value of gain values ​​of regions where the multiple first physical objects are located is used as the target gain value.

15. The device according to claim 14, characterized in that The average value of the gain values ​​of the area where the multiple first physical objects are located is determined based on the gain values ​​of the area where the multiple first physical objects are located and the number of the multiple first physical objects, and the weighted average value of the gain values ​​of the area where the multiple first physical objects are located is determined based on the gain values ​​of the area where the multiple first physical objects are located and the total number of pixels in the area where the multiple first physical objects are located.

16. The device according to any one of claims 9 to 15, characterized in that: The exposure compensation amount required by the image acquisition system to acquire the second image is determined according to the target gain value and the exposure compensation amount required by the image acquisition system to acquire the first image.

17. An image processing device, characterized in that: including a transceiver, a memory and a processor; The transceiver is used to receive and send data; The memory is used to store computer program instructions and data; The processor is used to execute and call the computer program instructions and data in the memory so that the image processing device performs the method as claimed in any one of claims 1-8.

18. An image processing system, characterized in that: It comprises an image acquisition system, an image processing device as described in any one of claims 9 to 16 or an image processing device as described in claim 17, and the image acquisition system is used to obtain images corresponding to the surrounding environment information of the vehicle.

19. A vehicle, characterized in that: Includes the image processing system as claimed in claim 18.

20. A chip, characterized in that: The chip is coupled to a memory, and the chip reads a computer program stored in the memory to execute the method according to any one of claims 1 to 8.

Images