Parameter adjusting method, device and system

Abstract

The invention discloses an image signal processor ISP parameter adjustment method, device and system, relates to the technical field of artificial intelligence, and can automatically adjust an ISP parameter set according to an image perception result, and the whole process does not depend on a large amount of annotation data. In the scheme, a target sensor and a reference sensor can shoot the same scene, a first original image shot by the target sensor is input into a proxy neural network, and processing of a first ISP parameter set is carried out through the network to obtain a first processed image; and processing a second original image shot by the reference sensor through a second ISP parameter set to obtain a second processed image. Inputting the first processed image into a first sensing neural network for image recognition to obtain a first sensing result; and inputting the second processed image into a pre-trained second perception neural network for image recognition to obtain a second perception result. And then according to the two sensing results, an ISP parameter set in the proxy neural network is adjusted.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image signal processor (ISP) parameter adjustment method, device and system. Background Art

[0002] With the continuous development of image acquisition and processing technology, cameras are widely used in various fields, such as autonomous driving (AD), smart security, portable mobile terminals (such as mobile phones, tablets, etc.), cameras, intelligent robots, drones, machine vision, etc.

[0003] At present, the quality of the image output by the camera depends not only on the camera module and process, but also on the image signal processor technology. Specifically, the image formation process of the camera is generally to shoot the scene through the lens first, and the light reflected by the scene is projected onto the image sensor through the lens. Then, the image sensor undergoes analog-to-digital conversion to generate a digital image signal, i.e., raw data. The raw data is processed by an image signal processor (ISP) with different parameter configurations, such as noise removal, automatic white balance correction (AWB), color correction (CC), gamma correction, color and contrast enhancement, automatic exposure control, etc., and finally, images that can be processed by various algorithms or displayed on the screen can be generated, such as standard three primary colors (red, green, blue, RGB) format images, portable network graphics (JPG) format images, etc.

[0004] Therefore, the relevant parameters of ISP will directly affect the imaging quality of the camera. However, the traditional way to adjust ISP parameters is mainly for technicians to manually adjust according to subjective visual evaluation, which not only consumes huge time and manpower costs, but also causes the debugging effect of ISP parameters to be too dependent on the personal experience of technicians, and the overall debugging efficiency is relatively low. Summary of the invention

[0005] The present application provides an image signal processor ISP parameter adjustment method, device and system, which can automatically adjust ISP parameters according to the image perception results output by a first perception neural network and the image perception results output by a pre-trained second perception neural network, and the entire process does not rely on a large amount of labeled data.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, a method for adjusting ISP parameters of an image signal processor is provided, the method comprising: acquiring a first original image, the first original image being obtained by photographing a first scene with a target sensor; inputting the first original image into a pre-trained proxy neural network to obtain a first processed image, wherein the proxy neural network is used to process the original image photographed by the target sensor according to a first ISP parameter set, and output a corresponding processed image; acquiring a second processed image, the second processed image being obtained by processing a second original image according to a second ISP parameter set, the second original image being obtained by photographing the first scene with a reference sensor, and the second ISP parameter set being adapted to the reference sensor; inputting the first processed image into a first perception neural network to obtain a first perception result; inputting the second processed image into a pre-trained second perception neural network to obtain a second perception result; adjusting the first ISP parameter set in the proxy neural network according to the first perception result and the second perception result to obtain a third ISP parameter set, and the third ISP parameter set being adapted to the target sensor.

[0008] Optionally, the reference sensor is an optimized sensor (i.e., configured with parameters that can guarantee quality requirements), and the second original image captured by the reference sensor is processed by a second ISP parameter set adapted to the reference sensor, and a second processed image that meets the quality requirements can be output. The second processed image can be a standard red, green, and blue (RGB) image. The target sensor is an unoptimized sensor, and the first original image captured by the reference sensor is processed by a first ISP parameter set that is not adapted to the reference sensor, and the output first processed image cannot meet the quality requirements.

[0009] Optionally, the pre-trained second perception neural network can be adapted to the reference sensor, which can accurately complete corresponding perception tasks, such as detection tasks, segmentation tasks, classification tasks, etc., for the second processed image output according to the second ISP parameter set.

[0010] Optionally, the first perception neural network may not be adapted to the target sensor, and may not achieve the expected processing effect when performing a corresponding perception task for the first processed image output according to the first ISP parameter set.

[0011] The solution provided in the first aspect above directly uses the optimized reference sensor and the second processed image output by the second ISP parameter set to perceive in the trained second perception neural network. The perception result obtained can be used as pseudo-annotation data of the first perception neural network. Therefore, for the original image collected by the unoptimized target sensor, the original image can be first processed by the proxy neural network simulation to output the simulated first processed image, and then the simulated first processed image is input into the first perception neural network for perception. The perception result obtained can be evaluated with the aforementioned pseudo-annotation data, and the first ISP parameter set simulated by the proxy neural network is optimized by the evaluation result. Thus, a semi-supervised learning ISP parameter optimization scheme is realized. The method of the present application directly uses the perception result of the trained second perception neural network as pseudo-annotation data, and combines the perception result of the first perception neural network to automatically optimize the first ISP parameter set simulated by the proxy neural network, thereby improving the optimization efficiency of the ISP parameter set.

[0012] In a possible implementation, the ISP parameter adjustment method of the present application further includes: training the first perception neural network according to the first perception result and the second perception result. In this way, the present application can optimize not only the ISP parameter set but also the first perception neural network through the perception result of the trained second perception neural network and the perception result of the first perception neural network.

[0013] In a possible implementation, the ISP parameter adjustment method of the present application further includes: switching between a first optimization operation and a second optimization operation, wherein the first optimization operation includes adjusting a first ISP parameter set in a proxy neural network according to a first perception result and a second perception result, and the second optimization operation includes training a first perception neural network according to the first perception result and the second perception result. In this way, the present application can select one between the ISP parameter set optimization and the first perception neural network optimization by switching the optimization operation, thereby avoiding the problem of poor optimization effect caused by optimizing two variables at the same time.

[0014] In a possible implementation, the switching between the first optimization operation and the second optimization operation may include: switching between the first optimization operation and the second optimization operation every preset number of iterations. In this way, the present application can optimize back and forth between the ISP parameter set optimization and the first perceptual neural network optimization multiple times, which not only realizes the joint optimization of the ISP parameter set optimization and the first perceptual neural network optimization, but also ensures the optimization effect of the ISP parameter set optimization and the first perceptual neural network optimization.

[0015] In a possible implementation, the switching between the first optimization operation and the second optimization operation may include: performing the second optimization operation during the first X iterations, and switching to performing the first optimization operation after the Xth iteration, where X is a positive integer. In this way, the present application may optimize the first perceptual neural network to a certain extent through multiple iterations before optimizing the first ISP parameter set, so as to avoid the perception capability of the unoptimized target perceptual neural network being too low, thereby affecting the optimization of the ISP parameter set.

[0016] In a possible implementation, the switching between the first optimization operation and the second optimization operation may include: performing the first optimization operation during the first Y iterations, and switching to performing the second optimization operation after the Y iterations, where Y is a positive integer. In this way, the present application may optimize the first ISP parameter set to a certain degree through multiple iterations before optimizing the first perceptual neural network, so as to avoid excessive deviation of the unoptimized first ISP parameter set, which may affect the optimization of the first perceptual neural network.

[0017] In a possible implementation, the ISP parameter adjustment method of the present application also includes: updating the network parameters of the second perceptual neural network according to the network parameters of the trained first perceptual neural network. In this way, the present application can also use the optimized first perceptual neural network to fine-tune the network parameters of the second perceptual neural network so that the second perceptual neural network can also adapt to the target sensor. That is, the second perceptual neural network can also perceive the processed images output by the target sensor and the optimized third ISP parameter set in accordance with customer requirements, and obtain accurate perception results. Thereby further optimizing the perception ability of the pre-trained second perceptual neural network.

[0018] In a possible implementation, the second perception result is used as the annotation result of the first perception result to evaluate the perception ability of the first perception neural network. At this time, the above-mentioned adjustment of the first ISP parameter set in the proxy neural network based on the first perception result and the second perception result includes: comparing the first perception result with the second perception result to obtain a calculation loss, the calculation loss indicates the error between the first perception result and the second perception result; adjusting the first ISP parameter set in the proxy neural network according to the calculation loss. In this way, the perception result of the trained second perception neural network can be used to evaluate the quality of the perception result of the first perception neural network, so as to feed back the quality of the perception result to the proxy neural network to complete the optimization of the first ISP parameter set.

[0019] In a possible implementation, the training of the first perceptual neural network according to the first perceptual result and the second perceptual result includes: comparing the first perceptual result with the second perceptual result to obtain a calculation loss, the calculation loss indicating the error between the first perceptual result and the second perceptual result; and training the network parameters in the first perceptual neural network using a back propagation algorithm according to the calculation loss. In this way, the perceptual result of the trained second perceptual neural network can be used to evaluate the quality of the perceptual result of the first perceptual neural network, so that the quality of the perceptual result is fed back to the first perceptual neural network to complete the tuning of the network parameters (network weights) in the first perceptual neural network.

[0020] In a possible implementation, the target sensor and the reference sensor are arranged adjacent to each other. In this way, it can be ensured that the two sensors capture the same scene at the same angle. Optionally, there may be a slight difference in viewing angle between the two sensors, so the target sensor and the reference sensor may be image-aligned to eliminate the difference in viewing angle.

[0021] In a possible implementation, the target sensor and the reference sensor are disposed on a movable device, and the method is executed during the movement of the movable device. In this way, the ISP parameter adjustment can be completed online and in real time during the image acquisition process.

[0022] In a possible implementation, the ISP parameter adjustment method of the present application further includes: configuring the third ISP parameter set as the parameters of the first ISP, and the first ISP is used to process the third original image obtained by the target sensor shooting the second scene according to the third ISP parameter set. In this way, after obtaining the optimized third ISP parameter set, it can be adapted to the target sensor, so that the original image captured by the target sensor in the actual application scenario can be directly processed by the optimized third ISP parameter set, thereby ensuring that in the actual application scenario, the RGB image that meets the quality requirements can also be obtained through the shooting of the target sensor.

[0023] In a second aspect, an image signal processor ISP parameter adjustment device is provided, the device comprising: a first acquisition module, a processing module, a second acquisition module, a first perception module, a second perception module and an adjustment module.

[0024] A first acquisition module is used to acquire a first original image, where the first original image is obtained by capturing a first scene by a target sensor;

[0025] a processing module, configured to input the first original image into a pre-trained proxy neural network to obtain a first processed image, wherein the proxy neural network is configured to process the original image captured by the target sensor according to the first ISP parameter set and output a corresponding processed image;

[0026] A second acquisition module is used to acquire a second processed image, where the second processed image is obtained by processing a second original image according to a second ISP parameter set, where the second original image is obtained by photographing the first scene with a reference sensor, and the second ISP parameter set is adapted to the reference sensor;

[0027] A first perception module, used for inputting the first processed image into a first perception neural network to obtain a first perception result;

[0028] A second perception module, used for inputting the second processed image into a pre-trained second perception neural network to obtain a second perception result;

[0029] The adjustment module is used to adjust the first ISP parameter set in the proxy neural network according to the first perception result and the second perception result to obtain a third ISP parameter set, and the third ISP parameter set is adapted to the target sensor.

[0030] In a possible implementation, the adjustment module is further used to: train the first perception neural network according to the first perception result and the second perception result.

[0031] In one possible implementation, the adjustment module is also used to switch between a first optimization operation and a second optimization operation, wherein the first optimization operation includes adjusting a first ISP parameter set in the proxy neural network according to the first perception result and the second perception result, and the second optimization operation includes training the first perception neural network according to the first perception result and the second perception result.

[0032] In a possible implementation, the adjustment module is further configured to: switch between the first optimization operation and the second optimization operation every preset number of iterations.

[0033] In a possible implementation, the adjustment module is further configured to: execute the second optimization operation in the first X iterations, and switch to executing the first optimization operation after the Xth iteration, where X is a positive integer.

[0034] In a possible implementation, the adjustment module is further configured to: perform a first optimization operation during the first Y iterations, and switch to performing a second optimization operation after the Y iterations, where Y is a positive integer.

[0035] In a possible implementation, the ISP parameter adjustment device of the present application further includes: an updating module, configured to update the network parameters of the second perceptual neural network according to the network parameters of the trained first perceptual neural network.

[0036] In one possible implementation, the second perception result can be used as a labeling result of the first perception result. The adjustment module is also used to: compare the first perception result with the second perception result to obtain a calculation loss, where the calculation loss indicates an error between the first perception result and the second perception result; and adjust the first ISP parameter set in the proxy neural network according to the calculation loss.

[0037] In a possible implementation manner, the target sensor is disposed adjacent to the reference sensor.

[0038] In a possible implementation, the target sensor and the reference sensor are arranged on a movable device, and the acquisition module is further used to: during the movement of the movable device, acquire a first original image captured by the target sensor, and acquire a second original image captured by the reference sensor. The proxy module is further used to: during the movement of the movable device, input the first original image into the pre-trained proxy neural network. The first perception module is further used to: during the movement of the movable device, input the first processed image into the first perception neural network to obtain a first perception result. The second perception module is further used to: during the movement of the movable device, input the second processed image into the pre-trained second perception neural network to obtain a second perception result; the adjustment module is further used to: during the movement of the movable device, according to the first perception result and the second perception result, adjust the first ISP parameter set in the proxy neural network to obtain a third ISP parameter set, and the third ISP parameter set is adapted to the target sensor.

[0039] In a possible implementation, the ISP parameter adjustment device of the present application also includes: a configuration module, used to configure the third ISP parameter set as the parameters of the first ISP, and the first ISP is used to process the third original image obtained by the target sensor shooting the second scene according to the third ISP parameter set.

[0040] In a possible implementation, the first acquisition module is further used to acquire a third original image obtained by the target sensor photographing the second scene; and the processing module is used to process the third original image according to a third ISP parameter set to obtain a third processed image.

[0041] In a third aspect, an image signal processor ISP parameter adjustment system is provided, the system comprising: a target sensor, a reference sensor and an ISP parameter adjustment device, wherein:

[0042] A target sensor, used for photographing a first scene to obtain a first original image, and sending the first original image to the ISP parameter adjustment device;

[0043] An ISP parameter adjustment device, configured to, upon receiving a first original image, input the first original image into a proxy neural network to obtain a first processed image, and input the first processed image into a first perception neural network to obtain a first perception result, wherein the proxy neural network is configured to process the original image captured by the target sensor according to the first ISP parameter set, and output a corresponding processed image;

[0044] A reference sensor is used to capture the first scene to obtain a second original image, process the second original image according to a second ISP parameter set to obtain a second processed image, and send the second processed image to the ISP parameter adjustment device, wherein the second ISP parameter set is adapted to the reference sensor;

[0045] The ISP parameter adjustment device is also used to input the second processed image into the second perception neural network to obtain a second perception result when the second processed image is received, and adjust the first ISP parameter set in the proxy neural network according to the first perception result and the second perception result to obtain a third ISP parameter set, and the third ISP parameter set is adapted to the target sensor.

[0046] In a possible implementation, a second ISP is integrated on the reference sensor, wherein the second ISP is configured with a second ISP parameter set adapted to the reference sensor. In this case, the reference sensor is used to capture the first scene to obtain a second original image, the second original image is processed by the second ISP to obtain a second processed image, and the second processed image is sent to the ISP parameter adjustment device.

[0047] In a possible implementation, the second ISP may also be independent of the reference sensor. In this case, the ISP parameter adjustment system further includes the second ISP, wherein:

[0048] A reference sensor is used to capture the first scene to obtain a second original image, and send the second original image to a second ISP, wherein the second ISP is configured with a second ISP parameter set adapted to the reference sensor;

[0049] The second ISP is used to process the second original image according to the second ISP parameter set to obtain a second processed image, and send the second processed image to the ISP parameter adjustment device.

[0050] In a possible implementation, the third ISP parameter set may be configured as parameters of the first ISP. In this case, the ISP parameter adjustment system further includes the first ISP, wherein:

[0051] The target sensor is further used to capture the second scene to obtain a third original image, and send the third original image to the first ISP;

[0052] The first ISP is used to process the third original image according to the third ISP parameter set to obtain a third processed image.

[0053] In a possible implementation, the ISP parameter adjustment device is also used to train the first perception neural network based on the first perception result and the second perception result.

[0054] In one possible implementation, the ISP parameter adjustment device is also used to switch between a first optimization operation and a second optimization operation, wherein the first optimization operation includes adjusting a first ISP parameter set in the proxy neural network based on the first perception result and the second perception result, and the second optimization operation includes training the first perception neural network based on the first perception result and the second perception result.

[0055] In a fourth aspect, an ISP parameter adjustment device is provided, which has the function of implementing the above-mentioned first aspect and any possible design method thereof. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes at least one module or unit corresponding to the above-mentioned function. The ISP parameter adjustment device can be an on-board computing platform, which enables the vehicle to execute the ISP parameter adjustment method in the above-mentioned first aspect and any possible design method thereof.

[0056] In a fifth aspect, a chip system is provided, which includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive computer instructions; when the processor executes the computer instructions, the chip system executes the ISP parameter adjustment method in the above-mentioned first aspect and any possible design method thereof.

[0057] In a sixth aspect, a computer-readable storage medium is provided, comprising computer instructions. When the computer instructions are executed on a computer, the computer executes the ISP parameter adjustment method in the first aspect and any possible design thereof.

[0058] In a seventh aspect, a mobile device is provided, the mobile device comprising the ISP parameter adjustment device in the second aspect or any implementation thereof, or the ISP parameter adjustment system in the third aspect or any implementation thereof, or the ISP parameter adjustment device in the fourth aspect or any implementation thereof. The mobile device may be a vehicle.

[0059] It can be understood that the beneficial effects that can be achieved by the above-mentioned device of the second aspect, the system of the third aspect, the vehicle-mounted computing platform of the fourth aspect, the chip system of the fifth aspect, the computer storage medium of the sixth aspect, and the vehicle of the seventh aspect can be referred to the beneficial effects of the first aspect and any possible implementation thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of the structure of a vehicle provided in an embodiment of the present application;

[0061] Figure 2 A system schematic diagram of a vehicle provided in an embodiment of the present application;

[0062] Figure 3 A system schematic diagram of another vehicle provided in an embodiment of the present application;

[0063] Figure 4 A system schematic diagram of another vehicle provided in an embodiment of the present application;

[0064] Figure 5 A flowchart of an ISP parameter adjustment method provided in an embodiment of the present application;

[0065] Figure 6 A schematic diagram of a training process of a proxy neural network provided in an embodiment of the present application;

[0066] Figure 7 A flowchart of another ISP parameter adjustment method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In the description of this application, unless otherwise specified, " / " indicates that the objects associated with each other are in an "or" relationship, for example, A / B can represent A or B; "and / or" in this application is merely a description of the association relationship between associated objects, indicating that three relationships may exist, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.

[0068] In the description of this application, unless otherwise specified, "plurality" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, a and b and c, where a, b, and c can be single or multiple.

[0069] In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish between items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not necessarily limit the difference.

[0070] In the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0071] The features, structures or characteristics in this application may be combined in one or more embodiments in any suitable manner. In various embodiments of this application, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0072] Some optional features in the embodiments of the present application can be implemented independently in some scenarios without relying on other features to solve corresponding technical problems and achieve corresponding effects. They can also be combined with other features according to needs in some scenarios.

[0073] In this application, unless otherwise specified, the same or similar parts between the various embodiments can refer to each other. In the various embodiments of this application, if there is no special description and logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships. The implementation methods of this application do not constitute a limitation on the scope of protection of this application.

[0074] In addition, the business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0075] The embodiments of the present application provide an ISP parameter adjustment method, device and system, which can be applied to autonomous driving, intelligent security, portable mobile terminals (such as mobile phones, tablets, etc.), cameras, intelligent robots, drones, machine vision or other fields that require image acquisition, and the present application does not specifically limit this. The following will take the field of autonomous driving as an example to specifically describe the parameter adjustment method, device and system provided in the embodiments of the present application.

[0076] The field of autonomous driving can include intelligent vehicles, unmanned vehicles, etc., which usually rely on computer vision (CV), radar, cameras and global positioning systems to enable motor vehicles to achieve autonomous driving without the need for human active operation. Among them, computer vision is an integral part of various intelligent / autonomous systems. In essence, it uses various imaging systems to replace human visual organs to obtain input information, and then computers replace human brains to complete the processing and interpretation of these input information. Figuratively speaking, it is to install eyes (cameras) and brains (algorithms) on computers to replace human eyes to identify, track and measure targets, so that computers can perceive the environment. Therefore, the ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

[0077] At present, in the field of autonomous driving, it is often necessary to use a vehicle-mounted camera (imaging device) to obtain the environmental image around the vehicle, and computer vision is used to perceive or analyze the environmental image, so as to assist in the realization of various autonomous driving tasks (such as planning an autonomous driving route, controlling the lateral or longitudinal movement of the vehicle, etc.). For vehicle-mounted cameras of different specifications, the image imaging process first collects optical information of the vehicle's surrounding environment through a lens, and forms a digital signal on an image sensor such as a complementary metal oxide semiconductor (CMOS) to generate unprocessed raw data. The raw data of these different vehicle-mounted cameras are processed by an image signal processor (ISP) with different parameter configurations to generate various algorithms (such as computer vision) or RGB images for screen display.

[0078] Therefore, when computer vision perceives the RGB images output by the vehicle-mounted camera, the accuracy of the computer vision perception results depends largely on the quality of the RGB images, which is usually affected by multiple factors such as the lighting conditions of the external environment, the optical properties of the camera, the photoelectric conversion characteristics of sensors such as CMOS, and the processing algorithm and processing capabilities of the ISP.

[0079] In order to make the vehicle camera output high-quality image data under different lighting conditions, the relevant parameters of the camera and ISP are usually adjusted according to the above factors to complete the image quality debugging of computer vision. However, the entire process of image quality optimization currently faces the following five problems:

[0080] First, it is highly dependent on the personal experience of the optimization technicians. Currently, there are a large number of camera and ISP parameters related to image quality, and there is a certain correlation between many parameters, which will affect each other. Therefore, different optimization technicians may choose different parameters and different debugging methods when facing the same image problem, resulting in different image effects.

[0081] Second, the image quality standards are not unified and cannot be measured. Current image quality optimization is often debugged by comparing the standard image effects of the company's own or provided by the customer. When evaluating the final image quality, image experts will make subjective judgments based on the contrast, color reproduction, clarity, transparency, brightness and similarity with the standard image of the optimized image. If the expert believes that the image quality meets the standard, the optimization will be stopped. If the expert believes that the image quality does not meet the standard, the optimization will continue to be targeted. However, different customers and different experts often have different requirements for image quality, and subjective judgments cannot objectively measure the quality of the image, resulting in high communication costs with customers during the image optimization process and it is very easy to do repeated work.

[0082] Third, the robustness is not high. In order to improve the efficiency of image optimization, the current image optimization team often divides the image optimization work into modules such as brightness, color, and noise according to function, and assigns different optimization technicians to be responsible for each module. Due to the mutual influence between different functional modules, solving a certain image quality problem will cause other problems, resulting in increased time costs.

[0083] Fourth, the manpower investment is huge. At present, the entire image optimization process has to go through the stages of camera selection and calibration, laboratory verification and debugging, multiple rounds of external environment debugging, and customer-side actual vehicle testing. This not only leads to huge manpower investment, but also a very long time span for the entire image optimization process.

[0084] Fifth, debugging and verification are complicated, and the upper limit of image quality optimization is low. Since there are many adjustable parameters and different effects will be produced under different conditions, repeated and tedious effect verification is required. However, the cognitive level of optimization technicians is limited, which leads to a limited upper limit of image quality optimization.

[0085] In order to solve the above-mentioned problems of manual optimization, an automated image optimization technology based on a neural network model has been developed in the past two years. However, although the current automated image optimization technology can significantly shorten the time for adjusting ISP parameters, it requires a large amount of accurately labeled data (annotated data) for training the neural network model, and the accurate labeling of data itself requires a lot of manpower and time costs. This means that the investment cost has not been significantly reduced, and the final image optimization effect will be affected by the lack of training data quantity, quality, labeling accuracy, etc.

[0086] Based on the above problems, the embodiment of the present application proposes an ISP parameter adjustment method and device, which can automatically adjust the ISP parameters according to the image perception results output by the perception neural network, and the whole process does not rely on a large amount of labeled data. It not only gets rid of the dependence on manual optimization experience, does not require the participation of professional optimization personnel, but also gets rid of the need for a large amount of labeled data, greatly saving labor costs and time costs.

[0087] The ISP parameter adjustment device provided in the present application can be set in a vehicle, can also be set in other equipment, and can also exist as an independent device. Figure 1 , an ISP parameter adjustment device is set in a vehicle as an example for explanation.

[0088] like Figure 1 As shown, the vehicle 100 may be a vehicle with a wireless communication function, and the wireless communication function may be provided in an onboard terminal, an onboard module, an onboard unit, a chip (system) or other parts or components of the vehicle 100. Optionally, the vehicle 100 may be in an automatic driving state or an assisted driving state, that is, the vehicle 100 is fully autonomously driven without the need for driver control or with only a small amount of driver control.

[0089] In the embodiment of the present application, the vehicle 100 may also include a group of optimized cameras 11 and ISP 12, and a group of unoptimized cameras 13 and ISP 14. Among them, the ISP parameter set in ISP 12 has been debugged and optimized, and ISP 12 is adapted to camera 11, so that camera 11 can output RGB images that meet quality requirements. The ISP parameter set in ISP 14 is still to be debugged and optimized. The ISP parameter adjustment method provided in the embodiment of the present application can optimize the ISP parameter set in ISP 14 to adapt to camera 13, so that camera 13 can eventually output RGB images that meet quality requirements. The above-mentioned group of optimized cameras 11 and ISP 12, and a group of unoptimized cameras 13 and ISP 14 are set on the vehicle 100, so as to realize optimized image acquisition and unoptimized image acquisition on the vehicle 100, and at the same time, based on the optimized image and unoptimized image acquired in real time, the ISP parameter adjustment device on the vehicle 100 is used to optimize the parameters of the unoptimized ISP 14.

[0090] Optionally, the above-mentioned set of optimized cameras 11 and ISP 12, and the set of unoptimized cameras 13 and ISP 14 can also be set in other devices, and the other devices transmit the acquired optimized images and unoptimized images to the ISP parameter adjustment device. As a way, when the ISP parameter adjustment device is set on the vehicle 100, other devices can be connected to the vehicle 100 to transmit images to the ISP parameter adjustment device in the vehicle 100.

[0091] Optionally, the camera 11 and the camera 13 are arranged close to each other, such as adjacent to each other or the distance between the camera 11 and the camera 13 is less than a preset distance value, to ensure that the camera 11 and the camera 13 can approximately capture images of the same scene. For example, the camera 11 and the camera 13 are placed closely on the vehicle 100 (without leaving a gap in between), and the shooting angles face the same direction.

[0092] The camera 11 includes a lens 111 and an image sensor 112. The RGB image output process of the camera 11 includes: firstly collecting optical information of the vehicle's surrounding environment through the lens 111, then generating a raw image on the image sensor 112, and then the ISP 12 processes the raw image with an optimized ISP parameter set to obtain an RGB image that meets the quality requirements.

[0093] Similarly, the camera 13 includes a lens 131 and an image sensor 132. The RGB image output process of the camera 13 includes: firstly, optical information of the vehicle's surrounding environment is collected through the lens 131 (at this time, it can be considered that the lens 131 and the lens 111 are photographing the same environment), and then a raw image is generated on the image sensor 132, and then the ISP 14 processes the raw image with unoptimized ISP parameters to obtain an RGB image with unstable quality.

[0094] The vehicle 100 may also be provided with at least one other sensor, such as a vehicle-mounted radar (such as a millimeter wave radar, a lidar, an ultrasonic radar, etc.), a rain sensor, a vehicle posture sensor (such as a gyroscope), an inertial measurement unit (IMU), a positioning system such as a global positioning system (GPS), etc.

[0095] like Figure 1 As shown, the vehicle 100 may also include a computer system 15. In the embodiment of the present application, some or all functions of the vehicle 100 are controlled by the computer system 15. Optionally, the above-mentioned ISP parameter adjustment device may be a computer system 15 or a component or assembly in the computer system 15, such as the ISP parameter adjustment device may be a processor or chip in the computer system 15. The computer system 15 may include at least one processor that executes instructions stored in a non-transitory computer-readable medium such as a data storage device. The computer system 15 may also be a plurality of computing devices that control individual components or subsystems of the vehicle 100 in a distributed manner.

[0096] The processor may include one or more processing units, for example, the processor may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an ISP, a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0097] In the embodiment of the present application, the ISP 12 may be deployed in the camera 11, that is, the camera 11 may be composed of a lens 111, an image sensor 112 and an ISP 12. The ISP 12 may also be deployed in the above-mentioned processor, or deployed in the vehicle 100 in the form of an independent device. Similarly, the ISP 14 may be deployed in the camera 13, that is, the camera 13 may be composed of a lens 131, an image sensor 132 and an ISP 14. The ISP 14 may also be deployed in the above-mentioned processor, or deployed in the vehicle 100 in the form of an independent device. In one example, a processor may include multiple image signal processors, and the multiple image signal processors may include the above-mentioned ISP 12 and ISP 14.

[0098] In an embodiment of the present application, the above-mentioned processor can execute the ISP parameter adjustment method of the embodiment of the present application to automatically adjust the ISP parameter set in the ISP 14 to be debugged and optimized. The processor can be deployed with neural network models such as an ISP neural network (also referred to as a proxy neural network), a target perception (recognition) neural network (also referred to as a first perception neural network), and a reference perception neural network (also referred to as a second perception neural network). Among them, the target perception neural network and the reference perception neural network are deep neural network models with the same perception task, and the perception task can be any one or more tasks such as a detection task (such as detecting a face, detecting the position of a specific target such as a motor vehicle, non-motor vehicle or pedestrian), a segmentation task (such as detecting which specific image pixels belong to categories such as pedestrians and motor vehicles), and a classification task (such as detecting which targets in the image are pedestrians and which targets are motor vehicles). It can also be other perception tasks. The present application does not limit the type and number of perception tasks.

[0099] The reference perception neural network may be a pre-trained (hereinafter referred to as "pre-trained") neural network model, which may be a perception neural network adapted to the optimized camera 11 and ISP 12. Optionally, the reference perception neural network can complete the required perception task for the RGB image output by the optimized camera 11 and ISP 12, and output a relatively accurate perception result.

[0100] Optionally, the reference perception neural network can be a standard perception neural network provided by the customer and adapted to the optimized camera 11 and ISP 12, and its network perception capability can meet the perception capability required by the customer. As a method, the optimized camera 11 and ISP 12, as well as the adapted reference perception neural network can be uniformly provided by the customer.

[0101] The target perception neural network may be a perception neural network that is not adapted to the camera 13 and the ISP 14, and it still needs to be trained until it is adapted to the camera 13 and the ISP 14. Optionally, the target perception neural network may be an initial neural network model that has not undergone any training, or it may be any existing trained neural network model. Its network perception capability for the RGB images output by the camera 13 and the ISP 14 may not meet the customer's requirements, and it may not be able to complete the required perception tasks. The output perception results may also be biased and inaccurate.

[0102] In an embodiment of the present application, the ISP neural network model is used to simulate the processing effect of the ISP parameter set in ISP 14 on the raw image collected by the camera 13, so as to output the RGB image output by the camera 13 under the processing effect of the simulated ISP 14. The target perception neural network is used to perceive the RGB image output by the ISP neural network model to output the target perception result (also referred to as the first perception result). The reference perception neural network is used to directly perceive the RGB image output by the ISP 12 to output the reference perception result (also referred to as the second perception result). The processor is used to adjust the ISP parameter set in the ISP neural network according to the above-mentioned target perception result and the reference perception result to obtain the adjusted ISP parameter set, which can be used as the optimized parameter set of ISP 14 and configured in ISP 14, thereby optimizing ISP 14 to adapt to the camera 13, so that the camera 13 can finally output an RGB image that meets the quality requirements. The specific ISP parameter adjustment method can be referred to the following introduction.

[0103] It can be understood that the entire process described above does not rely on a large number of annotated raw images, but directly uses the RGB images output by the optimized camera to perform perception in the trained reference perception neural network. The perception results obtained can be used as pseudo-annotated data for the target perception neural network. Therefore, when the RGB images output by the ISP neural network model that simulate the output of the unoptimized camera are perceived in the target perception neural network, the perception results obtained can be evaluated with the pseudo-annotated data to obtain an evaluation result, and through the evaluation result, the ISP parameter set simulated by the ISP neural network is optimized, realizing a semi-supervised learning ISP parameter optimization scheme.

[0104] Optionally, the ISP parameter adjustment method of the embodiment of the present application can be run in the above-mentioned GPU, NPU or other processors capable of running a neural network model. Before running the ISP parameter adjustment method, any of the above-mentioned processors can load the ISP parameter adjustment method from a data storage device.

[0105] In some embodiments, the data memory may contain instructions (e.g., program code) that can be executed by the processor to perform various functions of the vehicle 100, including those described above. The data memory may also contain additional instructions, including instructions for the camera 11 to send RGB images, instructions for the image sensor 132 in the camera 13 to send raw images, and instructions for interacting with and / or controlling the cameras 11 and 13. In addition to instructions, the memory may also store data, such as RGB images, raw images, and other information. This information may be used by the vehicle 100, the computer system 15 during operation of the vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0106] Exemplarily, the data storage device may store an RGB image of the surrounding environment acquired by the vehicle 100 based on the camera 11, and may also store a raw image of the surrounding environment acquired by the vehicle 100 based on the camera 13. The storage device may also store an optimized ISP parameter set of the ISP 14 acquired from the processor. In addition to the above, the storage device may also store algorithms of neural network models such as an ISP neural network, a target perception neural network, and a reference perception neural network, as well as program codes corresponding to the ISP parameter adjustment method provided in the embodiments of the present application.

[0107] Alternatively, the computer system 15 may be located remotely from the vehicle 100 and in wireless communication with the vehicle 100. Alternatively, some of the processes described herein may be performed by a processor within the vehicle 100 and other processes may be performed by a remote processor.

[0108] The computer system 15 can communicate with the software deployment server through a network. The network can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN). Optionally, the network can also be a wireless network, such as a wireless fidelity (Wi-Fi) network, a cellular network, etc.

[0109] Optionally, the data storage device may store an operating system (OS) and application programs of the computer system 15. The application programs include programs related to autonomous driving, such as programs for controlling the driving route or speed of an autonomous vehicle. The application programs also exist on the system of the deploying server. In one embodiment, when the application program needs to be executed, the computer system 15 may download the application program from the deploying server.

[0110] In the embodiment of the present application, the above application may include an application for controlling ISP parameter adjustment. The processor of the computer system 15 calls the application to perform the following steps:

[0111] The target raw image (also referred to as the first original image) captured by the camera 13 is input into the ISP neural network, so as to simulate the processing of the target ISP parameter set (also referred to as the first ISP parameter set) in the ISP 14 through the network to obtain a target processed image (also referred to as the first processed image), such as a target RGB image. The reference raw image (also referred to as the second original image) captured by the camera 11 is processed by the optimized reference ISP parameter set (also referred to as the second ISP parameter set) in the ISP 12 to obtain a reference processed image (also referred to as the second processed image), such as a reference RGB image. The target processed image is then input into the target perception neural network to perform image recognition through the network to obtain a target perception result; and the reference processed image is input into the reference perception neural network to perform image recognition through the network to obtain a reference perception result. Then, according to the target perception result and the reference perception result, the target ISP parameter set in the ISP neural network is adjusted to obtain an optimized parameter set (also referred to as the third ISP parameter set).

[0112] In other embodiments of the present application, the computer system 15 may also receive information from other computer systems or transfer information to other computer systems. Alternatively, data such as RGB images or raw images collected from the camera 11 and the camera 13 of the vehicle 100 may be transferred to another computer, and the data may be processed by another computer. For example, data from the computer system 15 may be transmitted to a computer system on the cloud side via a network for further processing.

[0113] In one example, the computer system on the cloud side may include a server having multiple computers, such as a load balancing server group. In order to receive, process and transmit data from the computer system of the vehicle 100, the cloud side server exchanges information with different nodes of the network. The computer system on the cloud side may have a configuration similar to that of the computer system in the vehicle 100, and has a processor, a memory, instructions, and data.

[0114] In one example, the cloud-side server may receive data (such as RGB images or raw images collected by camera 11 and camera 13) from vehicle 100 via a wireless communication network. The cloud-side server may control vehicle 100 by running a program related to parameter adjustment stored therein based on the received data. The program related to parameter adjustment may be the program for optimizing the ISP parameter set in ISP 14 based on the RGB image of camera 11, the raw image of camera 13, the ISP neural network, the target perception neural network, and the reference perception neural network. In one example, after obtaining the optimized ISP parameter set, the cloud-side server may send it to vehicle 100 so that vehicle 100 configures ISP 14 based on the optimized ISP parameter set.

[0115] In some embodiments, the vehicle 100 may also include various other subsystems, such as a travel system, a control system, one or more peripheral devices, and a power supply, and a user interface. Optionally, the vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple elements. In addition, each subsystem and element of the vehicle 100 may be interconnected by wire or wirelessly.

[0116] Among them, the travel system may include components for providing power movement for the vehicle 100, such as an engine, an energy source, a transmission, and wheels / tires. The vehicle 100 can interact with external sensors, other vehicles, other computer systems, or users through peripheral devices. The peripheral devices may include a wireless communication system, an onboard computer, a microphone, and / or a speaker. The peripheral devices may also provide a means for the user of the vehicle 100 to interact with the user interface. For example, the onboard computer can provide information to the user of the vehicle 100, and the user interface can operate the onboard computer to receive user input.

[0117] The control system is used to control the operation of the vehicle 100 and its components. The control system may include various components, such as a steering system, a throttle, a brake unit, a computer vision system, a route control system, and an obstacle avoidance system. Among them, the steering system can be operated to adjust the forward direction of the vehicle 100, which can be a steering wheel system. The throttle can be used to control the operating speed of the engine and thus control the speed of the vehicle 100. The brake unit can be used to control the deceleration of the vehicle 100.

[0118] The computer vision system can process and analyze the images captured by the cameras 11 and 13 to identify various types of objects and / or features in the environment surrounding the vehicle 100. The computer vision system can use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system can be used to map the environment, track objects, estimate the speed of objects, etc.

[0119] The route control system is used to determine the driving route of the vehicle 100. The route control system can combine data from sensors, GPS, and one or more predetermined maps to determine the driving route for the vehicle 100. Optionally, the optimization technician can pre-set the driving route for the vehicle 100 so that the camera 11 and the camera 13 on the vehicle 100 can output the RGB image on the driving route. Among them, the route control system can be used to plan the driving route for the vehicle 100 in real time according to the pre-set driving route.

[0120] The obstacle avoidance system is used to identify, evaluate and avoid or otherwise overcome obstacles in the environment of the vehicle 100. Of course, in one example, the control system may additionally or alternatively include components other than the above components. Alternatively, some of the above components may be reduced.

[0121] In some embodiments, computer system 15 may control functions of vehicle 100 based on input received from sensors such as camera 11 and camera 13 and various subsystems (eg, travel system, control system) and from a user interface.

[0122] Understandably, Figure 1 The structure shown is for illustration only and does not limit the structure of the vehicle in the embodiment of the present application. In other embodiments of the present application, the vehicle 100 may include more or fewer components than those shown in the figure, or combine some components, or separate some components, or arrange the components differently, or have different Figure 1 Equivalent functions or comparisons shown Figure 1 The functions shown are more diverse and can be configured in different ways. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0123] Optionally, when the vehicle 100 is an autonomous driving vehicle, during the autonomous driving process of the vehicle 100, the vehicle 100 or a computer device associated therewith (such as a computer system 15, a computer vision system, a memory) can automatically optimize the ISP parameter set in the ISP 14 related to the camera 13 in real time online based on the RGB image of the camera 11, the raw image of the camera 13, the ISP neural network, the target perception neural network, the reference perception neural network and other neural network models. Compared with the related art that takes a lot of time for data collection and parameter optimization and can only run the algorithm offline, the solution provided by the present application can effectively reduce the time for data collection and parameter optimization, and the algorithm operation of the entire process can be run online in real time.

[0124] It is understood that the vehicle 100 may be a car, a truck, a motorcycle, a bus, a ship, an airplane, a helicopter, a lawn mower, an entertainment vehicle, an amusement park vehicle, construction equipment, a tram, a golf cart, a train, and a cart, etc., and the present application embodiment does not make any special restrictions. For example, the vehicle 100 may also be a smart car or a smart robot with autonomous driving capability in the field of smart home.

[0125] In other embodiments of the present application, the vehicle 100 may further include a hardware structure and / or a software module to implement the above functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether one of the above functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.

[0126] For example, reference Figure 2 , the vehicle 100 may include the following modules: a processing module 201: used to simulate the processing of raw image data by the hardware ISP under the configuration of a specific ISP parameter set (such as the ISP parameter set of the hardware ISP 14) through the ISP neural network model, so as to simulate the generation of a corresponding target processed image (such as a target RGB image). At the same time, during the training process of the ISP neural network model, the processing module 201 is also used to optimize the ISP parameter set simulated by the ISP neural network model according to the difference between the RGB image generated by the simulation and the RGB image generated by the hardware ISP, so as to ensure that the processing effect of the final trained ISP neural network model on the raw image is the same as that of the hardware ISP. In an embodiment of the present application, the processing module 201 can be used to simulate the processing effect of the ISP parameter set of the hardware ISP 14 on the target raw image captured by the unoptimized camera 13, so as to output a target processed image (such as a target RGB image) output by the camera 13 under the simulated hardware ISP 14 processing effect.

[0127] The first perception module 202 is used to input the target processed image (such as the target RGB image) simulated and generated by the unoptimized camera 13 and the processing module 201 into the target perception neural network to generate a target perception result of the target processed image, and to evaluate the perception result of the target perception result with the pseudo-annotated data generated by the second perception module 203 to optimize the ISP parameter set in the ISP neural network model according to the quality of the perception result of the target perception network.

[0128] The second perception module 203 is used to input the reference processed image (such as a reference RGB image) generated by the optimized camera 11 and ISP 12 into the reference perception neural network to generate a reference perception result for the reference processed image, which can be used as pseudo-annotated data in the current vehicle environment.

[0129] In some embodiments, network parameters in the target sensing network may also be optimized according to the quality of the sensing results of the target sensing network.

[0130] The vehicle 100 may further include a first acquisition module and a second acquisition module, wherein the first acquisition module is used to acquire a target raw image and send it to the processing module 201 ; the second acquisition module is used to acquire a reference processed image (such as a reference RGB image) and send it to the second perception module 203 .

[0131] Optionally, the first acquisition module is an unoptimized camera 13, that is, the target raw image can be directly captured and sent to the processing module 201 to obtain the target processed image. The second acquisition module includes an optimized camera 11 and ISP 12, that is, the reference raw image can be directly captured and acquired, and then processed by the ISP parameter set of the optimized ISP 12 to generate a reference processed image, and sent to the second perception module 203 to obtain a reference perception result. Among them, ISP 12 can be integrated into the camera 11 or independent of the camera 11. In one scenario, the first acquisition module can also be connected to the unoptimized camera 13 to obtain the target raw image from the camera 13.

[0132] In one scenario, the second acquisition module may also be connected to the ISP 12 to directly acquire a reference processed image from the ISP 12. The ISP 12 is used to acquire a reference raw image from the optimized camera 11, and process it according to the ISP parameter set of the ISP 12 to generate a reference processed image.

[0133] In one scenario, if the ISP 12 is integrated in the camera 11 , the second acquisition module may also be directly connected to the camera 11 to directly acquire the reference processed image from the camera 11 .

[0134] In one scenario, the second acquisition module may also be an optimized ISP 12. In this case, the second acquisition module may be connected to the optimized camera 11 to acquire a reference raw image from the camera 11. After acquiring the reference raw image, the second acquisition module may process the reference raw image according to the ISP parameter set of the ISP 12 to generate a reference processed image, and send the image to the second perception module 203.

[0135] Optionally, if the unoptimized camera 13, the optimized camera 11, and the ISP 12 are located far away from the vehicle 100 and are in wireless communication with the vehicle 100, the first acquisition module can be used to acquire the target raw image from the unoptimized camera 13 through the wireless communication and send it to the processing module 201. The second acquisition module can be used to directly acquire the reference processed image from the optimized camera 11 and the ISP 12 through the wireless communication.

[0136] According to the scheme of the embodiment of the present application, the reference processed image (such as the reference RGB image) output by the optimized camera can be directly used for perception in the trained reference perception neural network, and the obtained perception result can be used as pseudo-annotated data of the target perception neural network. Therefore, when the target processed image (such as the target RGB image) output by the ISP neural network model that simulates the output of the unoptimized camera is perceived in the target perception neural network, the obtained perception result can be evaluated with the pseudo-annotated data, so as to optimize the ISP parameter set simulated by the ISP neural network through the evaluation result, thereby realizing a semi-supervised learning ISP parameter optimization scheme, which gets rid of the dependence on a large amount of landmark data and greatly reduces the time for data collection and parameter optimization.

[0137] In some embodiments, Figure 3 As shown, functional software and algorithms such as ISP parameter optimization algorithm and ISP parameter decoding algorithm can also be deployed on the vehicle 100. Among them, the ISP parameter optimization algorithm is used to optimize the ISP parameter set in the ISP neural network model according to the quality of the perception results of the target perception network. The ISP parameter decoding algorithm is used to decode the optimized ISP parameter set in the ISP neural network model to conform to the configuration of the hardware ISP 14. The decoded ISP parameters can then be configured to the hardware ISP 14.

[0138] Optionally, the vehicle 100 may include an adjustment module for evaluating the quality of the perception results of the target perception network, and optimizing the ISP parameter set simulated by the ISP neural network by running the above-mentioned ISP parameter optimization algorithm and ISP parameter decoding algorithm and other functional software and algorithms. In some scenarios, the adjustment module may also optimize the network parameters of the target perception neural network by evaluating the quality of the perception results of the target perception network.

[0139] Optionally, Figure 1The computer system 15 in the vehicle 100 may include the processing module 201, the first perception module 202, the second perception module 203, and the ISP parameter optimization algorithm and the ISP parameter decoding algorithm, so as to realize automatic optimization of the ISP parameter set of the hardware ISP 14. Optionally, the processing module 201, the first perception module 202, the second perception module 203, the ISP parameter optimization algorithm and the ISP parameter decoding algorithm may also form an independent device, such as an ISP parameter adjustment device, located in the vehicle 100.

[0140] Optionally, the above-mentioned processing module 201, first perception module 202, second perception module 203, ISP parameter optimization algorithm and ISP parameter decoding algorithm can also be deployed in a cloud-side computer system (such as a cloud-side server).

[0141] Optionally, when the computer system (including the processor) in the camera 13 has sufficient computing power, the processing module 201, the first perception module 202, the second perception module 203, the ISP parameter optimization algorithm and the ISP parameter decoding algorithm can also be deployed in the computer system of the camera 13. In this case, the camera 13 can be connected to the camera 11 and the ISP 12 for communication so as to obtain the RGB image generated by the camera 11 from the camera 11 and the ISP 12.

[0142] In some embodiments, Figure 4 As shown, the second perception module 203 may include an image registration module and a reference network perception model.

[0143] Among them, the reference network perception model is the above-mentioned trained reference perception neural network, which is used to perceive the RGB image generated by the optimized camera 11 to obtain a reference perception result. Optionally, the reference network perception model is adapted to the RGB image generated by the camera 11 and the hardware ISP 12, that is, the camera 11, the hardware ISP 12 and the reference network perception model here can be a set of adapted systems. In this way, when the reference raw image captured by the camera 11 is processed by the hardware ISP 12, a high-quality RGB image is obtained, which can be accurately perceived by the adapted reference perception neural network to accurately complete the perception task of the RGB image. Among them, this set of adapted systems can be pre-debugged or directly provided by the customer. In this way, the parameter adjustment method provided by this scheme can optimize one or more unoptimized RGB image generation systems or unoptimized RGB image perception systems according to a set of RGB image generation systems that meet the image quality requirements and RGB image perception systems that meet the perception ability requirements. Optionally, in some embodiments, the reference network perception model can also be called a teacher network model.

[0144] Among them, the image registration module is used to align the RGB image generated by the camera 11 and the hardware ISP 12 with the RGB image generated by the camera 13 and the hardware ISP 14 to eliminate the difference in viewing angle between the two cameras.

[0145] It is understandable that when the camera 11 and the camera 13 are adjacently arranged on the vehicle 100, even if the images are taken from the same viewing angle, there is still a certain viewing angle difference due to the different placement positions. The viewing angle difference between the two cameras can be eliminated by the image registration module to ensure that the contents in the RGB images obtained by the two cameras are all taken from the same scene.

[0146] It can be understood that in an ideal environment, such as a laboratory, when a group of cameras 11 and hardware ISP 12, and another group of cameras 13 and hardware ISP 14 are placed in the same position one after another to shoot the same scene from the same perspective, if the captured image content (including light presentation) can be guaranteed to be the same, then since there is no difference in perspective, image registration is not required in this case.

[0147] Optionally, the second perception module 203 may also include the above-mentioned optimized camera 11 and hardware ISP 12, and the unoptimized camera 13 and hardware ISP 14. Thus, the RGB image generated by the hardware ISP 12 and the RGB image generated by the hardware ISP 14 can be directly read for image registration, or the RGB image generated by the hardware ISP 12 can be directly read and input into the reference network perception model for perception.

[0148] Optionally, the above-mentioned optimized camera 11 and hardware ISP 12, as well as the unoptimized camera 13 and hardware ISP 14 can also be placed in the vehicle 100 in the form of independent devices, so that the reference network perception module can obtain the RGB image generated by the hardware ISP 12 and the RGB image generated by the hardware ISP 14 through the vehicle bus or other communication methods to perform image registration, or obtain the RGB image generated by the hardware ISP 12 through the vehicle bus or other communication methods, and input it into the reference network perception model for perception.

[0149] In some embodiments, Figure 4As shown, the processing module 201 may include an abstract parameter module and a proxy network model. The proxy network model may be a deep neural network model that simulates the data processing effect of a hardware ISP through training, and it may be the ISP neural network for training the data processing effect of the simulated ISP 14. In some embodiments, a simulated ISP parameter set (such as the ISP parameter set of the ISP 14 described above) and a raw image to be processed may be used as inputs of the proxy network model to simulate and generate an RGB image under the processing effect of the ISP parameter set as output of the proxy network model.

[0150] The above-mentioned abstract parameter module can be used to abstract and reduce the ISP parameter set of the hardware ISP to be simulated (such as the ISP parameter set of the above-mentioned ISP 14) to simplify the input of the proxy network model. At the same time, the abstract parameter module can also be used to decode the abstract parameters optimized by the proxy network model to conform to the configuration of the ISP parameters in the hardware ISP.

[0151] In some embodiments, Figure 4 As shown, the first perception module 202 can target a network perception model, a perception result evaluation module, and a parameter optimization module.

[0152] The target network perception model is a deep neural network with the same perception task as the reference network perception model, which can be the target perception neural network to be trained, and is used to perceive the RGB images generated by the camera 13 and the hardware ISP 14 to be optimized to obtain the target perception result. The RGB images generated by the camera 13 and the hardware ISP 14 to be optimized can be the RGB images generated by the ISP parameter set of the hardware ISP 14 simulated by the proxy network model. The target network perception model can also be called a student network model.

[0153] Optionally, the network parameters in the target network perception model can also be trained according to the quality of the perception results of the RGB images generated by the camera 13 and the hardware ISP 14 to be optimized, so as to ensure that the target network perception model finally trained is adapted to the RGB images generated by the camera 13 and the hardware ISP 14. In this way, when the ISP parameter set simulated by the proxy network is optimized and configured in the ISP 14 through the parameter adjustment method provided in the embodiment of the present application, the camera 13, the hardware ISP 14 and the finally trained target network perception model can also be optimized into a set of adapted systems. Therefore, when the target raw image captured by the camera 13 is processed by the hardware ISP 14 optimized by this solution, a high-quality RGB image can also be obtained, and it can be accurately perceived by the adapted target perception neural network, and the perception task of the RGB image can be accurately completed.

[0154] In this way, the parameter adjustment method provided by the present solution can optimize one or more unoptimized RGB image generation systems (such as unoptimized cameras 13 and ISP 14) or unoptimized RGB image perception systems (such as the above-mentioned target perception network) based on a set of RGB image generation systems that meet the image quality requirements (such as the optimized camera 11 and ISP 12) and an RGB image perception system that meets the perception capability requirements (such as the above-mentioned reference perception network), so that any other unoptimized cameras and ISPs can be optimized in the same way.

[0155] In an embodiment of the present application, the above-mentioned perception result evaluation module can use the reference perception result output by the above-mentioned reference network perception model as the annotation data to evaluate the target perception result output by the target network perception module to judge the quality of the perception ability of the target network perception simulation. Among them, the annotation data can be understood as the true value result of perception, which is usually accurately annotated or annotated manually in the relevant technology, but in the embodiment of the present application, since the reference network perception model, camera 11 and ISP 12 are an optimized system, the accuracy of the perception result finally output is high enough. In this way, when the reference RGB image output by camera 11 and ISP 12 based on the same scene captured by camera 13 and ISP 14 is perceived by the reference network perception model, the perception result obtained can be equivalent to the true value result of perception, which is used to evaluate the perception result of the target network perception model to analyze the quality of the perception result of the target network perception model.

[0156] In the embodiment of the present application, the parameter optimization module can be used to optimize the ISP parameter set abstracted in the aforementioned proxy network model according to the quality of the perception results of the target network perception model, so as to improve the perception ability index of the target network perception model for the perception task. For example, the mean average precision (mAP), average precision (AP), recall, harmonic mean of precision and recall, i.e. F1-score, and other indicators of the detection task, the mean intersection over union (mIoU), intersection over union (IoU), AP and other indicators of the segmentation task, and AP and other indicators of the classification task.

[0157] It can be understood that the second perception module 203, the processing module 201 and the first perception module 202 can execute the parameter adjustment method provided in the present application, that is, the reference RGB image is input into the Teacher perception network through the optimized camera 11 and ISP 12, and the generated reference perception result is used as the pseudo-annotated data of the Student perception network. The camera 13 that has not completed the optimization inputs the raw data into the learnable ISP neural network model, simulates and generates an RGB image under a certain parameter configuration (such as the ISP parameter set configuration of the hardware ISP 14), and the RGB image is then input into the Student perception network, and the generated target perception result is evaluated with the pseudo-annotated data (i.e., the reference perception result) generated by the Teacher network. The perception result is fed back to the ISP parameter set optimization algorithm to complete the optimization of the ISP parameter set learned and simulated by the ISP neural network. The optimized ISP parameters generate an executable ISP parameter set corresponding to the hardware ISP via the ISP parameter decoding algorithm, and the ISP parameters are effective. In this way, the image quality is accurately evaluated through the CV perception task, and the image quality measurement is completed. At the same time, the present application gets rid of the dependence on manual optimization experience, and no professional optimization personnel are required to participate in the optimization. Moreover, this application uses a semi-supervised learning approach to optimize all adjustable ISP parameters, which not only improves the robustness of ISP parameters and the upper limit of image quality optimization, but also gets rid of the need for a large amount of labeled data, greatly saving manpower and financial investment.

[0158] At the same time, since the optimized camera and hardware ISP and the unoptimized camera and hardware ISP involved in this solution can be set on the same mobile device, the acquisition of the reference RGB image and the target raw image can be completed at the same time. When the device used to execute the above parameter adjustment method in this solution is placed on the mobile device, the optimization of the ISP parameter set of the unoptimized hardware ISP can be completed online directly based on the acquired reference RGB image and target raw image.

[0159] The parameter adjustment method provided in the embodiment of the present application can be executed by a parameter adjustment device. The parameter adjustment device can be a cloud service device, a terminal device, or a system composed of a cloud service device and a terminal device. For example, the parameter adjustment method can be run on a device such as an on-board computing platform whose computing power is sufficient to execute the parameter adjustment method, and the computing power of the on-board computing platform can meet an update rate of 30Hz or higher. Exemplarily, the parameter adjustment method can be performed by Figure 1 The computer system 15 of the vehicle 100, the computer system on the cloud side, or the computer system on the vehicle 100 may execute Figure 2 The parameter adjustment device in the vehicle 100 is executed.

[0160] Combine the following Figures 5 to 7The parameter adjustment method of the embodiment of the present application is introduced in detail.

[0161] See also Figure 5 , Figure 5 A flowchart of an ISP parameter adjustment method provided in an embodiment of the present application is shown. Figure 1 The vehicle 100 shown in FIG. Figure 5 As shown, the method may include:

[0162] S510: Acquire a first original image, where the first original image is obtained by capturing a first scene with a target sensor.

[0163] In the embodiment of the present application, the camera includes a lens and an image sensor. The light emitted or reflected by the object enters the lens and finally converges on the image sensor to generate an original raw image. Among them, the lens is mainly used to converge the light emitted or reflected by all objects in the shooting angle (also referred to as the scene to be shot, the target scene, and can also be understood as the scene image that the user expects to shoot) to form an image. The image sensor is mainly used to perform photoelectric conversion on the received light signal, convert it into an electrical signal, and obtain a raw image. That is, the raw image is the unprocessed original image obtained by the image sensor.

[0164] Optionally, the image sensor may also input the raw image into an image signal processor ISP for subsequent processing to convert it into a standard image supported by the algorithm or displayed on the screen. The format of the standard image may also be a red, green, blue, RGB format, or other image formats, such as a YUV color image, a YCbCr color image, or a grayscale image. The embodiment of the present application is described by taking the standard image supported by the algorithm or displayed on the screen as an RGB image as an example.

[0165] Optionally, the image sensor that generates the raw image may include a photosensitive element, and the type of the image sensor may be determined by the type of the photosensitive element. For example, when the sensory element is CMOS, the image sensor is a CMOS sensor, and when the sensory element is a charge coupled device (CCD), the image sensor is a CCD sensor. Optionally, the color format of the raw image is determined by a color filter array (CFA), i.e., a filter, placed in front of the image sensor, and the raw image may be an image acquired under various CFA formats. Exemplarily, the raw image may be a Bayer image in RGGB format, an image in red yellow yellow blue (RYYB) format, a Bayer image in red green blue infrared (RGBIR) arrangement, and the like. In the embodiment of the present application, the raw image is taken as the most common Bayer format image as an example for illustration.

[0166] Based on the above analysis, it can be known that the image quality of the RGB image supported by the algorithm or displayed on the screen is usually affected by four factors: the light conditions of the external environment, the optical characteristics of the camera, the photoelectric conversion characteristics of image sensors such as CMOS, and the processing algorithm and processing power of the ISP. Since the optical characteristics of the camera and the photoelectric conversion characteristics of image sensors such as CMOS are usually pre-set by the manufacturer, they are usually not easy to change after sale. Therefore, at present, the processing algorithm and processing power of the hardware image signal processor are usually optimized by adjusting the ISP parameter set in the hardware image signal processor, so as to optimize the RGB image generated by the camera, that is, the camera can generate high-quality RGB image data under different light conditions.

[0167] The target sensor is an image sensor in a camera to be optimized. Figure 1 The camera 13 in the optimization process can be arranged on a vehicle, and the vehicle can obtain the raw images captured by the target sensor in the camera to be optimized in various scenes. The scene captured by the target sensor in the optimization process can be defined as the first scene. In the embodiment of the present application, the raw image captured by the target sensor in the camera to be optimized can be defined as the target raw image, or the first original image.

[0168] Optionally, the vehicle travels on a fixed route, and the shooting angle of the camera to be optimized on the vehicle is fixed, so that during the movement of the vehicle, the vehicle can obtain multiple frames of target raw images captured by the target sensor for the scene in the shooting angle in real time. This application will be described by taking a frame of target raw image captured by the target sensor, where the frame of target raw image captures the target scene as an example.

[0169] S520. Input the first original image into the pre-trained proxy neural network to obtain a first processed image, wherein the proxy neural network is used to process the original image captured by the target sensor according to the first ISP parameter set and output a corresponding processed image.

[0170] In an embodiment of the present application, the vehicle also includes a hardware image signal processor ISP corresponding to the camera to be optimized, and the parameter set in the image signal processor ISP is not optimized. Therefore, the raw image captured by the camera to be optimized is processed by the unoptimized hardware ISP to obtain an RGB image that often fails to meet the image quality requirements. In an embodiment of the present application, a proxy neural network model can be used to simulate the processing process of the unoptimized hardware ISP under different parameter sets. Therefore, in the process of adjusting the parameters of the hardware ISP, the hardware ISP can be replaced with the proxy neural network model, and then the ISP parameter set in the proxy neural network model can be optimized to obtain the final optimized ISP parameter set configured in the hardware ISP.

[0171] The ISP parameter set includes multiple parameters, such as color correction parameters, gamma correction parameters, contrast enhancement parameters, denoising parameters, sharpening parameters, etc. The embodiment of the present application does not limit the type and number of parameters to be adjusted in the hardware ISP, and can be reasonably set according to the actual application situation.

[0172] The above-mentioned proxy neural network is the ISP neural network model used in this application to simulate the processing effect of the unoptimized hardware ISP. The parameter set in the unoptimized hardware ISP is defined here as the target ISP parameter set, or the first ISP parameter set. By inputting the target raw image acquired by the target sensor into the proxy neural network, the vehicle can obtain the target processed image (processed image) simulated and generated by the proxy neural network, such as the target RGB image. The target processed image can also be defined as the first processed image here.

[0173] In some embodiments, the proxy neural network can be a pre-trained neural network model that can simulate the same processing effect as the above-mentioned unoptimized hardware ISP. It can be pre-stored in the vehicle or obtained from the cloud side or other devices, which is not limited in the embodiments of the present application.

[0174] As an example, Figure 2 The processing module 201 of the vehicle can be used to execute the training process of the above-mentioned proxy neural network, so that the processing effect of the proxy neural network on the raw image is the same as the processing effect of the unoptimized hardware ISP.

[0175] Alternatively, if Figure 6 As shown, the unoptimized camera and hardware ISP are Figure 1 Taking the camera 13 and ISP 14 in the example, the vehicle can collect raw images of the camera 13 in various scenes, and use the hardware ISP 14 to process these raw images under different ISP parameter configurations (such as using ISP offline simulation software to process raw images) to generate corresponding RGB images. These RGB images can be used as annotation data for characterizing the true value results of image processing, so as to be used for training the proxy neural network. Then, the raw images of various scenes collected by the camera 13 and the corresponding ISP parameter set are used as the input of the proxy neural network to obtain the simulated RGB images output by the proxy neural network. Then, the simulated RGB images can be evaluated based on the RGB images of the aforementioned annotated data (such as calculating the deviation loss between the two images) to optimize the network parameters in the proxy neural network according to the evaluation. Through repeated iterations, the finally optimized and trained proxy neural network can successfully simulate the processing effect of the hardware ISP 14 under different ISP parameter configurations.

[0176] Optionally, in order to reduce the computational complexity of the proxy neural network, the vehicle may also abstract and reduce the dimension of the input ISP parameter set, thereby simplifying the ISP parameter input of the proxy neural network.

[0177] In some embodiments, the proxy neural network may also be trained in real time iteration during the process of the vehicle collecting the target raw image. This embodiment of the present application is not limited thereto.

[0178] In some embodiments, the training process of the above-mentioned proxy neural network can also be performed in the cloud or other devices, and the vehicle can directly obtain the trained proxy neural network.

[0179] S530, obtaining a second processed image, where the second processed image is obtained by processing a second original image according to a second ISP parameter set, where the second original image is obtained by photographing the first scene with a reference sensor, and the second ISP parameter set is adapted to the reference sensor.

[0180] The reference sensor is an image sensor in an optimized camera. Figure 1The camera 11 in the optimized camera can also be set on the vehicle, and the vehicle can obtain the raw image captured by the reference sensor in the optimized camera in various scenes. In the embodiment of the present application, the raw image captured by the parameter sensor in the optimized camera can be defined as a reference raw image, or a second original image. The reference sensor and the aforementioned target sensor need to shoot the same scene to ensure that the scene picture captured by the target sensor is the same.

[0181] Optionally, the vehicle travels on a fixed route, and the shooting angle of the optimized camera on the vehicle is fixed and the same as the shooting angle of the target sensor to ensure that the scene image captured by the target sensor is the same. Therefore, during the movement of the vehicle, the vehicle can also obtain multiple frames of reference raw images captured by the reference sensor for the scene in the shooting angle in real time. This application will be described by taking a frame of reference raw image captured by the reference sensor, which captures the same target scene as the target raw as an example.

[0182] Optionally, the target sensor and the reference sensor can be located on the vehicle at the same time and arranged adjacent to each other. In this way, during the movement of the vehicle, the vehicle can obtain the target raw image captured by the target sensor, and can also obtain the reference raw image captured by the reference sensor. In this way, the target sensor and the reference sensor can generate the target raw image and the reference raw image in the same scene and under the same light conditions, and can perform the parameter adjustment method provided in the embodiment of the present application in real time online.

[0183] In the embodiment of the present application, the vehicle further includes a hardware image signal processor ISP adapted to the optimized camera, and the parameter set in the image signal processor ISP has been optimized, so that the raw image captured by the optimized camera is processed by the optimized hardware ISP to obtain a reference processed image with symbol image quality requirements, such as a reference RGB image. The parameter set in the optimized hardware ISP is defined here as a reference ISP parameter set, or a second IPS parameter set.

[0184] In the embodiment of the present application, after the vehicle obtains the reference raw image collected by the reference sensor, the optimized hardware ISP can be used to process the reference raw image to generate a reference processed image. The reference processed image can also be defined as the second processed image.

[0185] Optionally, if there is a pre-trained proxy neural network model corresponding to the optimized hardware ISP, which can successfully simulate the processing process of the optimized hardware ISP under different parameter sets. In the embodiment of the present application, the reference raw image collected by the reference sensor can also be processed by the proxy neural network model corresponding to the optimized hardware ISP, so as to obtain a reference processed image (second processed image) generated by model simulation. In this way, the participation of the hardware ISP may not be required in the entire optimization process.

[0186] S540: Input the first processed image into a first perception neural network to obtain a first perception result.

[0187] In an embodiment of the present application, a first perceptual neural network and a second perceptual neural network having the same visual perception task may be deployed on the vehicle, which may be used to perform visual perception on standard images such as RGB images. The first perceptual neural network is the aforementioned target perceptual neural network, which may be an untrained Student network model. The second perceptual neural network is the aforementioned reference perceptual neural network, which may be a pre-trained Teacher network model.

[0188] It is understandable that the currently trained or mature CV perception networks are usually large neural network models, which include a large number of learning parameters. However, a large number of learning parameters and a large amount of calculations required to apply these parameters will make it difficult to deploy the trained CV perception network on a resource-constrained computing device. Wherein resource constraints may refer to one or more of limited memory, limited processing power, and limited power supply. Such as the vehicle in the embodiment of the present application.

[0189] As a solution, the CV perception model can be compressed by model compression to reduce the learning parameters in the trained CV perception model, so that the compressed trained CV perception model can be deployed on a resource-constrained computing device to perform perception with minimal loss of accuracy in network performance. In an embodiment of the present application, a knowledge distillation (KD) method can be used to compress the CV perception model. Among them, knowledge distillation can refer to distilling knowledge from a CV perception model learned in a non-resource-constrained computing environment (commonly referred to as a teacher network model) to a smaller student network model. In this way, the present application can use the Teacher network model (reference perception neural network) to optimize the student network model (target perception neural network) to be suitable for deployment on resource-constrained computing devices while adjusting the ISP parameters.

[0190] In an embodiment of the present application, after the vehicle obtains a target processed image containing a target scene generated by the proxy neural network simulation, the target processed image can be input into the target perception neural network to obtain a target perception result generated by the Student network in the target scene.

[0191] It is understandable that for different visual perception tasks, the perception results are different. For example, when the visual perception task is target detection, the perception result may be the position of the target, and when the visual perception task is panoptic segmentation, the perception result may be the result of panoptic segmentation.

[0192] S550: Input the second processed image into a pre-trained second perception neural network to obtain a second perception result.

[0193] In the embodiment of the present application, after the vehicle obtains the reference processed image for the same target scene generated by the optimized camera and hardware ISP, it can be input into the reference perception neural network to obtain the reference perception result generated by the Teacher network in the target scene. The reference perception result can be used as pseudo-annotated data of the Student network to evaluate the target perception result output by the Student network.

[0194] Optionally, when the reference sensor and the target sensor are both located on the vehicle and are arranged adjacent to each other, there may still be a certain visual difference, resulting in a slight deviation in the captured content. In order to avoid affecting parameter optimization, the embodiment of the present application can align the two target sensors to eliminate the visual difference.

[0195] For example, Figure 7 As shown, the unoptimized camera and hardware ISP are Figure 1 Camera 13 and ISP 14 in the , the optimized camera and hardware ISP are Figure 1 Taking the camera 11 and ISP 12 in the figure as an example, the reference raw image captured by the optimized camera 11 can obtain a hardware-generated reference processed image after being processed by the optimized hardware ISP 12. At this time, under the same scene and lighting conditions, after the unoptimized camera 13 captures the target raw image, it can not only simulate and generate the corresponding target processed image through the aforementioned proxy network model, but also directly process it through the unoptimized hardware ISP 14 to obtain the hardware-generated target processed image. Optionally, the reference processed image generated by the hardware ISP 12 can be aligned based on the target processed image generated by the hardware ISP 14. After obtaining the aligned reference processed image, the vehicle can input it into the reference perception neural network.

[0196] S560. According to the first perception result and the second perception result, adjust the first ISP parameter set in the proxy neural network to obtain a third ISP parameter set, and the third ISP parameter set is adapted to the target sensor.

[0197] In an embodiment of the present application, the vehicle can optimize the target ISP parameter set in the proxy neural network based on the difference between the reference perception result output by the reference perception neural network and the target perception result output by the target perception neural network.

[0198] It can be understood that since the reference perception neural network, reference sensor and reference ISP parameter set of the corresponding hardware ISP are equivalent to an optimized system, the perception result for the same scene finally output by this system can be equivalent to a true value result or labeled result in the scene. Therefore, there is no need to manually label the training data, and the reference perception result output by the system can be directly used as the labeled data to evaluate the quality of the perception result of the target perception neural network.

[0199] Optionally, the vehicle may compare the target perception result with the reference perception result to obtain a calculation loss, which indicates the error between the target perception result and the reference perception result. Thus, the target ISP parameter set in the proxy neural network may be optimized based on the calculation loss. As a method, an ISP parameter optimization algorithm may be deployed on the vehicle, and the vehicle may feed back the calculation loss to the ISP parameter optimization algorithm. Thus, the ISP parameter optimization algorithm may optimize the target ISP parameter set in the proxy neural network based on the calculation loss to improve the perception capability index of the target perception neural network.

[0200] As a way, the vehicle can measure the difference between the target perception result and the reference perception result of the target perception neural network through a loss function or an objective function. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. The optimization of the target ISP parameter set in the proxy neural network can be understood as the process of minimizing the loss as much as possible.

[0201] Optionally, the vehicle can also train the target perception neural network according to the target perception result and the reference perception result, that is, optimize the network parameters of the target perception neural network. Optionally, the vehicle can feed back the calculated loss obtained by comparing the above target perception result with the reference perception result to the target perception neural network, so that the target perception neural network optimizes the network parameters in the target perception neural network through the back propagation algorithm.

[0202] Optionally, the vehicle can choose to optimize the target ISP parameter set in the proxy neural network, or choose to optimize the network parameters in the target perception neural network. That is, the embodiment of the present application provides two optimization operations, a first optimization operation and a second optimization operation. Among them, the first optimization operation indicates the optimization of the target ISP parameter set in the proxy neural network, and the second optimization operation indicates the optimization of the target perception neural network. In this way, it is possible to optimize the unoptimized camera and hardware ISP, and it is also possible to optimize the perception network adapted to the unoptimized camera and hardware ISP. And because the target perception neural network is after model compression, it can be widely deployed on various resource-constrained computing devices.

[0203] Optionally, the vehicle may switch between the first optimization operation and the second optimization operation. When switching to the first optimization operation, the vehicle may adjust the target ISP parameter set in the proxy neural network according to the target perception result and the reference perception result. When switching to the second optimization operation, the vehicle may train the target perception neural network according to the target perception result and the reference perception result.

[0204] Optionally, the switching of the above-mentioned optimization operation may be a switching between the first optimization operation and the second optimization operation every preset number of iterations. The preset number may be 1, that is, after completing one iteration, optimizing the target ISP parameter set in the proxy neural network, completing another iteration, switching to optimizing the target perception neural network, completing another iteration, and switching back to optimizing the target ISP parameter set in the proxy neural network, and repeating this process to achieve the joint optimization of the target ISP parameter set and the target perception neural network. The preset number may be any number greater than 1, such as 10 times, that is, after completing 10 iterations, optimizing the target ISP parameters in the proxy neural network, completing 10 iterations, switching to optimizing the target perception neural network, completing 10 iterations, and switching back to optimizing the target ISP parameter set in the proxy neural network, and repeating this process to achieve the joint optimization of the target ISP parameter set and the target perception neural network.

[0205] Optionally, the second optimization strategy is executed during the first X iterations, and the first optimization strategy is switched after the X iteration, where X is a positive integer. In this way, after the target perception neural network is successfully optimized to the perception capability index symbol requirement during the first X iterations, the optimization of the target ISP parameter set in the proxy neural network can be completed through multiple iterations, and then the optimization operation of the target perception neural network can be switched back. This repetition can avoid the perception capability of the unoptimized target perception neural network being too low, which affects the optimization of the target ISP parameter set.

[0206] Optionally, the first optimization strategy is executed in the first Y iterations, and the second optimization strategy is switched after the Y iteration, where Y is a positive integer. In this way, the target ISP parameter set in the proxy neural network can be successfully optimized to meet the requirements in the first Y iterations, and then the optimization of the target perception neural network is completed through multiple iterations, and then the optimization operation of the target ISP parameter set is switched back to the optimization operation of the target ISP parameter set, and this can be repeated to avoid the unoptimized target ISP parameter set from affecting the optimization of the target perception neural network.

[0207] It can be understood that after multiple iterations, after repeating the above parameter optimization process to the preset conditions, the above parameter optimization process can be stopped, and the target ISP parameter set in the final optimized proxy neural network is configured to the unoptimized hardware ISP (such as Figure 1 ISP 14 in). The preset condition may be that the iteration reaches a preset number of times, or that the above-mentioned calculation loss converges, or converges to a preset value. The embodiment of the present application does not limit the end condition of the above-mentioned parameter optimization process.

[0208] In the embodiment of the present application, the target ISP parameter set in the finally optimized proxy neural network can be defined as the third ISP parameter set. In this way, the third original image captured by the target sensor in the actual scene can be directly processed according to the optimized third ISP parameter set to obtain a third processed image that meets the quality requirements.

[0209] Optionally, the aforementioned optimized third ISP parameter set can be configured to the unoptimized hardware ISP, and an optimized hardware ISP can be obtained. The optimized hardware ISP can be applied to the actual scene with the target sensor to output an RGB image that meets the quality requirements. That is, the third original image captured by the target sensor in the actual scene can be directly sent to the optimized hardware ISP, so that the optimized hardware ISP processes the third original image according to the third ISP parameter set to obtain a third processed image that meets the quality requirements.

[0210] Optionally, if the ISP parameter set in the proxy neural network is abstracted and dimensionally reduced, then in order to comply with the configuration of the hardware ISP, the vehicle can decode the optimized ISP parameters into ISP parameters that are adapted to the hardware ISP through an ISP parameter decoding algorithm.

[0211] In some scenarios, the vehicle can also update the network parameters of the reference perception neural network according to the optimized target perception neural network, thereby optimizing the reference perception neural network. As an example, at time t, the parameter set of the reference perception neural network is The parameter set of the target perception neural network is Then the update strategy of the reference perception neural network at time t can be: Among them, β is a weight value of [0, 1].

[0212] In this way, in the process of optimizing the target ISP parameters in the target perception neural network or the proxy neural network, the network parameters of the reference perception neural network can also be optimized and updated in reverse. This allows the reference perception neural network to adapt to the target sensor, that is, it can also perceive the RGB image output by the target sensor and the optimized target ISP parameter set in accordance with customer requirements and obtain accurate perception results. This achieves the optimization of the perception ability of the reference perception neural network.

[0213] It can be understood that the above description is made using an example in which a non-optimized camera is installed on a vehicle. When a plurality of non-optimized cameras are installed on the vehicle, the ISP parameters of the plurality of non-optimized cameras can be optimized based on the same principle.

[0214] It can be understood that the above is only an example of the field of autonomous driving. For other fields to which the parameter adjustment method of this application is applicable, such as intelligent security, portable mobile terminals (such as mobile phones, tablets, etc.), cameras, intelligent robots, drones, machine vision, etc., it can be executed based on the same principle. The difference may lie in the definition of application scenarios, the definition and abstraction of ISP parameters, the design of network simulation ISP, the design of Student / Teacher perception network, the definition of perception result evaluation, the selection of ISP parameter optimization algorithm, etc. The embodiments of this application are not specifically limited and can be reasonably set according to specific applications.

[0215] The parameter adjustment method provided in the embodiment of the present application realizes ISP parameter adjustment of semi-supervised learning by adopting a group of unoptimized cameras A and ISP, a group of optimized cameras B and ISP, and a teacher-student network model architecture. It gets rid of the need for a large amount of manually labeled data, and also greatly saves the time and labor costs of ISP parameter optimization.

[0216] It is understandable that in order to achieve the above functions, the vehicle includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0217] In this embodiment, the functional modules of the vehicle can be divided according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0218] The embodiment of the present application also provides an ISP parameter adjustment device, which can be applied to the above-mentioned vehicle. The device is used to perform each function or step performed by the vehicle in the above-mentioned method embodiment. Optionally, the ISP parameter adjustment device can be an on-board computing platform or a computer system on the vehicle.

[0219] The embodiment of the present application also provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected through a line. The interface circuit can read the instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the vehicle can execute the various functions or steps executed by the vehicle in the above method embodiment. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiment of the present application.

[0220] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned vehicle, the vehicle executes each function or step executed by the vehicle in the above-mentioned method embodiment.

[0221] An embodiment of the present application also provides a computer program product. When the computer program product is run on a vehicle, the vehicle is enabled to execute each function or step executed by the vehicle in the above method embodiment.

[0222] Among them, the vehicle, ISP parameter adjustment device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0223] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0224] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0225] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for adjusting image signal processor ISP parameters, characterized in that: The method comprises: Acquire a first original image, where the first original image is obtained by capturing a first scene by a target sensor; Inputting the first original image into a pre-trained proxy neural network to obtain a first processed image, wherein the proxy neural network is used to process the original image captured by the target sensor according to a first ISP parameter set and output a corresponding processed image; Acquire a second processed image, where the second processed image is obtained by processing a second original image according to a second ISP parameter set, where the second original image is obtained by photographing the first scene with a reference sensor, and the second ISP parameter set is adapted to the reference sensor; Inputting the first processed image into a first perception neural network to obtain a first perception result; Inputting the second processed image into a pre-trained second perceptual neural network to obtain a second perceptual result; According to the first perception result and the second perception result, the first ISP parameter set in the proxy neural network is adjusted to obtain a third ISP parameter set, and the third ISP parameter set is adapted to the target sensor.

2. The method according to claim 1, characterized in that The method further comprises: The first perception neural network is trained according to the first perception result and the second perception result.

3. The method according to claim 2, characterized in that The method further comprises: Switching between a first optimization operation and a second optimization operation, wherein the first optimization operation includes adjusting the first ISP parameter set in the proxy neural network according to the first perception result and the second perception result, and the second optimization operation includes training the first perception neural network according to the first perception result and the second perception result.

4. The method according to claim 3, characterized in that The switching between the first optimization operation and the second optimization operation includes: Switching between the first optimization operation and the second optimization operation every preset number of iterations; or Performing the second optimization operation during the first X iterations, and switching to the first optimization operation after the Xth iteration, where X is a positive integer; or The first optimization operation is performed during the first Y iterations, and the second optimization operation is switched to be performed after the Y iteration, where Y is a positive integer.

5. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: The network parameters of the second perceptual neural network are updated according to the trained network parameters of the first perceptual neural network.

6. The method according to any one of claims 1 to 5, characterized in that: The adjusting the first ISP parameter set in the proxy neural network according to the first perception result and the second perception result includes: comparing the first perception result with the second perception result to obtain a calculated loss, the calculated loss indicating an error between the first perception result and the second perception result; The first ISP parameter set in the proxy neural network is adjusted according to the calculated loss.

7. The method according to any one of claims 1 to 6, characterized in that: The target sensor is disposed adjacent to the reference sensor.

8. The method according to claim 7, characterized in that The target sensor and the reference sensor are arranged on a movable device, and the method is executed during the movement of the movable device.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: The third ISP parameter set is configured as a parameter of the first ISP, and the first ISP is used to process a third original image obtained by the target sensor shooting the second scene according to the third ISP parameter set.

10. An image signal processor ISP parameter adjustment device, characterized in that: The ISP parameter adjustment device includes a memory and one or more processors; the memory and the processor are coupled; the memory is used to store computer program code, and when the processor executes the computer program code, the ISP parameter adjustment device executes the method as described in any one of claims 1-9, and the ISP parameter adjustment device includes a vehicle-mounted computing platform.

11. An image signal processor ISP parameter adjustment system, characterized in that: The ISP parameter adjustment system includes a target sensor, a reference sensor and an ISP parameter adjustment device, wherein: The target sensor is used to capture a first scene to obtain a first original image, and send the first original image to the ISP parameter adjustment device; The ISP parameter adjustment device is used for, when receiving the first original image, inputting the first original image into a proxy neural network to obtain a first processed image, and inputting the first processed image into a first perception neural network to obtain a first perception result, wherein the proxy neural network is used for processing the original image captured by the target sensor according to the first ISP parameter set, and outputting a corresponding processed image; The reference sensor is used to capture the first scene to obtain a second original image, process the second original image according to a second ISP parameter set to obtain a second processed image, and send the second processed image to the ISP parameter adjustment device, wherein the second ISP parameter set is adapted to the reference sensor; The ISP parameter adjustment device is also used to input the second processed image into the second perception neural network when receiving the second processed image to obtain a second perception result, and adjust the first ISP parameter set in the proxy neural network according to the first perception result and the second perception result to obtain a third ISP parameter set, wherein the third ISP parameter set is adapted to the target sensor.

12. The system according to claim 11, characterized in that The ISP parameter adjustment device is also used for: The first perception neural network is trained according to the first perception result and the second perception result.

13. The system according to claim 12, characterized in that The ISP parameter adjustment device is also used for: Switching between a first optimization operation and a second optimization operation, wherein the first optimization operation includes adjusting the first ISP parameter set in the proxy neural network according to the first perception result and the second perception result, and the second optimization operation includes training the first perception neural network according to the first perception result and the second perception result.

14. A movable device, characterized in that: The movable device comprises the image signal processor ISP parameter adjustment system according to any one of claims 11 to 13, and the movable device comprises a vehicle.

15. A chip system, characterized in that: The chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected via lines; the interface circuit is used to receive computer instructions; when the processor executes the computer instructions, the chip system executes the method as described in any one of claims 1-9.

16. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 9.

Images