ISP parameter adjustment method and device based on multi-agent reinforcement learning

Abstract

The embodiment of the invention provides an ISP parameter adjustment method and device based on multi-agent reinforcement learning. The method comprises the steps that original images of different reinforcement learning environments are acquired, RGB images in the original images are converted into RAW images, and an RAW-RGB data set is obtained; constructing a multi-agent reinforcement learning model, and training the multi-agent reinforcement learning model through the training set to obtain a trained multi-agent reinforcement learning model; and performing parameter prediction on an input RAW image according to the trained multi-agent reinforcement learning model to obtain ISP parameters. In this way, the internal association of the parameters in the same module and the coupling relation of the parameters among different modules can be considered through a multi-agent structure, effective feedback and decision making are carried out, the time for searching the optimal parameters is shortened, and more accurate ISP parameters are obtained.

Description

Technical Field

[0001] The present invention generally relates to the fields of computer vision, artificial intelligence and deep learning, and signal processing, and more specifically, to an ISP parameter adjustment method, device and equipment based on multi-agent reinforcement learning. Background Art

[0002] Image Signal Processor (ISP) is an image signal processing pipeline that converts images from RAW format to RGB format that can be seen by humans. Hardware image signal processors often generate better images than software and take less time. Image signal processors are used in a wide range of fields, such as mobile phones, cameras, and autonomous driving. In addition, image signal processors are also an important bridge between image sensors and downstream task applications. It consists of a series of modules responsible for different image processing tasks, such as denoising modules, white balance modules, sharpening modules, etc. Each module contains a set of adjustable hyperparameters, and the hyperparameters in all modules are closely related to the quality of the final image. At present, the industry mostly relies on imaging experts to manually tune the parameters of each module in the entire image signal processor to adapt it to different scenarios. Usually, experts evaluate the image quality generated by the image signal processor based on subjective feelings, and comprehensively evaluate the effect of a set of parameters based on objective indicators such as structural similarity or peak signal-to-noise ratio, and find a relatively optimal set of parameters among a large number of image signal processing parameter combinations. However, since experts can only adjust certain parameters based on a scene graph at a time, the entire process is time-consuming and labor-intensive, and inevitably introduces human subjective errors.

[0003] In recent years, with the in-depth research of the image signal processor parameter adjustment problem in academia and industry, many methods for image signal processing agents or image signal processing parameter prediction have emerged. Some methods based on evolutionary strategies continuously search and iterate in the parameter space by maximizing the parameter fitness, and finally find a set of parameters with the highest fitness on the current image in the continuously generated parameter combinations as the optimal parameter combination. Some methods based on convolutional neural networks directly learn a model for predicting parameters to generate a set of image signal processing parameters for different images. However, methods based on evolutionary strategies often have no targeted prediction capabilities based on new images, and can only find the corresponding parameters by re-searching. Methods based on convolutional neural networks require large-scale RAW-RGB data for model training, but the industry currently lacks relevant large-scale data sets.

[0004] At present, there are many problems in the parameter tuning of image signal processors, such as large scale of prediction parameters, poor tuning effect, poor correlation and interpretability of parameters inside and outside the module, etc. Summary of the invention

[0005] According to an embodiment of the present invention, an ISP parameter adjustment scheme based on multi-agent reinforcement learning is provided. This scheme considers the intrinsic correlation of parameters in the same module and the coupling relationship of parameters between different modules through a multi-agent structure, performs efficient feedback and decision-making, shortens the time for searching for optimal parameters, and obtains more accurate parameter values.

[0006] In a first aspect of the present invention, a method for adjusting ISP parameters based on multi-agent reinforcement learning is provided. The method comprises:

[0007] Obtaining original images of different reinforcement learning environments, converting RGB images in the original images into RAW images, and obtaining a RAW-RGB data set; the RAW-RGB data set includes a training set and a test set;

[0008] Constructing a multi-agent reinforcement learning model, and training the multi-agent reinforcement learning model through a training set to obtain a trained multi-agent reinforcement learning model; the multi-agent reinforcement learning model includes a plurality of single agents, each of which is composed of an Actor network model and a Critic network model;

[0009] The parameters of the input RAW image are predicted according to the trained multi-agent reinforcement learning model to obtain the ISP parameters.

[0010] Furthermore, the multi-agent reinforcement learning model is trained by the training set to obtain a trained multi-agent reinforcement learning model, including:

[0011] S201, inputting the RAW image in the training set and the parameters corresponding to each module in the ISP into the ISP to obtain an RGB image;

[0012] S202, performing downstream task processing on the RGB image to obtain a score and reward value of the current image by the reinforcement learning environment;

[0013] S203, input the RGB image into each single agent, output the corresponding action, and splice the actions output by all single agents to obtain the total action;

[0014] S204, adding preset noise to the total action and fusing it with the parameters corresponding to each module in the current ISP to obtain the parameters corresponding to each module in the next round of ISP;

[0015] S205. Iterate S201 to S204 under the constraints of the loss function of the Actor network model and the loss function of the Critic network model to obtain a trained multi-agent reinforcement learning model.

[0016] Furthermore, each round of iteration generates a tuple, which includes a local observation value of the current environment, a state of the current environment, a total action, a score of the current image, and a reward value;

[0017] In each round of iteration, the tuple is sent to the Actor network model, the cumulative discounted return and the loss function of the Actor network model are calculated, and the weight parameters of the Actor network model are updated;

[0018] In each round of iteration, the tuple is sent to the Critic network model, the loss function of the Critic network model is calculated, and the weight parameters of the Critic network model are updated.

[0019] Furthermore, the Actor network model includes:

[0020] A first Resnet50 feature extraction network is used to take the RGB image as input and output a first intermediate feature; the first intermediate feature and the parameters corresponding to the modules in a single ISP are used as local observation values ​​of the current environment;

[0021] A first multilayer perceptron, the first multilayer perceptron comprising a plurality of fully connected layers, configured to take a local observation value of a current environment as input and output a single result of the local observation value;

[0022] The first adaptive average pooling layer is used to take a single result of a local observation as input and output the action of a single agent.

[0023] Furthermore, the Critic network model includes:

[0024] The second Resnet50 feature extraction network is used to take the RGB image as input and output the second intermediate feature; the second intermediate feature and the parameters corresponding to all modules in the ISP are used as the state of the current environment;

[0025] a second multilayer perceptron, the second multilayer perceptron comprising a plurality of fully connected layers, configured to take the state of the current environment as input and output a single result of the state; and the output dimension of the second multilayer perceptron is different from that of the first multilayer perceptron;

[0026] The second adaptive average pooling layer is used to take a single result of the state as input and output the state value of a single agent.

[0027] Furthermore, the loss function of the Actor network model is:

[0028] ;

[0029] in, is the loss function of the Actor network model; is the network parameter in the Actor network model; i represents the i-th single agent; B is the size of batch size, and n is the number of single agents; is the advantage function value of the current single agent, and S represents the entropy of the strategy; is a hyperparameter that controls the entropy coefficient, is the importance sampling ratio of the current strategy to the previous round strategy; represents the action output by the i-th single agent; represents the observation value of the i-th single agent; for summing variables; It is the hyperparameter that controls the boundary in the clip function, and its value is (0,1); represents the current strategy probability; represents the strategy probability of the previous round.

[0030] Furthermore, the loss function of the Critic network model is:

[0031] ;

[0032] in, is the loss function of the Actor network model; is the network parameter in the Critic network model; i represents the i-th single agent; t is the time step; is the cumulative discounted return; B is the batch size; n is the number of single agents; Indicates status The value of is the state corresponding to the i-th single agent; It is the hyperparameter that controls the boundary in the clip function, and its value is (0,1); Indicates the state value of the previous round; is the summation variable.

[0033] Furthermore, the calculation of the cumulative discount return includes:

[0034] ;

[0035] Where T is the expected step length; is the decay factor, the value range is (0,1]; t is the time step, R is the reward value in the tuple; is the cumulative discounted return; i represents the i-th single agent; j represents the index of a time step.

[0036] In a second aspect of the present invention, a method for image evaluation based on multi-agent reinforcement learning is provided. The method comprises:

[0037] The ISP parameters are obtained by the ISP parameter adjustment method based on multi-agent reinforcement learning as described in the first aspect above; the ISP parameters and the RAW image are input into the trained multi-agent reinforcement learning model to output the final RGB image.

[0038] In a third aspect of the present invention, an electronic device is provided. The electronic device has at least one processor; and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method of the first aspect of the present invention.

[0039] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0041] Figure 1 A flowchart of an ISP parameter adjustment method based on multi-agent reinforcement learning according to an embodiment of the present invention is shown;

[0042] Figure 2 A flowchart of multi-agent reinforcement learning model training according to an embodiment of the present invention is shown;

[0043] Figure 3 A block diagram of an image evaluation method based on multi-agent reinforcement learning according to an embodiment of the present invention is shown;

[0044] Figure 4 A schematic diagram of a network structure of a single agent according to an embodiment of the present invention is shown;

[0045] Figure 5 A block diagram of an exemplary electronic device capable of implementing embodiments of the present invention is shown;

[0046] Among them, 500 is an electronic device, 501 is a computing unit, 502 is a ROM, 503 is a RAM, 504 is a bus, 505 is an I / O interface, 506 is an input unit, 507 is an output unit, 508 is a storage unit, and 509 is a communication unit. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0049] Figure 1 A flowchart of the ISP parameter adjustment method based on multi-agent reinforcement learning in an embodiment of the present invention is shown.

[0050] The method includes:

[0051] S101. Obtain original images of different reinforcement learning environments, convert RGB images in the original images into RAW images, and obtain a RAW-RGB dataset.

[0052] In this embodiment, the data source of the original image may be a MS COCO (Microsoft Common Objects in Context) dataset, which is a large-scale target detection, segmentation, key point detection and description dataset.

[0053] In this embodiment, the image signal processor ISP includes several modules, such as a noise reduction module, a white balance module, a demosaicing module, a sharpening module, a color and tone correction module, etc.

[0054] As an implementation method of this embodiment, the MS COCO dataset is processed using an image reversal method to convert the original RGB image into a corresponding bayyer format image, that is, a RAW image, and finally a RAW-RGB dataset is obtained.

[0055] The RAW-RGB dataset includes a training set and a test set. The division of the training set and the test set of the RAW-RGB dataset is consistent with the original MS COCO dataset.

[0056] S102: construct a multi-agent reinforcement learning model, and train the multi-agent reinforcement learning model through a training set to obtain a trained multi-agent reinforcement learning model.

[0057] In this embodiment, the multi-agent reinforcement learning model includes multiple single agents, and each single agent is composed of an Actor network model and a Critic network model.

[0058] like Figure 4 As shown, in this embodiment, the Actor network model includes:

[0059] A first Resnet50 feature extraction network is used to take the RGB image as input and output a first intermediate feature; the first intermediate feature and the parameters corresponding to the single module in the ISP are used as local observation values ​​of the current environment;

[0060] A first multilayer perceptron (MLP), the first multilayer perceptron comprising a plurality of fully connected layers, configured to take local observations of the current environment as input and output a single result of the local observations;

[0061] The first adaptive average pooling layer (Adaptive Average Pooling) is used to take the single result of the local observation value as input and output the action of the single agent. The action of the single agent is the adjustment amount of the ISP parameter. The dimension of the action of the single agent is consistent with the number of ISP parameters in the functional module to be predicted by the single agent. The result of the ISP parameter of this time can be obtained by adding the action to the ISP parameter of the previous time.

[0062] In this embodiment, the discounted rewards are accumulated before calculating the loss function of the Actor network model. The calculation of the accumulated discounted rewards includes:

[0063] ;

[0064] Where T is the expected step length; is the decay factor, the value range is (0,1]; t is the time step, R is the reward value in the tuple; is the cumulative discounted return; i represents the i-th single agent; j represents the index of a time step, that is, the time step from t−T+1 to t.

[0065] The loss function of the Actor network model is:

[0066] ;

[0067] in, is the loss function of the Actor network model; is the network parameter in the Actor network model; i represents the i-th single agent; B is the size of batch size, and n is the number of single agents; is the advantage function value of the current single agent, and S represents the entropy of the strategy; is a hyperparameter that controls the entropy coefficient, is the importance sampling ratio of the current strategy to the previous round strategy; represents the action output by the i-th single agent; represents the observation value of the i-th single agent; for summing variables; It is a hyperparameter that controls the boundary in the clip function. It is used to limit the amplitude of the strategy update and its value is (0,1). represents the current strategy probability; represents the strategy probability of the previous round.

[0068] In the formula of the loss function of the Actor network model, the purpose of the first half of the plus sign is to maximize the advantage function, and the purpose of the second half of the plus sign is to maximize the entropy of the strategy to avoid too concentrated action distribution. Through the constraints of the loss function, the intelligent body will be more and more inclined to give high-value actions, making the parameter prediction more and more accurate.

[0069] In this embodiment, the Critic network model includes:

[0070] The second Resnet50 feature extraction network is used to take the RGB image as input and output the second intermediate feature; the second intermediate feature and the parameters corresponding to all modules of the ISP are used as the state of the current environment;

[0071] a second multilayer perceptron, the second multilayer perceptron comprising a plurality of fully connected layers, configured to take the state of the current environment as input and output a single result of the state; and the output dimension of the second multilayer perceptron is different from that of the first multilayer perceptron;

[0072] The second adaptive average pooling layer is used to take a single result of the state as input and output the state value (Value) of a single agent. The state value is a one-dimensional state value, which is used to evaluate the current state. The advantage function (Advantage Function) can be calculated through the state value. The advantage function is used to effectively optimize the strategy of the Actor network model, which is equivalent to a judgment of the action output by the Actor network model. When the Actor's action is executed, whether the new state reached is good or bad.

[0073] Compute the advantage function:

[0074] = ;

[0075] in, is the advantage function value of the current single agent; For cumulative discount returns; Indicates status The value of is the state corresponding to the i-th agent.

[0076] In this embodiment, the network structures of the Actor network model and the Critic network model are almost the same, and the difference between the two lies mainly in the output dimension of the final fully connected layer. Specifically, the image will first be pre-trained Resnet50 for preliminary feature extraction, and then the obtained features will pass through multiple layers of fully connected layers (the dimensions of the hidden layers in the fully connected network are generally 512 and 256) to obtain the final action and state values. The Actor network model and the Critic network model set the initial weight parameters.

[0077] The loss function of the Critic network model is:

[0078] ;

[0079] in, is the loss function of the Actor network model; is the network parameter in the Critic network model; i represents the i-th single agent; t is the time step; is the cumulative discounted return; B is the batch size; n is the number of single agents; Indicates status The value of is the state corresponding to the i-th single agent; It is a hyperparameter that controls the boundary in the clip function. It is used to limit the amplitude of the strategy update and its value is (0,1). To represent the state value of the previous round; is the summation variable.

[0080] In this embodiment, if Figure 2 As shown, the multi-agent reinforcement learning model is trained by the training set to obtain a trained multi-agent reinforcement learning model, including:

[0081] S201, input the RAW image in the training set and the parameters corresponding to each module in the ISP to the ISP to obtain an RGB image. If it is the first round of iteration, the input is the initialization parameters corresponding to each module in the ISP. Starting from the second round of iteration, the input is the parameters corresponding to each module in the ISP output in the previous round.

[0082] S202: Perform downstream task processing on the RGB image to obtain a score and reward value of the current image in the reinforcement learning environment.

[0083] The downstream tasks can be set as instance segmentation, target detection and image quality assessment tasks. The instance segmentation task and target detection task both use the average accuracy mAP of all categories to score the generated RGB images, while the image quality assessment task uses the structural similarity SSIM indicator to score the generated RGB images. The reward value is defined as the difference between the current image score and the image score generated by the previous round of parameter adjustment.

[0084] S203, input the RGB image into each single agent, output the corresponding action, and concatenate the actions output by all single agents to obtain the total action.

[0085] In this embodiment, taking the target detection task as an example, the RGB image is processed by the corresponding downstream task (target detection) to obtain the reward value and mAP50 result of the environment's feedback on the current image; at the same time, the obtained RGB image will be input into each single agent network, and the corresponding action of the agent will be output. The actions output by all single agents are spliced ​​together to obtain a total action. The output action is the change in the corresponding parameter, and then the actions output by all agents are spliced ​​together to obtain a total action.

[0086] S204, adding preset noise to the total action and fusing it with the parameters corresponding to each module in the current ISP to obtain the parameters corresponding to each module in the next round of ISP; the dimension of the preset noise here is the same as the action dimension, and the value of each dimension is between (0,1), in order to increase the diversity of training data.

[0087] S205. Iterate S201 to S204 under the constraints of the loss function of the Actor network model and the loss function of the Critic network model to obtain a trained multi-agent reinforcement learning model.

[0088] In this embodiment, each round of iteration generates a tuple, which includes the local observation value of the current environment, the state of the current environment, the total action, the score and reward value of the current image; in each round of iteration, the tuple is sent to the Actor network model, the cumulative discounted return and the loss function of the Actor network model are calculated, and the weight parameters of the Actor network model are updated; in each round of iteration, the tuple is sent to the Critic network model, the loss function of the Critic network model is calculated, and the weight parameters of the Critic network model are updated.

[0089] In the above embodiment, the tuple is stored in the agent's buffer, and the data in the tuple is called from the buffer for iteration during the model training process.

[0090] S103. Predict the parameters of the RAW images in the test set according to the trained multi-agent reinforcement learning model to obtain ISP parameters.

[0091] In some embodiments, the method further includes testing the trained multi-agent reinforcement learning model through a test set, specifically including:

[0092] First, the RAW image in the test set and the ISP parameters corresponding to each module in the ISP are input into the image signal processor to obtain the corresponding RGB image.

[0093] Secondly, the RGB image is processed by the corresponding downstream tasks to obtain the reward value and score of the environment's feedback on the current image; at the same time, the RGB image obtained will be passed through the Resnet50 feature extraction network to extract intermediate features, and the intermediate features and the ISP parameters corresponding to each module will be used as the current local observation values; the intermediate features and all ISP parameters will be used as the state of the current environment.

[0094] Then, each Actor network takes corresponding actions according to its local observations and states. The action output by each agent is the change in the corresponding parameters. Then the actions of each agent are concatenated together to get a total action. The total action change is applied to the previous ISP parameters to get the new round of ISP parameters.

[0095] Iterate the above test steps to a fixed number of iterations, and the ISP parameter prediction work for a RAW image is completed. Repeat the above operation for the RAW images in the test set to complete the entire test process.

[0096] As an embodiment of the present invention, Figure 3 As shown, an image evaluation method based on multi-agent reinforcement learning is also provided, including:

[0097] S301, obtaining ISP parameters through an ISP parameter adjustment method based on multi-agent reinforcement learning;

[0098] S302. Input the ISP parameters and the RAW image into the trained multi-agent reinforcement learning model to output more accurate parameters and high-quality RGB images.

[0099] According to the embodiments of the present invention, the present invention utilizes a multi-agent structure to cleverly integrate the intrinsic correlation of parameters in the same module and the coupling relationship of parameters between different modules. The model also adopts a centralized training and decentralized execution framework, allowing the value network to learn a global value function to guide different strategy networks to perform more efficient actions. The agent and the environment can efficiently feedback and make decisions, thereby improving the performance and efficiency of the entire model and obtaining more accurate parameters and high-quality images.

[0100] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0101] In the technical solution of the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0102] According to an embodiment of the present invention, the present invention further provides an electronic device.

[0103] Figure 5 A schematic block diagram of an electronic device 500 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0104] The electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0105] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0106] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as methods S101 to S103. For example, in some embodiments, methods S101 to S103 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the methods S101 to S103 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute methods S101 - S103 in any other appropriate manner (eg, by means of firmware).

[0107] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0109] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0111] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0112] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0113] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0114] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An ISP parameter adjustment method based on multi-agent reinforcement learning, characterized in that: include: Obtain original images of different reinforcement learning environments, convert RGB images in the original images into RAW images, and obtain the RAW-RGB dataset; The RAW-RGB dataset includes a training set and a test set; Constructing a multi-agent reinforcement learning model, and training the multi-agent reinforcement learning model through a training set to obtain a trained multi-agent reinforcement learning model; the multi-agent reinforcement learning model includes a plurality of single agents, each of which is composed of an Actor network model and a Critic network model; The parameters of the input RAW image are predicted according to the trained multi-agent reinforcement learning model to obtain the ISP parameters.

2. The method according to claim 1, characterized in that The multi-agent reinforcement learning model is trained by the training set to obtain a trained multi-agent reinforcement learning model, including: S201, inputting the RAW image in the training set and the parameters corresponding to each module in the ISP into the ISP to obtain an RGB image; S202, performing downstream task processing on the RGB image to obtain a score and reward value of the current image by the reinforcement learning environment; S203, input the RGB image into each single agent, output the corresponding action, and splice the actions output by all single agents to obtain the total action; S204, adding preset noise to the total action and fusing it with the parameters corresponding to each module in the current ISP to obtain the parameters corresponding to each module in the next round of ISP; S205. Iterate S201 to S204 under the constraints of the loss function of the Actor network model and the loss function of the Critic network model to obtain a trained multi-agent reinforcement learning model.

3. The method according to claim 2, characterized in that Each round of iteration generates a tuple, which includes the local observation value of the current environment, the state of the current environment, the total action, the score of the current image, and the reward value; In each round of iteration, the tuple is sent to the Actor network model, the cumulative discounted return and the loss function of the Actor network model are calculated, and the weight parameters of the Actor network model are updated; In each round of iteration, the tuple is sent to the Critic network model, the loss function of the Critic network model is calculated, and the weight parameters of the Critic network model are updated.

4. The method according to claim 2, characterized in that: The Actor network model includes: A first Resnet50 feature extraction network is used to take the RGB image as input and output a first intermediate feature; the first intermediate feature and the parameters corresponding to the single module are used as local observation values ​​of the current environment; A first multilayer perceptron, the first multilayer perceptron comprising a plurality of fully connected layers, configured to take a local observation value of a current environment as input and output a single result of the local observation value; The first adaptive average pooling layer is used to take a single result of a local observation as input and output the action of a single agent.

5. The method according to claim 4, characterized in that The Critic network model includes: The second Resnet50 feature extraction network is used to take the RGB image as input and output the second intermediate feature; the second intermediate feature and the parameters corresponding to all modules are used as the state of the current environment; a second multilayer perceptron, the second multilayer perceptron comprising a plurality of fully connected layers, configured to take the state of the current environment as input and output a single result of the state; and the output dimension of the second multilayer perceptron is different from that of the first multilayer perceptron; The second adaptive average pooling layer is used to take a single result of the state as input and output the state value of a single agent.

6. The method according to claim 3, characterized in that The loss function of the Actor network model is: ； in, is the loss function of the Actor network model; is the network parameter in the Actor network model; i represents the i-th single agent; B is the size of batch size, and n is the number of single agents; is the advantage function value of the current single agent, and S represents the entropy of the strategy; is a hyperparameter that controls the entropy coefficient, is the importance sampling ratio of the current strategy to the previous round strategy; represents the action output by the i-th single agent; represents the observation value of the i-th single agent; for summing variables; It is the hyperparameter that controls the boundary in the clip function, and its value is (0,1); represents the current strategy probability; represents the strategy probability of the previous round.

7. The method according to claim 3, characterized in that The loss function of the Critic network model is: ； in, is the loss function of the Actor network model; is the network parameter in the Critic network model; i represents the i-th single agent; t is the time step; is the cumulative discounted return; B is the batch size; n is the number of single agents; Indicates status The value of is the state corresponding to the i-th single agent; It is the hyperparameter that controls the boundary in the clip function, and its value is (0,1); Indicates the state value of the previous round; is the summation variable.

8. The method according to claim 3, characterized in that The calculation of the cumulative discount return includes: ； Where T is the expected step length; is the decay factor, the value range is (0,1]; t is the time step, R is the reward value in the tuple; is the cumulative discounted return; i represents the i-th single agent; j represents the index of a time step.

9. An image evaluation method based on multi-agent reinforcement learning, characterized in that: include: The ISP parameters are obtained by using the ISP parameter adjustment method based on multi-agent reinforcement learning as described in any one of claims 1 to 8; The ISP parameters and the RAW image are input into the trained multi-agent reinforcement learning model to output the final RGB image.

10. An electronic device comprising at least one processor; and a memory connected in communication with the at least one processor; characterized in that: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of claims 1 to 8.

Images