Control method of image signal processor and control device performing the method

Abstract

The present disclosure is a control method of an image signal processor for an artificial neural network, the method can be configured to include: a step of acquiring an image; a step of determining at least one image characteristic data corresponding to the image; and a step of determining an image correction parameter (SFR preset) for improving the inference accuracy of an artificial neural network model based on the at least one image characteristic data and the inference accuracy attribute of the artificial neural network model.

Description

Technical Field

[0001] This disclosure relates to a control method for an image signal processor and a control device for executing the method. Specifically, this disclosure relates to a control method for an image signal processor used in an artificial neural network and a control device for executing the method. Background Technology

[0002] Cameras can improve image quality by processing RAW image data acquired from the image sensor via an image signal processor (ISP). This ISP is typically tuned to its settings by imaging experts. Therefore, even using the same RAW image data, the image processing results can vary depending on the operator's preferences, visual perception, cognitive abilities, and so on.

[0003] On the other hand, with the recent advancements in machine learning and their application to computer vision, various techniques are being developed for detecting objects in images captured by cameras. However, if an image signal processor (ISP) with human-adjusted camera settings is used, the image data used for learning / validation / inference may be non-uniform depending on the operator's characteristics, thus limiting the improvement of the inference accuracy of artificial neural network models. In fact, the inference accuracy of artificial neural network models may even decrease.

[0004] The background description provided is intended to facilitate understanding of this disclosure. It should not be construed as an admission that the matters described in the background section are prior art. Summary of the Invention

[0005] Technical issues

[0006] To improve the detection rate of artificial neural network models, techniques for resolving the resolution of traditional RAW image data have been previously disclosed. However, these techniques have the drawback of requiring a separate processor for the artificial neural network model.

[0007] Furthermore, techniques have been previously disclosed for improving the detection rate of artificial neural network models while gradually manipulating the settings of a conventional image signal processor (ISP). However, since the characteristics of artificial neural network models are not fundamentally considered, a problem arises: the image signal processor must repeat the above operation whenever a new image is input into the artificial neural network model.

[0008] Therefore, there is a need for a method that can select the inference accuracy attribute of an artificial neural network model based on the characteristics of image data and control the parameters of an image signal processor (ISP) based on the inference accuracy attribute, as well as an apparatus for performing the method.

[0009] Therefore, the inventors of this disclosure attempt to develop a method for selecting the inference accuracy attribute of an artificial neural network model based on the characteristics of image data and controlling the parameters of an image signal processor (ISP) based on the inference accuracy attribute, as well as an apparatus for performing the method.

[0010] In particular, the inventors of this disclosure have configured a method and apparatus to enable images to be processed uniformly by providing an image signal processor (ISP) to subdivide the degree of preprocessing according to image characteristics, thereby significantly improving the mAP (mean average precision) of artificial neural network models.

[0011] The problems to be solved by this disclosure are not limited to the above-mentioned technical problems. Those skilled in the art will clearly understand other technical problems that have not been mentioned through the following description.

[0012] Technical solution

[0013] To address the aforementioned problems, an example of a control method for an image signal processor for an artificial neural network is provided according to this disclosure.

[0014] A control method for an image signal processor for an artificial neural network according to an example of the present disclosure may include: a step of acquiring an image; a step of determining at least one image characteristic data corresponding to the image; and a step of determining image correction parameters for improving the inference accuracy of the artificial neural network model based on the at least one image characteristic data and an inference accuracy attribute of at least one artificial neural network model.

[0015] According to one example of this disclosure, at least one image characteristic data may include at least one of the following: an image histogram (RGB, CbCr, Y histogram), RGB maximum value, RGB minimum value, mean and standard deviation of pixel values, sum of RGB values ​​of individual pixels (sum of color values), signal-to-noise ratio (SNR), frequency composition, and edge composition.

[0016] According to an example of this disclosure, the step of determining image correction parameters may include: analyzing an inference accuracy attribute that indicates a change in the inference accuracy of an artificial neural network model; and determining image correction parameters based on the inference accuracy attribute and image characteristic data to determine the degree of preprocessing of the image input to the artificial neural network model.

[0017] According to this disclosure, the step of analyzing the inference accuracy attribute can be a step of determining the change in the inference accuracy of an artificial neural network model based on at least one image characteristic among the image's brightness, noise, blur level, contrast, and color temperature.

[0018] According to one example of this disclosure, a change in inference accuracy can indicate the inference accuracy of an artificial neural network model that varies depending on the level of characteristics of the image.

[0019] According to an example of this disclosure, the steps of analyzing the inference accuracy properties of an artificial neural network model may include: the step of progressively modulating a reference image dataset applied to the artificial neural network model based on at least one image feature; and the step of calculating the mean average accuracy (mAP) of the artificial neural network model for multiple image datasets modulated at each level.

[0020] According to one example of this disclosure, the step of determining image correction parameters may be a step of determining at least one of a plurality of image correction parameter presets using a preset library matched with an image signal processor that processes the image.

[0021] According to one example of this disclosure, the step of determining image correction parameters may include the step of calculating a compensation function, which is used to selectively determine the image correction parameters by matching an inferred accuracy attribute with image characteristic data.

[0022] According to one example of this disclosure, the image correction parameters may correspond to the values ​​of special function registers of the image signal processor that processes the image.

[0023] According to an example of this disclosure, the control method for an image signal processor for an artificial neural network may further include: receiving a processed image based on image correction parameters from the image signal processor that processes the image; and outputting an inference result by inputting the processed image into the artificial neural network model.

[0024] According to this disclosure, the control method for an image signal processor for an artificial neural network may further include the step of identifying an image sensor and an image signal processor capable of acquiring and processing images.

[0025] According to an example of this disclosure, the step of determining image correction parameters may further include the step of correcting the compensation function used to determine the image correction parameters by controlling the shooting parameters of the image sensor.

[0026] According to one example of this disclosure, the step of determining image correction parameters may be a step of determining image correction parameters for improving the inference accuracy of multiple artificial neural network models based on the inference accuracy properties of multiple artificial neural network models stored in memory.

[0027] To address the aforementioned problems, an image processing system for artificial neural networks is provided according to another example of this disclosure.

[0028] According to one example of this disclosure, an image processing system for an artificial neural network may include: an image signal processor configured to perform image processing on an image; and a compensation unit operatively coupled to the image signal processor.

[0029] According to one example of this disclosure, the compensation unit can be configured to acquire an image, generate at least one image characteristic data corresponding to the image, obtain at least one inference accuracy attribute, and determine image correction parameters of the image signal processor based on the at least one image characteristic data and the at least one inference accuracy attribute.

[0030] According to one example of this disclosure, image characteristic data may include at least one of the following: an image histogram (RGB, CbCr, Y histogram), RGB maximum value, RGB minimum value, mean and standard deviation of pixel values, sum of RGB values ​​of individual pixels (sum of color values), signal-to-noise ratio (SNR), frequency composition, and edge composition.

[0031] According to one example of this disclosure, an image processing system for artificial neural networks may further include a neural processing unit configured to process an artificial neural network model.

[0032] According to one example of this disclosure, the compensation unit can be configured to selectively determine image correction parameters based on at least one inference accuracy attribute and image characteristic data to determine the degree of preprocessing of the image input to the artificial neural network model.

[0033] According to one example of this disclosure, at least one inference accuracy attribute may include information about the variation in inference accuracy of an artificial neural network model corresponding to at least one of the image's brightness, noise, blur level, contrast, and color temperature.

[0034] According to one example of this disclosure, the compensation unit may also include a preset library configured to control the image signal processor.

[0035] According to one example of this disclosure, the compensation unit may selectively determine at least one of a plurality of image correction parameter presets from a preset library.

[0036] According to one example of this disclosure, the image correction parameters may correspond to the special function register values ​​of the image signal processor.

[0037] According to one example of this disclosure, a neural processing unit can be configured to receive an image processed by an image signal processor, input the processed image into an artificial neural network model, and output an inference result.

[0038] According to one example of this disclosure, an image processing system for an artificial neural network may also include an image sensor capable of acquiring an image.

[0039] According to one example of this disclosure, the compensation unit can be configured to control the image sensor’s capture parameters based on at least one inferred accuracy attribute.

[0040] According to one example of this disclosure, the compensation unit can be configured to recognize an image signal processor.

[0041] According to one example of this disclosure, the neural processing unit can be configured to process the inference operations of the artificial neural network model based on the processed image and the weights of the artificial neural network model.

[0042] Specific details of other examples are included in the detailed embodiments and accompanying drawings.

[0043] Invention Effects

[0044] This disclosure selects image characteristics and inference accuracy attributes of artificial neural network models, and based on these, provides detailed control values ​​for an image signal processor (ISP) that can process images without biasing at least one attribute.

[0045] Furthermore, compared to existing techniques that simply provide image signal processor (ISP) control values ​​through gradual transformations, this disclosure reduces time and cost by numerically calculating the inference accuracy of an artificial neural network model that varies according to the level of image characteristics and providing control values ​​accordingly.

[0046] In addition, this disclosure can improve the inference accuracy of artificial neural network models by controlling the control values ​​(image correction parameters, SFR presets) of the image signal processor (ISP).

[0047] Furthermore, this disclosure provides different control values ​​(image correction parameters, SFR presets) based on the type of image sensor, image signal processor (ISP), and artificial neural network model, thereby improving inference accuracy.

[0048] The effects of this disclosure are not limited to the examples above, and this disclosure includes a variety of additional effects. Attached Figure Description

[0049] Figure 1 This is a schematic conceptual diagram illustrating an image processing system for an artificial neural network according to an example of this disclosure.

[0050] Figure 2 and Figure 3This is a schematic conceptual diagram illustrating the functions performed by various elements of an image processing system for an artificial neural network according to an example of this disclosure.

[0051] Figure 4 This is a schematic flowchart of a control method for an image signal processor according to an example of the present disclosure.

[0052] Figure 5 This is a detailed flowchart of a control method for an image signal processor according to an example of the present disclosure.

[0053] Figures 6a to 8d This is a schematic diagram illustrating a method for selecting the inference accuracy attribute of an artificial neural network model based on image characteristics, according to an example of this disclosure.

[0054] Figures 9 to 13 This is a schematic conceptual diagram illustrating the functions performed by various elements of an image processing system for an artificial neural network according to various examples of this disclosure.

[0055] Figure 14 This is a schematic conceptual diagram illustrating another example of an image processing system for an artificial neural network according to this disclosure. Detailed Implementation

[0056] The advantages and features of this disclosure, as well as methods of implementing them, will become apparent from the examples described in detail below in conjunction with the accompanying drawings. However, this disclosure is not limited to the examples disclosed below and will be implemented in various forms. These examples are provided to make this disclosure complete and to fully inform those skilled in the art to which this disclosure pertains; the scope of this disclosure is defined only by the scope of the claims. Regarding the description of the drawings, similar reference numerals may be used for similar elements.

[0057] In this document, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of a corresponding feature (e.g., an element such as a number, function, action, or component) and do not exclude the presence of additional features.

[0058] In this document, expressions such as “A or B”, “at least one of A and / or B”, or “one or more of A and / or B” can include all possible combinations of the items listed together. For example, “A or B”, “at least one of A and B”, or “at least one of A or B” can refer to all examples of (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B.

[0059] When used herein, expressions such as “first,” “second,” “first,” or “second” may modify individual elements regardless of order and / or importance, and are used only to distinguish one element from another without limiting the element. For example, “first user equipment” and “second user equipment” may represent different user equipment regardless of order or importance. For example, a first element may be named a second element without departing from the scope of the claims described herein, and similarly, a second element may be renamed a first element.

[0060] It should be understood that when an element (e.g., a first element) is referred to as being "functionally or communicatively connected (operably or communicatively coupled)" or "in contact (connected)" with another element (e.g., a second element), that element may be directly connected to other elements or may be connected through another element (e.g., a third element). On the other hand, it can be understood that when an element (e.g., a first element) is referred to as being "directly connected to" or "directly in contact with" another element (e.g., a second element), there is no other element (e.g., a third element) between that element and the other element.

[0061] The expressions “~configured to” used in this document may be used interchangeably with, for example, “~suitable for,” “~capable to,” “~designed to,” “~adapted to,” “~manufactured to,” or “~able to.” The term “configured (or configured to)” does not necessarily mean only that the hardware is “specifically designed to.” Rather, in some cases, the expression “device configured to” may mean that the device is “capable of…” by utilizing other devices or components. For example, the phrase “processors configured (or configured to perform) A, B, and C” may refer to a dedicated processor (e.g., an embedded processor) for performing the corresponding operations or a general-purpose processor (e.g., a CPU or application processor) capable of performing the corresponding operations by executing one or more software programs stored in a storage device.

[0062] The terminology used in this document is for the purpose of describing specific examples only and is not intended to limit the scope of other examples. Singular expressions may include plural expressions unless the context clearly specifies otherwise. The terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art described herein. Terms used in this document that are defined in a general dictionary may be interpreted as having the same or similar meaning as in the relevant technical context, and should not be construed as having an ideal or overly formal meaning unless explicitly defined herein. In some cases, even terms defined in this document may not be construed as excluding the examples in this document.

[0063] The features of the various examples of this disclosure can be combined or integrated in whole or in part, as will be fully understood by those skilled in the art. Various technical linkages and drives are possible, and the various examples can be implemented independently of each other or together in a related relationship.

[0064] To clearly explain this specification, the terms used herein are defined below.

[0065] In this specification, "image" can include not only a single image acquired from an image sensor, but also multiple images or videos acquired from an image sensor. More specifically, an image can include multiple images or videos acquired through multiple image sensors.

[0066] The “artificial neural network model” in this specification can be a model trained to perform inferences such as object detection, object segmentation, image / video reconstruction, image / video enhancement, object tracking, event recognition, event prediction, anomaly detection, density estimation, event search, and measurement.

[0067] For example, the artificial neural network model can be a model such as BiseNet, ShelfNet, AlexNet, DenseNet, EfficientNet, EfficientDet, GoogleNet, MnasNet, MobileNet, ResNet, ShuffleNet, SqueezeNet, VGG, YOLO, RNN, CNN, DBN, RBM, LSTM, etc. However, this disclosure is not limited to this, and various artificial neural network models trained to infer objects or the location of objects from images processed from an image signal processor (ISP) can be applied to this disclosure.

[0068] Hereinafter, an example of this disclosure will be described in detail with reference to the accompanying drawings.

[0069] Figure 1 This is a schematic conceptual diagram illustrating an image processing system for an artificial neural network according to an example of this disclosure.

[0070] Reference Figure 1 The image processing system 10 for artificial neural networks may include an image signal processor (ISP) 100, a processor 150, and a first memory 300.

[0071] In various examples, the image processing system 10 for artificial neural networks can be configured to selectively include or exclude the second memory 400.

[0072] In various examples, the image processing system 10 for artificial neural networks can be configured as a system-on-a-chip (SoC).

[0073] The image signal processor (ISP) 100 is an image processing processor and can be operatively connected to a camera module used for capturing images. The image signal processor (ISP) can process images acquired from an image sensor in various ways depending on the application.

[0074] For example, the image signal processor 100 can be configured to demosaic RAW image data with a Bayer array into the RGB color space.

[0075] For example, the image signal processor 100 can be configured to handle color space conversion from RGB color space to another color space.

[0076] For example, the image signal processor 100 can be configured to process a color correction matrix that corrects the color difference of an image according to various optical and electrical characteristics.

[0077] For example, the image signal processor 100 can be configured to process gamma correction for adjusting the gamma curve of image data.

[0078] For example, the image signal processor 100 can be configured to perform noise reduction for reducing noise in an image to reduce image artifacts.

[0079] For example, the image signal processor 100 can be configured to process edge enhancement for emphasizing edges.

[0080] For example, the image signal processor 100 can be configured to handle motion-adaptive noise reduction.

[0081] For example, the image signal processor 100 can be configured to process high dynamic range imaging (HDR) of the entire image.

[0082] For example, the image signal processor 100 may include processing modules that perform the functions described above, and each processing module may be pipelined.

[0083] However, the image processing functions of the image signal processor 100 are not limited to the functions described above.

[0084] Processor 150 may be operatively connected to image signal processor (ISP) 100, first memory 300, and second memory 400. Processor 150 may correspond to a computing device such as a central processing unit (CPU) or application processor (AP). For example, processor 150 may be a microprocessor (MPU) or a microcontroller (MCU). As another example, processor 150 may correspond to image signal processor (ISP) 100.

[0085] In various examples, the processor 150 can be implemented as an integrated chip (IC), such as a system-on-a-chip (SoC) that integrates various computing devices such as a neural processing unit (NPU), a CPU, and a GPU.

[0086] In various examples, processor 150 can function as a computing device for advanced driver assistance systems (ADAS) and also as a computing device for various object recognition application systems.

[0087] In various examples, the processor 150 can be implemented as an integrated chip that integrates various computing devices such as an ISP that receives data from a Bayer array and outputs it as YUV input or RGB input video (image), as well as a CPU.

[0088] In various examples, when processor 150 is a neural processing unit (NPU) or is configured to include a neural processing unit (NPU), processor 150 can have the computational power to process artificial neural network models (DNNs) in real time. In other words, because artificial neural network models (DNNs) have data-intensive computational characteristics, processor 150 can be configured to include an NPU for inference operations of thirty frames per second or more. However, the examples in this disclosure are not limited to NPUs.

[0089] The first memory 300 is a memory mounted on a semiconductor die and can be used to cache or store data processed in the image processing system 10 for artificial neural networks. The first memory 300 may include one of the following types of memory: ROM, SRAM, DRAM, resistive RAM, magnetoresistive RAM, phase-change RAM, ferroelectric RAM, flash memory, or HBM. The first memory 300 may include at least one memory cell. The first memory 300 can be configured as a homogeneous memory cell or a heterogeneous memory cell.

[0090] In various examples, the first memory 300 can be configured as internal memory or on-chip memory.

[0091] The second memory 400 may include one of the following types of memory: ROM, SRAM, DRAM, resistive RAM, magnetoresistive RAM, phase-change RAM, ferroelectric RAM, flash memory, and HBM. The second memory 400 may include at least one memory cell. The second memory 400 may be configured as a homogeneous memory cell or a heterogeneous memory cell.

[0092] In various examples, the second memory 400 can be configured as internal memory or on-chip memory.

[0093] The processor 150, image signal processor 100, first memory 300, and second memory 400 of the image processing system 10 for artificial neural networks can be electrically connected to communicate with each other.

[0094] The following will describe in detail the representative functions of an image processing system for artificial neural networks according to this disclosure.

[0095] Figure 2 and Figure 3 This is a schematic conceptual diagram illustrating the functions performed by various elements of an image processing system for an artificial neural network according to an example of this disclosure.

[0096] Reference Figure 2 An example of an image processing system 10 for an artificial neural network according to this disclosure can be configured to include at least a compensation unit 210.

[0097] An image processing system 10 for an artificial neural network according to an example of this disclosure can be configured to receive images from an image sensor 500. However, it is not limited thereto, and the image processing system 10 for an artificial neural network can be configured to also include an image sensor 500.

[0098] An image processing system 10 for an artificial neural network according to an example of this disclosure can be configured to provide control signals to an image signal processor 100. However, it is not limited thereto, and the image processing system 10 for an artificial neural network can be configured to further include an image signal processor 100.

[0099] An image processing system 10 for an artificial neural network according to an example of this disclosure can be configured to provide a processor 150 with a processed image for the artificial neural network. However, it is not limited thereto, and the image processing system 10 for an artificial neural network can be configured to also include a processor 150.

[0100] Reference Figure 1 and Figure 2 An example of an image processing system 10 for an artificial neural network according to this disclosure includes a compensation unit 210 and may be configured to further include at least one of an image sensor 500, an image signal processor 100, and a processor 150.

[0101] According to an example of the present disclosure, the compensation unit 210 of the image processing system 10 for artificial neural networks can be configured to communicate with the image sensor 500 and / or the image signal processor 100, respectively.

[0102] The compensation unit 210 can be configured to analyze the image to be inferred by the processor 150 in order to control the image signal processor 100.

[0103] Specifically, the analyzer and selector 210a of the compensation unit 210 can be configured to store profile information of the inference accuracy attribute of the artificial neural network model (DNN) previously analyzed by the processor 150, which can improve the inference accuracy of the artificial neural network model (DNN) processed by the processor 150.

[0104] The compensation unit 210 can be configured as a separate processor. Alternatively, the compensation unit 210 can be configured to be included in the processor 150. Alternatively, the compensation unit 210 can be configured to be included in the image signal processor 100.

[0105] The compensation unit 210 can be configured to include an analyzer and selector 210a, a multiplexer unit 210b, and a preset library 210c.

[0106] The analyzer and selector 210a can be configured to analyze characteristic data of images acquired from the image sensor 500. The analyzer and selector 210a can be configured to select inference accuracy attributes from a preset library 210c based on the characteristic data of the analyzed images, which can improve the inference accuracy of the artificial neural network model (DNN).

[0107] The preset library 210c can be configured to store multiple setting values ​​for controlling the image signal processor 100. The image signal processor 100 can be configured to receive specific setting values ​​and process images to improve the inference accuracy of the images to be processed by the processor 150. The preset library 210c can be controlled by the multiplexer unit 210b. The preset library 210c can be configured to provide the image signal processor 100 with setting values ​​selected in response to selection signals from the analyzer and selector 210a.

[0108] The multiplexer unit 210b can be configured to receive characteristic data analyzed by the analyzer and selector 210a and select control values ​​to be provided to the image signal processor 100.

[0109] The analyzer and selector 210a will be described in detail below. The analyzer and selector 210a can be configured to analyze the characteristic data of the image to be processed by the processor 150 and compare the analysis results with the inference accuracy properties of the artificial neural network model (DNN).

[0110] In other words, the inference accuracy attribute according to an example of this disclosure can refer to the inference accuracy attribute of a specific artificial neural network model (DNN). The unit of inference accuracy of the artificial neural network model (DNN) can change depending on the application of the artificial neural network model (DNN).

[0111] In other words, the inference accuracy attribute can represent the object detection rate or image classification rate of an artificial neural network model (DNN) that varies according to changes in the data characteristics of the image. The object detection rate can be expressed as the mean average precision (mAP) (%).

[0112] In other words, the inference accuracy attribute can refer to information obtained by analyzing the tendency of the inference accuracy of a specific artificial neural network model (DNN) processed by the processor 150 to change based on the changes in the data characteristics of the image input to the processor 150.

[0113] In other words, inference accuracy can refer to information that analyzes whether a specific artificial neural network model (DNN) can improve inference accuracy when the image has data with certain biases.

[0114] It should also be noted that the inference accuracy of an artificial neural network model (DNN) can vary depending on whether the weights are being trained or have already been trained.

[0115] For example, a specific artificial neural network model can be trained on a training dataset consisting of one thousand images classified into one hundred classes. In this case, the trained weights of the artificial neural network model can be changed according to variations in the characteristic data of each image in the dataset.

[0116] In other words, trained artificial neural network models can achieve high accuracy when inferring images similar to those in the trained dataset.

[0117] In other words, when a trained artificial neural network model is trained on a dataset consisting of dark images, the inference accuracy of dark images can be improved.

[0118] In other words, when a trained artificial neural network model is trained on a dataset consisting of bright images, the inference accuracy of bright images can be improved.

[0119] In other words, when a trained artificial neural network model is trained on a dataset consisting of sharp images, the accuracy of inferences about sharp images can be improved.

[0120] In other words, when a trained artificial neural network model is trained on a dataset consisting of blurred images, the accuracy of inference for blurred images can be improved.

[0121] In other words, the inference accuracy of an artificial neural network (DNN) model is affected by the similarity between the image feature data in the training dataset and the features of the image to be inferred. This trend is because artificial neural network (DNN) models are designed to infer images that are similar to those trained on.

[0122] In other words, the compensation unit 210 can improve the inference accuracy of the trained artificial neural network model by providing the image signal processor 100 with setpoints for controlling image processing.

[0123] The analyzer and selector 210a can be configured to utilize the inference accuracy properties of a trained artificial neural network model.

[0124] The inference accuracy attribute of an artificial neural network model can refer to the attribute obtained by analyzing the inference accuracy characteristics of the artificial neural network model. In other words, the inference accuracy attribute of an artificial neural network model includes the inference accuracy characteristics of the artificial neural network model analyzed previously.

[0125] Once the artificial neural network model (DNN) to be processed by processor 150 is determined, the inference accuracy attribute of the DNN can be selected. By selecting the inference accuracy attribute, data showing the variation in inference accuracy of the DNN based on the data characteristics of the image can be generated. This will be referenced later. Figures 6a to 8d To describe the choice of inference accuracy attributes.

[0126] For example, the analyzer and selector 210a can infer data such as the brightness level or noise level of the training image dataset of the artificial neural network model (DNN) by selecting the inference accuracy attribute of the DNN. Therefore, how the image signal processor 100 processes the images acquired from the corresponding image sensor 500 determines whether the accuracy of the image inferred by the processor 150 is improved.

[0127] Therefore, the compensation unit 210 can provide the image signal processor 100 with control values ​​for processing images. These control values ​​can be called special function register parameters. Special function register parameters (SFR parameters) can represent values ​​used to control various processing modules included in the image signal processor 100. The control values ​​can be provided in a preset form.

[0128] In other words, the compensation unit 210 can provide the image signal processor (ISP) 100 with one of several preset special function register parameters (SFR presets) corresponding to the inference accuracy attributes of the artificial neural network model (DNN) to be processed by the processor 150. The special function register parameters can be referred to as image correction parameters.

[0129] In other words, the compensation unit 210 can provide the image signal processor 100 with preset special function register parameters for determining the degree of preprocessing of each image to be processed by the processor 150.

[0130] The image signal processor 100 can preprocess the image based on the image correction parameters provided by the compensation unit 210.

[0131] Reference Figure 3 The image signal processor 100 can be configured to include multiple processing modules with a pipelined structure. Each processing module can be configured to perform image processing for a specific function. Image correction parameters, which are parameters of special function registers, can be input to each processing module.

[0132] You can press Figure 3 The image processing procedure of the image signal processor 100 is executed sequentially as shown. For example, the image signal processor 100 receives converted RGB image data and executes the image processing functions of each processing module. Each module can process the image based on the image correction parameters provided by the compensation unit 210. The image signal processor 100 can then transmit the processed image to the processor 150. Since the image processing of the pre-processed image is based on the inference accuracy properties of the artificial neural network model (DNN) to be processed by the processor 150, the inference accuracy can be improved.

[0133] In various examples, depending on the type of image signal processor 100, additional functions besides the image processing described in each processing module may be performed, or some functions may be removed or integrated.

[0134] Let's refer to it again. Figure 2 The processor 150 can input preprocessed images into an artificial neural network model (DNN) to perform inferences such as image classification and object recognition.

[0135] In this way, in artificial neural network models that use images processed based on image correction parameters as input, inference accuracy can be improved compared to traditional techniques.

[0136] The following will refer to Figures 4 to 8d The process of using the compensation unit 210 to control the image signal processor 100 is described in detail.

[0137] Figure 4 This is a schematic flowchart of a control method for an image signal processor according to an example of the present disclosure.

[0138] Reference Figure 4The compensation unit 210 can acquire image S110. For example, the compensation unit 210 can acquire an image provided by the image sensor 500, or receive an image from an external device via a communication interface (not shown). The image sensor 500 can be a sensor including a Bayer color filter. In this case, the image sensor 500 can perform color space conversion from RAW image data with a Bayer array to RGB, YUV, HSV, YCBCR, etc.

[0139] Reference Figure 4 The compensation unit 210 can generate at least one image characteristic data S120 corresponding to the image. Specifically, the analyzer and selector 210a of the compensation unit 210 can extract at least one of the histogram, maximum value, minimum value, mean value, standard deviation value, sum of RGB values, signal-to-noise ratio (SNR), frequency composition and edge composition of each channel of the image, and analyze the image characteristic data of the corresponding image to generate characteristic data.

[0140] In other words, the analyzer and selector 210a can analyze various image feature data that may be related to the inference accuracy among the features of the image to be input into the artificial neural network model (DNN).

[0141] Meanwhile, in order to extract various image characteristic data, the analyzer and selector 210a may include a computation unit that can perform at least one of the following operations for analyzing the aforementioned image characteristic data: histogram operation, minimum operation, maximum operation, summation operation, average operation, mean operation, standard deviation operation, convolution operation, and Fourier transform operation.

[0142] For example, the analyzer and selector 210a may be configured to include a processor configured to process operations. For example, the analyzer and selector 210a may be configured to utilize some modules of the image signal processor 100 to process operations. For example, the analyzer and selector 210a may be configured to use application-specific integrated circuits, application processors, central processing units, graphics processing units, microprocessor units, etc., to process operations.

[0143] After step S120, the compensation unit 210 may determine image correction parameters (SFR preset) S130 for improving the inference accuracy of the artificial neural network model based on at least one image feature data and the inference accuracy attribute of the artificial neural network model.

[0144] Specifically, the compensation unit 210 can receive an attribute that analyzes the variation in the inference accuracy of the artificial neural network model (DNN) to be processed by the processor 150. Here, the term variation in inference accuracy refers to the variation in the inference accuracy of the artificial neural network model (DNN) as the level of at least one image characteristic—brightness, noise, blur level, contrast, and color temperature—gradually changes.

[0145] For example, an artificial neural network model (DNN) can be a model trained to perform inferences such as image classification, object detection, object segmentation, object tracking, event recognition, event prediction, anomaly detection, density estimation, and measurement.

[0146] Here, in order to perform accurate calculations, according to an example of this disclosure, the artificial neural network model (DNN) can be a trained model. Alternatively, the artificial neural network model (DNN) can be a model that has been trained to achieve a level of inference accuracy exceeding a threshold. When additional training is provided to the neural network model, the inference accuracy attribute can be changed. Thus, when a trained artificial neural network model is provided, the inference accuracy attribute selection of the compensation unit 210 can be performed.

[0147] The following describes a method for analyzing the inference accuracy attributes of the artificial neural network model to be used by the compensation unit 210. The inference accuracy attributes of the previously analyzed artificial neural network model can be provided to the compensation unit 210. Specifically, the inference accuracy attributes of the artificial neural network model can be provided corresponding to the artificial neural network model to be processed by the processor 150.

[0148] The mean precision (mAP) is used as an example to analyze the inference accuracy property of artificial neural network models.

[0149] After the artificial neural network model (DNN) to be inferred by processor 150 is determined, inference accuracy attribute analysis can be performed. This will be discussed later. Figures 6a to 8d Describe the choice of inference accuracy attributes.

[0150] For example, the compensation unit 210 can be configured to analyze image feature data to be processed by the processor 150.

[0151] The analyzer and selector 210a can be configured to be provided with the inference accuracy properties of the artificial neural network model (DNN) being analyzed.

[0152] The analyzer and selector 210a can be configured to compare the provided inference accuracy attributes with image characteristic data to set image correction parameter values ​​that can improve the inference accuracy of the image signal processor 100.

[0153] The analyzer and selector 210a can be configured to include a library 210c, which includes multiple preset image correction parameters corresponding to the inference accuracy attributes and image characteristic data being analyzed.

[0154] In other words, the compensation unit 210 can selectively determine image correction parameters for determining the preprocessing level of the image input to the artificial neural network based on the inference accuracy attribute and the analyzed image characteristic data. Here, the image correction parameters (SFR preset) are parameters to be provided to the image signal processor 100 and may correspond to the values ​​of the special function registers of the image signal processor 100.

[0155] For example, image correction parameters (SFR presets) can be defined as special function register values ​​that can be provided to the image signal processor 100, as shown in Table 1 below. The preset library 210c shown in Table 1 can vary depending on the type of the image signal processor 100 and its identification information, and different preset libraries 210c can be stored in the compensation unit 210 or in a separate memory. For example, the preset library 210c can be stored in the first memory 300. However, it is not limited to this; the preset library 210c can be stored in the memory within the compensation unit 210.

[0156] [Table 1]

[0157]

[0158] The compensation unit 210 can determine the optimal image correction parameters from the N image correction parameters (SFR presets) shown in [Table 1] based on image characteristic data and the inference accuracy attribute of the artificial neural network model (DNN).

[0159] The compensation unit 210 transmits the determined optimal image correction parameters (SFR preset) to the image signal processor 100. Thus, the processor 150 can receive the image processed based on the image correction parameters from the image signal processor 100. For example, the processor 150 can output object inference results by inputting the processed image into an artificial neural network (DNN) model trained for object inference.

[0160] In various examples, the weights of an artificial neural network model (DNN) can be provided along with the processed image to a processor 150 for inference operations.

[0161] In various examples, compensation unit 210 can determine image correction parameters for improving the inference accuracy of multiple artificial neural network models based on the inference accuracy properties of the models stored in a specific memory. For example, the specific memory could be a first memory 300 or a second memory 400. Specifically, even artificial neural network models with the same layer structure may have different inference accuracies depending on the training image dataset. Therefore, when multiple artificial neural network models exist, compensation unit 210 can select individual image correction parameters for each of the multiple artificial neural network models to improve inference accuracy.

[0162] Here, the first artificial neural network model can be an image classification model. The second artificial neural network model can be an object recognition model. The third artificial neural network model can be an object segmentation model. The weights of each artificial neural network model may differ depending on the training data. Therefore, the inference accuracy attributes of each artificial neural network model may differ from one another. Thus, the inference accuracy attributes of each artificial neural network model can be analyzed separately. The inference accuracy attributes of each analyzed artificial neural network model can be provided to the compensation unit 210. Therefore, the compensation unit 210 can be configured to receive or store the inference accuracy attributes of at least one artificial neural network model.

[0163] The following will refer to Figure 5 Describe in detail the method for selecting the inference accuracy properties of artificial neural network models to determine the optimal parameters.

[0164] Figure 5 This is a detailed flowchart of a control method for an image signal processor according to an example of the present disclosure.

[0165] Reference Figure 5 The compensation unit 210 can identify the image sensor 500 and the image signal processor 100S210. Specifically, since the image sensor 500 and the image signal processor 100 have different image data processing characteristics according to their types, the processor 150 can identify these two devices before selecting the image correction parameters (SFR preset) of the image signal processor 100. However, this disclosure is not limited to device identification, and the device identification step can be omitted.

[0166] After step S210, processor 150 can determine the artificial neural network model (DNN) to infer the object based on the image S220. Specifically, the first memory 300 can store multiple artificial neural network models, from which the user can select one.

[0167] Following step S220, the processor 150 may progressively modulate the reference image dataset S230 of the artificial neural network model determined in the previous steps. Here, the reference image dataset may, for example, be a dataset of tens of thousands of validation images used for performance evaluation of the artificial neural network model.

[0168] The processor 150 can modulate the attributes of an image step by step based on at least one of the image characteristics, such as brightness, noise, blur level, contrast, and color temperature.

[0169] After step S230, processor 150 can calculate the mean precision (mAP) of the artificial neural network model for multiple image datasets modulated at each level, S240, and can select the attributes of the artificial neural network model based on the calculation results (S250).

[0170] Related to the above, Figures 6a to 8d This is a schematic diagram illustrating a method for selecting attributes of an artificial neural network model based on image characteristics, according to an example of this disclosure.

[0171] In reference Figures 6a to 8d Before proceeding with the description, it will be understood that some processes for selecting a reference image dataset for the artificial neural network model and analyzing attributes (e.g., creating a modulated image dataset, etc.) are performed by the processor 150 or analysis system (not shown) included in the image processing system 10 for the artificial neural network. Furthermore, since hundreds of thousands of analysis images must be inferred for the attributes, the attribute analysis task may be difficult to process in real time. Additionally, memory may be required to store the hundreds of thousands of analysis images for the attributes. Therefore, it may be efficient to perform the attribute analysis work separately beforehand and utilize only the analyzed attribute results.

[0172] Figures 6a to 6c Using image brightness to analyze the inference accuracy properties of artificial neural network models. (Refer to...) Figure 6a A reference image dataset can be prepared for generating attributes. As a usable dataset, publicly available standard datasets or datasets created for training specific features can be used.

[0173] For example, 50,000 images suitable for evaluating artificial neural network inferences can be used as a dataset. For attribute generation, six altered image datasets can be generated by progressively modulating the average brightness of the images based on a reference image dataset. One altered image dataset contains 50,000 images with different brightness levels. Therefore, the six altered datasets comprise a total of 300,000 images. Alternatively, if the reference image dataset is included, an image dataset of 350,000 images with a total of seven brightness levels can be prepared for attribute analysis.

[0174] For example, a modulated image dataset with six levels of image brightness values ​​Y can be generated by changing the brightness value Y of a reference image in the YCbCr color space within a reference image dataset in units of thirty gray levels. The unit for changing the brightness value is not limited to this and can be set by the user.

[0175] The analysis system can input various image datasets into an artificial neural network (DNN) model to obtain the mean average precision (mAP) attribute of the DNN model based on changes in average brightness. Here, the image dataset can include both modulated image datasets and reference image datasets. Attribute analysis can be performed in a separate system or processor 150.

[0176] Reference Figure 6b As a result of profiling, it can be seen that when the average brightness of the reference image dataset is increased by 30 levels (ΔY = 30), the artificial neural network model (DNN) has the highest mAP, while when the average brightness is decreased by 90 levels (ΔY = -90), the mAP is the lowest.

[0177] In other words, by referring to the results of obtaining attributes, the inference accuracy characteristics of artificial neural network models can be analyzed based on the difference in inference accuracy between the reference image dataset and the modulated image dataset. For example, referring to Figure 6b The properties of the image signal processor 100 can be used to determine which relatively brighter images are used as the training image dataset for the artificial neural network model. For example, the artificial neural network model to be processed by the processor 150 can be analyzed to have the highest inference accuracy in images that are thirty levels or more brighter. That is, the average brightness value of the fourth modulation dataset (ΔY = 30) can be determined as the optimal value. Thus, the compensation unit 210 can control the image signal processor 100 to make the average brightness value of the image to be processed by the processor 150 approximate the average brightness value of the fourth modulation dataset (ΔY = 30).

[0178] Therefore, the compensation unit 210 can be configured to control the image signal processor 100 to change the brightness of the image input to the artificial neural network model (DNN) according to the inference accuracy characteristics of the artificial neural network model. Thus, the compensation unit 210 can improve the inference accuracy of the artificial neural network model.

[0179] Reference Figure 6c The compensation unit 210 can use the preset library 210c to select the image correction parameters (SFR preset) of the image signal processor 100.

[0180] Specifically, the compensation unit 210 can calculate the average brightness of the input image based on image characteristic data. The compensation unit 210 can then select image correction parameter presets to be applied to the corresponding image from the preset library 210c based on the calculated average brightness of the image.

[0181] For example, the first brightness correction preset A1 could mean that when the average brightness level of the input image is 0 to 32, the image signal processor 100 performs a correction that increases the average brightness of the image by eighty levels (ΔY = 80). As another example, the eighth brightness correction preset A8 could mean that when the average brightness level of the input image is 224 to 255, the image signal processor 100 performs a correction that decreases the average brightness of the image by ten levels (ΔY = -10).

[0182] Thus, the compensation unit 210 can determine image correction parameters for improving the inference accuracy of the artificial neural network model by considering the inference accuracy attributes of 1. the average brightness of the input image and 2. the average brightness of the reference image dataset of the artificial neural network model. Therefore, the inference accuracy of the artificial neural network model can be improved.

[0183] More specifically, the image signal processor 100 can correct the brightness level ΔY of the image based on the image correction parameters received from the compensation unit 210. For example, the image signal processor 100 can convert RAW image data with a Bayer array to the YCbCr color space and then correct the brightness level ΔY of the image. Furthermore, the image signal processor 100 converts the RAW image data with a Bayer array to RGB for each pixel. Next, the image signal processor 100 can correct the brightness level of the image to match the YCbCr-based image correction parameters ΔY by adjusting the RGB values ​​or gamma curve values.

[0184] exist Figures 7a to 7c Image noise can be used to analyze the inference accuracy properties of artificial neural network (DNN) models. (See reference...) Figure 7a A reference image dataset can be prepared to generate inference accuracy attributes. As a usable dataset, publicly available standard datasets or datasets created for training specific features can be used.

[0185] For example, 30,000 images suitable for evaluating artificial neural network inference can be used as a dataset. To generate inference accuracy attributes, four modified image datasets can be generated by progressively modulating the noise levels of the images based on a reference image dataset. One modulated image dataset contains 30,000 images with different noise levels. Therefore, the four modulated datasets contain a total of 120,000 images. Alternatively, if the reference image dataset is included, a dataset of 150,000 images with a total of five noise levels can be prepared for attribute analysis.

[0186] More specifically, the reference image dataset used to analyze brightness attributes and the reference image dataset used to analyze noise attributes may be the same as or different from each other.

[0187] For example, by adding white noise to a reference image dataset in decibels (dB), four different modulated image datasets with various levels of noise processing can be generated. The unit for adding noise is not limited to this, and the user can specify the unit for adding the noise.

[0188] The processor 150 can input various modulated image datasets into an artificial neural network model (DNN) to obtain the mean average precision (mAP) property of the DNN model as the noise changes. Here, the image dataset can include both the modulated image dataset and the reference image dataset.

[0189] Reference Figure 7b As a result of obtaining the attributes, it can be confirmed that the artificial neural network model adds noise to the reference image dataset in units of 3dB, and the mAP decreases as the signal-to-noise ratio (SNR) decreases.

[0190] In other words, by referring to the results of the inference accuracy attribute, the inference accuracy characteristics of the artificial neural network model can be analyzed based on the difference in inference accuracy between the reference image dataset and the modulated image dataset. For example, referring to Figure 7b Based on the inference accuracy attribute, it can be estimated that clear images with low noise are preferably used as the training image dataset for the artificial neural network model. For example, the artificial neural network model (DNN) to be processed by processor 150 can be analyzed to have the highest inference accuracy in images with a relatively high signal-to-noise ratio. That is, a signal-to-noise ratio value of 30 dB or higher can be determined as the optimal value. Thus, compensation unit 210 can control image signal processor 100 to make the signal-to-noise ratio of the image to be processed by processor 150 approximately 30 dB.

[0191] Therefore, the compensation unit 210 can be configured to control the image signal processor 100 so that the signal-to-noise ratio of the image input to the artificial neural network model (DNN) can be changed according to the inference accuracy characteristics of the artificial neural network model. Thus, the compensation unit 210 can improve the inference accuracy of the artificial neural network model.

[0192] Reference Figure 7c The compensation unit 210 can use the preset library 210c to select the image correction parameters (SFR preset) of the image signal processor 100.

[0193] Specifically, the compensation unit 210 can calculate the signal-to-noise ratio of an image based on image characteristic data. The compensation unit 210 can select preset image correction parameters to be applied to the corresponding image from a preset library according to the calculated signal-to-noise ratio of the image.

[0194] For example, a first signal-to-noise ratio (SNR) preset B1 could mean that when the average SNR of the input image is 15 dB or lower, the image signal processor 100 processes the noise filter level to level 4. As another example, a third SNR preset B3 could mean that when the SNR of the input image is 25 dB, the image signal processor 100 processes the noise filter level to level 2. In this case, the noise correction characteristics can be improved as the noise filter level increases. For example, the noise filter can include, but is not limited to, low-pass filters, smoothing filters, convolution filters, etc. For example, it can be... Figure 3 Noise reduction is performed in the noise reduction and edge enhancement modules shown.

[0195] Thus, the compensation unit 210 can select image correction parameters (SFR preset) for improving the inference accuracy of the artificial neural network model by considering the inference accuracy attribute of the signal-to-noise ratio of the input image and the signal-to-noise ratio of the reference image dataset of the artificial neural network model, and provide them to the image signal processor 100. Therefore, the inference accuracy of the artificial neural network model can be improved.

[0196] More specifically, the image signal processor 100 can correct noise in the image based on image correction parameters received from the compensation unit 210. For example, the image signal processor 100 can correct noise in the image by adjusting a noise threshold.

[0197] exist Figures 8a to 8d In this study, the blur level of an image can be used to analyze the inference accuracy property of a digital neural network (DNN) model. (See reference...) Figure 8a A reference image dataset can be prepared to generate inference accuracy attributes. As a usable dataset, publicly available standard datasets or datasets created for training specific features can be used.

[0198] For example, 30,000 images suitable for evaluating artificial neural network inference can be used as a dataset. To generate inference accuracy attributes, three modified image datasets can be generated by progressively modulating the blur levels (σ) of the images based on a reference image dataset. One modulated image dataset contains 30,000 images with different blur levels. Therefore, the three modulated datasets contain a total of 90,000 images. Alternatively, if the reference image dataset is included, a dataset of 120,000 images with a total of four blur levels can be prepared for attribute analysis.

[0199] For example, the value of σ can be adjusted by applying the Gaussian filter of [Equation 1] to the reference image dataset.

[0200] [Equation 1]

[0201]

[0202] The processor 150 can generate three modulated image datasets with different blur levels by increasing the value of σ in units of two based on the blur level (σ = 0). The unit of change for the blur level is not limited to this and can be specified by the user.

[0203] In other words, the reference image datasets used to analyze brightness, noise, and blur level attributes can be the same or different from each other.

[0204] The processor 150 can input various modulated image datasets into an artificial neural network model (DNN) to obtain the mAP property of the DNN model according to the variation of the blur level. Here, the image dataset may include both the modulated image dataset and the reference image dataset.

[0205] Reference Figure 8b As a result of obtaining the attributes, it can be confirmed that the mAP of the artificial neural network model gradually decreases as the fuzziness level increases.

[0206] In other words, by referring to the results of the inference accuracy attribute, the inference accuracy characteristics of the artificial neural network model can be analyzed based on the mAP difference between the reference image dataset and the modulated image dataset. For example, referring to Figure 8bBased on the inference accuracy attribute results, it can be estimated that relatively clear images are preferably used as the training image dataset for the artificial neural network model. For example, the artificial neural network model to be processed by processor 150 can be analyzed to have the highest inference accuracy in relatively clear images. That is, a value of σ being zero can be determined as the optimal value. Thus, compensation unit 210 can select image correction parameters (SFR preset) for controlling image signal processor 100 to make the blur level σ of the image to be processed by processor 150 close to zero.

[0207] Therefore, the compensation unit 210 can be configured to control the image signal processor 100 to change the blur level of the image input to the artificial neural network model (DNN) according to the inference accuracy characteristics of the artificial neural network model. Thus, the compensation unit 210 can improve the inference accuracy of the artificial neural network model.

[0208] Reference Figure 8c The compensation unit 210 can use the preset library 210c to determine the image correction parameters (SFR preset) of the image signal processor 100.

[0209] Specifically, the compensation unit 210 can calculate the frequency composition of the image based on image characteristic data. Specifically, the compensation unit 210 can determine that an image with high frequencies is considered sharp, and an image with low frequencies is considered blurry. The compensation unit 210 can select an image correction parameter preset (SFR preset) to be applied to the corresponding image from a preset library based on the calculated frequency information.

[0210] Specifically, the compensation unit 210 can calculate the blur level based on image characteristic data. The compensation unit 210 can map blur levels corresponding to frequencies in the image. The compensation unit 210 can selectively determine image correction parameters to be applied to the corresponding image from a preset library based on the mapped blur level.

[0211] For example, the first blur preset C1 could mean that when the blur level of the input image is σ = 0, the image signal processor 100 does not perform edge enhancement. As another example, the fourth blur preset C4 could mean that when the blur level of the input image is σ = 6, the image signal processor 100 performs correction corresponding to edge enhancement level 3. In this case, edge characteristics can be improved as the level of the edge enhancement filter increases.

[0212] In this manner, the compensation unit 210 can select image correction parameters for improving the mAP of the artificial neural network model by considering the properties (mAP variation characteristics) of 1. the blur level of the input image and 2. the blur level of the reference image dataset of the artificial neural network model, and provide them to the image signal processor 100. Therefore, the inference accuracy of the artificial neural network model can be improved.

[0213] More specifically, the image signal processor 100 can correct the blur level of the image based on image correction parameters received from the compensation unit 210. For example, such as Figure 8d As shown, the image signal processor 100 can perform edge correction for emphasizing an image by adjusting the weights of the kernel filter. For example, it can... Figure 3 Edge enhancement is performed in the noise reduction and edge enhancement modules shown.

[0214] In other words, the compensation unit 210 can utilize the inference accuracy attribute of an artificial neural network model (DNN) corresponding to at least one of the image characteristics, namely brightness, noise, blur level, contrast, and color temperature. Therefore, the compensation unit 210 can selectively provide the image signal processor 100 with appropriate image correction parameters (SFR presets) S260.

[0215] In various examples of this disclosure, additional inference accuracy attributes can also be analyzed and reflected. For example, additional inference accuracy attributes can be analyzed simultaneously with the values ​​of certain image characteristics in a modulated dataset.

[0216] In various examples, the compensation unit 210 can selectively provide image correction parameters (SFR presets) using two methods.

[0217] As an example, as shown above in [Table 1], compensation unit 210 can selectively provide at least one S260-1 of a plurality of image correction parameter presets by using preset library 210c. To select the optimal image correction parameter, compensation unit 210 can be configured to analyze at least one image characteristic data from the image. Compensation unit 210 can analyze the image characteristic data for each image frame. Alternatively, compensation unit 210 can analyze the image characteristic data for each of a plurality of image frames.

[0218] As another example, compensation unit 210 can calculate compensation functions that can selectively determine various image correction parameters of image signal processor 100, and can selectively provide image correction parameters S260-2 based on the compensation functions. Here, the compensation function can be a function that can determine image correction parameters by matching the inference accuracy attribute of the artificial neural network model (DNN) with the image characteristic data of the image to be processed by processor 150. In this case, since the inference accuracy attribute of the artificial neural network model (DNN) varies depending on the training image dataset, compensation unit 210 can derive multiple compensation functions corresponding to one artificial neural network model using multiple image characteristics respectively.

[0219] In various examples, the image processing system 10 for an artificial neural network can control the shooting parameters of the image sensor 500 to correct the compensation function that determines the image correction parameters. For example, the image processing system 10 for an artificial neural network can control the signal amplification value of the image sensor 500, or control the exposure time (shutter speed), and thereby correct the compensation function.

[0220] The following will describe an example of selectively modifying the configuration of the image processing system 10 for artificial neural networks described above.

[0221] Figures 9 to 13 This is a schematic conceptual diagram illustrating the functions performed by various elements of an image processing system for an artificial neural network according to various examples of this disclosure.

[0222] Reference Figure 9 An example of an image processing system 10 for artificial neural networks according to this disclosure can be configured to include, with Figure 2 The compensation unit 210 is different from the compensation unit in the image signal processor 100a. That is, the image signal processor 100a can selectively determine image correction parameters based on image characteristic data analysis and the inference accuracy properties of artificial neural network models, and can perform the image processing described in each processing module.

[0223] Thus, the image signal processor 100 can determine the image correction parameters by integrating the compensation unit 210, thereby improving the ease of use of the image processing system 10 for artificial neural networks.

[0224] Reference Figure 10 An example image processing system 10 for an artificial neural network according to this disclosure can utilize a compensation function to selectively determine image correction parameters. In the above method, the image correction parameters to be determined based on the image properties are selected one by one from a preset library. On the other hand, Figure 10The image signal processor 100a can determine the image correction parameters through the compensation function.

[0225] The image signal processor 100a can selectively determine image correction parameters based on multiple compensation functions corresponding to each of the image's brightness, noise, blur level, contrast, and color temperature.

[0226] In various examples, the compensation function can selectively determine the image correction parameters based on function approximation or curve fitting algorithms.

[0227] In various examples, the compensation function can be implemented as a separate artificial neural network model, and in this case, it can be updated based on reinforcement learning.

[0228] Reference Figure 11 When the image processing system 10 for the artificial neural network is running in a specific environment, the image signal processor 100a can fix the image correction parameters. For example, when the image signal processor 100a is running only in a specific environment, at a specific time, and at a specific location, the image correction parameters can be fixed to optimized values.

[0229] In various examples, even in dynamic environments, the image signal processor 100a can fix the image correction parameters. For example, the image signal processor 100a can calculate an image correction parameter that has a high probability of inference accuracy among multiple image correction parameters and assign it a fixed value.

[0230] Reference Figure 12 When applied to devices requiring object recognition functions, such as autonomous driving and CCTV, the image processing system 10 for artificial neural networks may include an image sensor 500. An image signal processor 100a may be configured to control the shooting parameters of the image sensor 500. For example, the image signal processor 100a may control the signal amplification of the image sensor 500 or control the exposure time (shutter speed).

[0231] Meanwhile, in various examples, such as Figure 9 As shown, the image signal processor (ISP) operates in a preset selection mode for selectively determining image correction parameters from a preset library. Alternatively, as... Figure 10 As shown, the image signal processor (ISP) can operate in an image correction parameter (SFR) generation mode that uses a compensation function to selectively determine image correction parameters.

[0232] Reference Figure 13The processor 150 of the image processing system 10 for artificial neural networks can be implemented as a neural processing unit (NPU) 200 specifically for inference operations of an artificial neural network model (DNN). That is, the NPU 200 is configured to process the weight values ​​of the artificial neural network model stored in the first memory 300 together with the image processed by the image signal processor 100. The NPU 200 can be configured to infer image classification, object recognition, or segmentation, etc., from the processed image.

[0233] Here, an artificial neural network model refers to a network of artificial neurons that, upon receiving multiple inputs or stimuli, multiply and add weights, and transform and transmit values ​​obtained by adding additional biases through an activation function. An artificial neural network model trained in this way can be used to infer results from input data.

[0234] In one example of this disclosure, the neural processing unit (NPU) 200 may be a semiconductor implemented as an electrical / electronic circuit. The electrical / electronic circuit may represent a plurality of electronic devices (e.g., transistors, capacitors).

[0235] Specifically, the neural processing unit (NPU) 200 may include multiple processing elements (PEs) 220, an NPU memory 230, a controller 240, and a special function unit (SFU) (hereinafter referred to as a "function computing unit") 260. Each of the multiple processing elements 220, NPU memory 230, controller 240, and function computing unit 260 may be a semiconductor circuit connected to multiple transistors. Therefore, some of them may be difficult to identify and distinguish with the naked eye, but can be identified solely through computation.

[0236] For example, any circuitry in the neural processing unit (NPU) 200 of the image processing system 10 for artificial neural networks can operate in connection with multiple processing elements 220, or in connection with a controller 240. The controller 240 can be configured to perform the functions of a control unit configured to control the artificial neural network inference operations of the neural processing unit 200.

[0237] Multiple processing elements 220 can perform operations for artificial neural networks. Specifically, multiple processing elements 220 (e.g., PE1, PE2, PE3, ..., PEn, where n is a natural number) configured to compute feature maps and weights of an artificial neural network model (DNN) can be configured. For example, the multiple processing elements 220 can be configured as an (N×M) matrix (where N and M are natural numbers) according to the characteristics of the artificial neural network model (DNN), and thus can include (N×M) processing elements.

[0238] Multiple processing elements 220 can perform functions such as addition, multiplication, and accumulation required for artificial neural network operations. In other words, multiple processing elements 220 can be configured to perform multiplication and accumulation (MAC) operations. Multiple processing elements 220 can be configured to include MAC operators and / or arithmetic logic unit (ALU) operators. However, the number and configuration of multiple processing elements 220 are not limited thereto.

[0239] In detail, the multiple processing elements 220 may optionally include additional specific functional units to handle additional special functions. For example, at least one processing element may also include a batch normalization unit, an activation function unit, an interpolation unit, etc.

[0240] At the same time, although it has already been Figure 13 The description describes multiple processing elements 220 configured in an array, where operators implemented using multiple multiplier and adder trees can be arranged in parallel by permuting the MAC of one processing element. In this case, multiple processing elements can be defined as at least one processing element comprising multiple operators.

[0241] NPU memory 230 can store at least some of the feature maps and weights of an artificial neural network model (DNN) that can be inferred from multiple processing elements 220.

[0242] Here, an artificial neural network model (DNN) may include information about the structure or data locality of the artificial neural network model.

[0243] The controller 240 can control multiple processing elements 220 and NPU memory 230 based on information about the structure of the artificial neural network model or data locality information.

[0244] The function computation unit 260 can compute feature values ​​of an artificial neural network or compute functions of various network layers, such as activation function computation, normalization, and pooling. For example, the function computation unit 260 can be connected to multiple processing elements 220 to process data output from the multiple processing elements 220.

[0245] The controller 240 can be configured to control the operations of the multiple processing elements 220 used for inference operations of the neural processing unit (NPU) 200 and the read / write order of the NPU memory 230.

[0246] In various examples, controller 240 can be configured to control multiple processing elements 220 and NPU memory 230 based on information about the structure of the artificial neural network model (DNN) or data locality information.

[0247] The controller 240 can analyze the structure of the artificial neural network model to be run in the multiple processing elements 220, or it can receive information that has already been analyzed. For example, the artificial neural network data that may be included in the artificial neural network model (DNN) may include at least a portion of node data (i.e., feature maps) of each layer, layer layout data, locality information or information about the structure, information about each layer, and weight data (i.e., weight kernels) of each connection network connecting the nodes of each layer. The artificial neural network data may be stored in a memory located within the controller 240 or in the NPU memory 230. Here, the feature maps of each layer may have corresponding respective memory address values, and each weight data may have corresponding respective memory address values.

[0248] More specifically, the controller 240 can access the first memory 300 based on the structural information or data locality information of the artificial neural network model (DNN). Figure 13 The memory 300 in Figure 14 The controller 240 retrieves the address values ​​of the weight data and feature maps of the layers of the artificial neural network model (which are the same as the first memory 300 in the NPU). Therefore, the controller 240 can store the data retrieved through the first memory 300 in the NPU memory 230.

[0249] In addition, the controller 240 can schedule the operation order of the artificial neural network model to be executed by the neural processing unit 200 / the operation order of the multiple processing elements 220 based on the structural information or data locality information of the artificial neural network model (e.g., layout data of the artificial neural network layers).

[0250] Generally speaking, CPUs consider factors such as fairness, efficiency, stability, and response time when scheduling operations to maximize the amount of processing that can be performed in the same amount of time.

[0251] In contrast, since the controller 240 performs scheduling based on structural information or data locality information of an artificial neural network model, it can operate differently from the general CPU scheduling concept.

[0252] In other words, the controller 240 can run the neural processing unit 200 in a processing order determined based on structural information or data locality information of the artificial neural network model and / or structural information or data locality information of the neural processing unit 200 to be used. However, this scheduling method is not limited to information about structure or data locality.

[0253] To this end, the controller 240 can store information about the structure or data locality of the artificial neural network, and can optimize the processing of the neural processing unit (NPU) 200 by scheduling the operation order of the artificial neural network model (DNN). Thus, since the image processing system 10 for the artificial neural network infers objects through a separate neural processing unit (NPU) 200, the image inference processing speed can be further improved.

[0254] at the same time, Figure 14 This is a schematic conceptual diagram illustrating another example of an image processing system for an artificial neural network according to this disclosure.

[0255] Reference Figure 14 When used in image processing systems for artificial neural networks, such as 10 Figure 12 When applied to devices requiring object inference functions, such as autonomous driving or CCTV, the image sensor 500 may also be included.

[0256] In other words, when the image processing system 10 for artificial neural networks includes an image sensor 500, based on the inference accuracy properties of the artificial neural network model (DNN), whenever a new image is inferred, the processor 150 can immediately determine and provide image correction parameters to be provided to the image signal processor 100.

[0257] So far, various examples of an image processing system 10 for an artificial neural network according to this disclosure have been described. According to this disclosure, by providing detailed control values ​​for the image signal processor 100 based on the analysis of image characteristics and the inference accuracy properties of the artificial neural network model (DNN), images can be processed to improve the inference accuracy of the artificial neural network without shifting at least one characteristic based on the user's visual perception.

[0258] A control method for an image signal processor for an artificial neural network, according to an example of the present disclosure, may be provided. The method may include: a step of acquiring an image; a step of generating at least one image characteristic data corresponding to the image; and a step of determining image correction parameters for improving the inference accuracy of the artificial neural network model based on the at least one image characteristic data and an inference accuracy attribute of the artificial neural network model.

[0259] The steps for determining image correction parameters may include: analyzing inference accuracy properties that indicate changes in the inference accuracy of the artificial neural network model; and determining image correction parameters based on the inference accuracy properties and image characteristic data to determine the degree of preprocessing of the image input to the artificial neural network model.

[0260] The steps of analyzing inference accuracy attributes can be steps of determining the changes in inference accuracy of an artificial neural network model based on at least one of the image characteristics of the image, such as brightness, noise, blur level, contrast, and color temperature.

[0261] Changes in inference accuracy can indicate the inference accuracy of an artificial neural network model that varies depending on the level of image characteristics.

[0262] The steps for analyzing the inference accuracy properties of an artificial neural network model may include: modulating a reference image dataset applied to the artificial neural network model step by step based on at least one image feature; and calculating the mean average accuracy (mAP) of the artificial neural network model for multiple image datasets modulated at each level.

[0263] The step of determining image correction parameters can be a step of determining at least one of a plurality of image correction parameter presets using a preset library that matches the image signal processor used to process the image.

[0264] The steps of determining image correction parameters may include calculating a compensation function, which is used to selectively determine image correction parameters by matching inferred accuracy attributes with image characteristic data.

[0265] Image correction parameters can correspond to the values ​​of special function registers in the image signal processor that processes the image.

[0266] It may include the steps of receiving an image processed based on image correction parameters from an image signal processor used for image processing; and the steps of outputting inference results by inputting the processed image into an artificial neural network model.

[0267] This may include steps of identifying image sensors and image signal processors capable of acquiring and processing images.

[0268] The steps of determining image correction parameters may include correcting the compensation function used to determine the image correction parameters by controlling the image sensor's shooting parameters.

[0269] Following the identification steps described above, this method may further include a step of determining an artificial neural network model for inferring objects from images.

[0270] The steps for determining image correction parameters can be based on the inference accuracy properties of multiple artificial neural network models stored in memory to determine image correction parameters used to improve the inference accuracy of multiple artificial neural network models.

[0271] After the step of receiving the processed image, a step may be included whereby the weights of the artificial neural network model, along with the processed image, are provided to a separate processor for inference operations of the artificial neural network model.

[0272] An image processing system for an artificial neural network includes a memory, an image signal processor configured to process images stored in the memory, and a processor operatively connected to the memory and the image signal processor. The processor can acquire images, generate at least one image feature data corresponding to the images, and determine image correction parameters for improving the inference accuracy of the artificial neural network based on the at least one image feature data and inference accuracy properties of the artificial neural network model.

[0273] A control method for an image signal processor for an artificial neural network according to an example of the present disclosure may include: a step of acquiring an image; a step of determining at least one image characteristic data corresponding to the image; and a step of determining image correction parameters for improving the inference accuracy of the artificial neural network based on the at least one image characteristic data and the inference accuracy attribute of at least one artificial neural network model.

[0274] Image characteristic data may include at least one of the following: image histogram (RGB, CbCr, Y histogram), maximum RGB value, minimum RGB value, mean and standard deviation of pixel values, sum of RGB values ​​of each pixel (sum of color values), signal-to-noise ratio (SNR), frequency composition, and edge composition.

[0275] The steps for determining image correction parameters may further include: analyzing inference accuracy properties that represent changes in the inference accuracy of the artificial neural network model; and determining image correction parameters based on the inference accuracy properties and image characteristic data to determine the degree of preprocessing of the image input to the artificial neural network model.

[0276] The steps of analyzing inference accuracy attributes can be steps of determining the changes in inference accuracy of an artificial neural network model based on at least one of the image characteristics of the image, such as brightness, noise, blur level, contrast, and color temperature.

[0277] Changes in inference accuracy can indicate the inference accuracy of an artificial neural network model that varies according to the level of image characteristics.

[0278] The steps for analyzing the inference accuracy properties of an artificial neural network model may include: modulating a reference image dataset applied to the artificial neural network model step by step based on at least one image feature; and calculating the mean average accuracy (mAP) of the artificial neural network model for multiple image datasets modulated at each level.

[0279] The steps to determine the correction parameters for an image.

[0280] This step can be used to determine at least one of multiple image correction parameter presets using a preset library that matches the image signal processor used to process the image.

[0281] The steps of determining image correction parameters may include calculating a compensation function, which is used to selectively determine image correction parameters by matching inferred accuracy attributes with image characteristic data.

[0282] Image correction parameters can correspond to the values ​​of special function registers in the image signal processor that processes the image.

[0283] The control method for an image signal processor for an artificial neural network may further include: receiving a processed image based on image correction parameters from the image signal processor that processes the image; and outputting an inference result by inputting the processed image into the artificial neural network model.

[0284] The control method for an image signal processor used in an artificial neural network may also include the step of identifying an image sensor and an image signal processor capable of acquiring and processing images.

[0285] The steps of determining the image correction parameters may also include correcting the compensation function that determines the image correction parameters by controlling the image sensor's shooting parameters.

[0286] The steps for determining image correction parameters can be steps for determining image correction parameters to improve the inference accuracy of multiple artificial neural network models based on the inference accuracy attributes of multiple artificial neural network models stored in memory.

[0287] According to one example of this disclosure, an image processing system for an artificial neural network may include: an image signal processor configured to process an image; and a compensation unit operatively coupled to the image signal processor.

[0288] The compensation unit can be configured to acquire an image, generate at least one image feature data corresponding to the image, acquire at least one inference accuracy attribute, and determine the image correction parameters of the image signal processor based on the at least one image feature data and the at least one inference accuracy attribute.

[0289] Image characteristic data may include at least one of the following: image histogram (RGB, CbCr, Y histogram), maximum RGB value, minimum RGB value, mean and standard deviation of pixel values, sum of RGB values ​​of each pixel (sum of color values), signal-to-noise ratio (SNR), frequency composition, and edge composition.

[0290] Image processing systems for artificial neural networks may also include neural processing units configured to process artificial neural network models.

[0291] The compensation unit can be configured to selectively determine image correction parameters for determining the degree of preprocessing of the image input to the artificial neural network model based on at least one inference accuracy attribute and image characteristic data.

[0292] At least one inference accuracy attribute may include information about the variation in inference accuracy of an artificial neural network model corresponding to at least one of the image's brightness, noise, blur level, contrast, and color temperature.

[0293] The compensation unit may also include a preset library configured to control the image signal processor.

[0294] The compensation unit can selectively determine at least one of a plurality of image correction parameter presets from a preset library.

[0295] Image correction parameters can correspond to the values ​​of special function registers in the image signal processor.

[0296] The neural processing unit can be configured to receive an image processed by an image signal processor, input the processed image into an artificial neural network model, and output inference results.

[0297] Image processing systems for artificial neural networks may also include image sensors capable of acquiring images.

[0298] The compensation unit can be configured to control the imaging parameters of the image sensor based on at least one inferred accuracy attribute.

[0299] The compensation unit can be configured to recognize the image signal processor.

[0300] The neural processing unit can be configured to handle the inference operations of the artificial neural network model based on the processed image and the weights of the artificial neural network model.

[0301] Although examples of this disclosure have been described in more detail with reference to the accompanying drawings, this disclosure is not necessarily limited to these examples, and various modifications can be made without departing from the spirit of this disclosure. Therefore, the examples disclosed herein are for illustrative purposes and not for limiting the technical spirit of this disclosure, and the scope of the technical spirit of this disclosure is not limited to these examples. Thus, it should be understood that the foregoing examples are illustrative in all respects and not restrictive. The scope of protection of this disclosure should be interpreted by the appended claims, and all technical concepts within their equivalent scope should be understood to be included within the scope of this disclosure. The examples of this disclosure disclosed in this specification and drawings are merely for the purpose of readily explaining the technical content of this disclosure and aiding in the understanding of specific examples, and are not intended to limit the scope of this disclosure. It will be apparent to those skilled in the art that other modifications based on the technical spirit of the invention can be implemented in addition to the examples described herein.

Claims

1. A control method for an image signal processor, the control method being used to preprocess an image input to at least one trained artificial neural network model, the control method comprising the following steps: Steps to acquire an image; The step of determining at least one image feature data corresponding to the image; as well as The step of determining image correction parameters for improving the inference accuracy of the artificial neural network model based on the at least one image feature data and the inference accuracy attribute of the at least one artificial neural network model. The at least one trained artificial neural network model is configured to perform inference on the image after it has been processed based on the image correction parameters. The inference accuracy attribute is determined based on multiple image datasets, and the inference accuracy attribute indicates the variation of the inference accuracy of the artificial neural network model based on at least one image feature. The inference accuracy attribute of the artificial neural network model varies depending on the weights that have been trained.

2. The control method according to claim 1, wherein The at least one image characteristic data includes at least one of the following: image histogram, maximum RGB value, minimum RGB value, mean pixel value, standard deviation, sum of RGB values ​​of each pixel, signal-to-noise ratio (SNR), frequency composition, and edge composition.

3. The control method according to claim 1, in, The step of determining the image correction parameters further includes the following steps: The steps of analyzing the inference accuracy attribute that indicates changes in the inference accuracy of the artificial neural network model; and The step of determining the image correction parameters, based on the inference accuracy attribute and the image characteristic data, to determine the degree of preprocessing of the image input to the artificial neural network model.

4. The control method according to claim 3, in, The step of analyzing the inference accuracy attribute is as follows: determining the change in the inference accuracy of the artificial neural network model based on at least one of the image characteristics of the image, namely brightness, noise, blur level, contrast, and color temperature.

5. The control method according to claim 4, in, The change in inference accuracy indicates the inference accuracy of the artificial neural network model as it varies according to the characteristic level of the image.

6. The control method according to claim 4, in, The step of analyzing the inference accuracy attribute of the artificial neural network model further includes the following steps: The step of progressively modulating a reference image dataset applied to the artificial neural network model based on at least one image feature; and The step of calculating the mean accuracy (mAP) of the artificial neural network model for multiple image datasets at various modulation levels.

7. The control method according to claim 3, in, The step of determining the image correction parameters is as follows: using a preset library that matches the image signal processor that processes the image to determine at least one of a plurality of image correction parameter presets.

8. The control method according to claim 3, in, The step of determining the image correction parameters further includes the step of calculating a compensation function, which is used to selectively determine the image correction parameters by matching the inferred accuracy attribute with the image characteristic data.

9. The control method according to claim 1, in, The image correction parameters correspond to the values ​​of the special function registers of the image signal processor that processes the image.

10. The control method according to claim 1, The control method further includes the following steps: The step of receiving a processed image based on the image correction parameters from the image signal processor that processes the image; as well as The step of outputting inference results by inputting the processed image into the artificial neural network model.

11. The control method according to claim 1, The control method further includes the following steps: The steps of identifying the image sensor and the image signal processor capable of acquiring and processing the image.

12. The control method according to claim 1, in, The step of determining the image correction parameters further includes the step of correcting the compensation function used to determine the image correction parameters by controlling the shooting parameters of the image sensor.

13. The control method according to claim 1, in, The steps for determining the image correction parameters are as follows: determining the image correction parameters for improving the inference accuracy of the multiple artificial neural network models based on the inference accuracy attributes of the multiple artificial neural network models stored in the memory.

14. An image processing system for preprocessing an image input to at least one trained artificial neural network model, the image processing system comprising: An image signal processor configured to perform image processing on an image; as well as A compensation unit, operatively connected to the image signal processor, The compensation unit is configured to acquire the image, generate at least one image feature data corresponding to the image, obtain at least one inference accuracy attribute, and determine the image correction parameters of the image signal processor based on the at least one image feature data and the at least one inference accuracy attribute. The at least one trained artificial neural network model is configured to perform inference on the image after it has been processed based on the image correction parameters. The inference accuracy attribute is determined based on multiple image datasets, and the inference accuracy attribute indicates the variation of the inference accuracy of the artificial neural network model based on at least one image feature. The inference accuracy attribute of the artificial neural network model varies depending on the weights that have been trained.

15. The image processing system according to claim 14, in, The at least one image characteristic data includes at least one of the following: image histogram, maximum RGB value, minimum RGB value, mean pixel value, standard deviation, sum of RGB values ​​of each pixel, signal-to-noise ratio (SNR), frequency composition, and edge composition.

16. The image processing system according to claim 14, The image processing system further includes a neural processing unit configured to process artificial neural network models, and in, The compensation unit is configured to selectively determine the image correction parameters based on the at least one inference accuracy attribute and the image characteristic data, the image correction parameters determining the degree of preprocessing of the image input to the artificial neural network model.

17. The image processing system according to claim 16, in, The at least one inference accuracy attribute includes information about the variation in inference accuracy of the artificial neural network model corresponding to at least one of the brightness, noise, blur level, contrast, and color temperature of the image.

18. The image processing system according to claim 14, in, The compensation unit further includes a preset library configured to control the image signal processor, and The compensation unit is configured to selectively determine at least one of a plurality of image correction parameter presets from the preset library.

19. The image processing system according to claim 14, in, The image correction parameters correspond to the special function register values ​​of the image signal processor.

20. The image processing system according to claim 16, in, The neural processing unit is configured to receive an image processed by the image signal processor, input the processed image into the artificial neural network model, and output an inference result.

21. The image processing system according to claim 16, The image processing system also includes an image sensor capable of acquiring images, and in, The compensation unit is configured to control the image sensor’s shooting parameters based on the at least one inferred accuracy attribute.

22. The image processing system according to claim 14, in, The compensation unit is configured to recognize the image signal processor.

23. The image processing system according to claim 16, in, The neural processing unit is configured to process the inference operations of the artificial neural network model based on the processed image and the weights of the artificial neural network model.

Images