Method for adjusting parameters of image signal processor, data processing method, and product

By adjusting the image signal processor parameters to make its style consistent with the training domain data set, the performance degradation caused by device inconsistency in the actual deployment of the deep learning model is solved, and the effect of improving inference performance and saving costs is achieved.

WO2024230542A1PCT designated stage expired Publication Date: 2024-11-14SHANGHAI CAMBRICON INFORMATION TECH CO LTD

Patent Information

Application Number
PCT/CN2024/090486
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-05
Filing Date
2024-04-29
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

In actual deployment, the inference performance of deep learning models is degraded due to the inconsistency between the lens and image sensors and training. The existing solutions require retraining the model, which is time-consuming and labor-intensive.

Method used

By obtaining the sample data set and training domain data set collected by the target lens and image sensor, the learnable image signal processing model is trained, and the image signal processor parameters are adjusted to make its style consistent with the training domain data set, thereby improving the model inference performance .

Benefits of technology

There is no need to retrain deep learning models, saving time and labor costs, and improving the inference performance of deep learning models in real deployment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024090486_14112024_PF_FP_ABST
    Figure CN2024090486_14112024_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed in the present disclosure are a method for adjusting parameters of an image signal processor, a data processing method, and a product. An electronic device of the present disclosure may be comprised in a computing processing device (602) of a combined processing device. The computing processing device may comprise one or more computing devices (610). The combined processing device may further comprise an interface device (604) and another processing device (606). The computing processing device (602) interacts with the another processing device (606) to jointly complete a computing operation specified by a user. The combined processing device may further comprise a storage device (608). The storage device (608) is separately connected to the computing processing device (602) and the another processing device (606), and is used for storing data of the computing processing device (602) and the another processing device (606). According to the solution of the present disclosure, data styles of learnable image signal processing models can be normalized to data styles of training domain data sets, thereby facilitating the improvement of the actual inference performance of a target network model.
Need to check novelty before this filing date? Find Prior Art

Description

Method for adjusting image signal processor parameters, data processing method and product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed on May 5, 2023, with application number 202310499018.X and entitled “Method for adjusting image signal processor parameters, data processing method and product”. Technical Field

[0003] The present disclosure generally relates to the field of image processing technology. More specifically, the present disclosure relates to a method for adjusting image signal processor parameters, a data processing method, an electronic device, and a computer-readable storage medium. Background Art

[0004] Current mainstream computer vision algorithms are typically deep learning models trained on large amounts of data. These deep learning models typically assume that the data used during training and inference have the same image style. However, this is not always the case in practice. In scenarios where deep learning models perform computer vision tasks based on image data output by an image signal processor ("ISP"), if the lenses, image sensors, and image signal processors ("ISP") used in actual deployments differ from those used during deep learning model training, the inference data style of the deep learning model will differ from the training data style, which will negatively impact the actual inference performance of the deep learning model.

[0005] In view of this, there is an urgent need to provide a solution that is conducive to improving the reasoning performance of deep learning models in actual deployment scenarios.

[0006] Summary of the Invention

[0007] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes, in multiple aspects, a method for adjusting image signal processor parameters, a data processing method, an electronic device, and a computer-readable storage medium.

[0008] In a first aspect, the present disclosure provides a method for adjusting parameters of an image signal processor, comprising: obtaining a sample data set formed by raw image data collected via a target lens and a target image sensor; obtaining a training domain data set for training a target network model; training a learnable image signal processing model using the sample data set and the training domain data set; and configuring the parameters of the trained learnable image signal processing model to an image signal processor.

[0009] In a second aspect, the present disclosure provides a data processing method, comprising: obtaining raw data to be processed collected via a target lens and a target image sensor; and processing the raw data to be processed using an image signal processor after adjusting parameters using the method for adjusting image signal processor parameters described above and in multiple embodiments below to obtain target image data.

[0010] In a third aspect, the present disclosure provides an electronic device comprising: a processor for executing program instructions; and a memory storing the program instructions, wherein when the program instructions are loaded and executed by the processor, the processor executes the method for adjusting image signal processor parameters described in the foregoing text and in multiple embodiments below, or the data processing method described in the foregoing text and in multiple embodiments below.

[0011] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by one or more processors, implement the method for adjusting image signal processor parameters as described above and in multiple embodiments below, or the data processing method as described above and in multiple embodiments below.

[0012] Through the scheme briefly described above in the present disclosure, a learnable image signal processing model can be trained by using a sample dataset and a training domain dataset collected through a target lens and a target sensor, so as to normalize the data style of the learnable image signal processing model to the data style of the training domain dataset, thereby facilitating improving the actual reasoning performance of the target network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an illustrative and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0014] FIG1 shows a flow chart of a method for adjusting image signal processor parameters according to an embodiment of the present disclosure;

[0015] FIG2 shows a schematic flow chart of a method including similarity evaluation according to an embodiment of the present disclosure;

[0016] FIG3 shows a flow chart of a method for similarity evaluation based on image feature distribution according to an embodiment of the present disclosure;

[0017] FIG4 shows a flow chart of a data processing method according to an embodiment of the present disclosure;

[0018] FIG5 shows a schematic block diagram of an electronic device according to an embodiment of the present disclosure;

[0019] FIG6 shows a structural diagram of a combined processing device according to an embodiment of the present disclosure; and

[0020] FIG7 shows a schematic structural diagram of a board according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of this disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this disclosure, not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this disclosure.

[0022] It should be understood that the terms “include” and “comprising” used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0023] It should also be understood that the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the disclosure. As used in this disclosure and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should be further understood that the term "and / or" as used in this disclosure and the claims refers to any and all possible combinations of one or more of the associated listed items, including and including these combinations.

[0024] As used in this specification and claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0025] The inventors have discovered that when faced with the problem of inconsistent feature distributions between training data and inference data for deep learning models based on computer vision, one solution is to enhance the cross-domain robustness of the deep learning model by training the deep learning model to learn domain-invariant features. However, when different lenses, image sensors, or image signal processors are used in actual deployment scenarios, the feature distribution of the inference domain data will change. According to the above solution, the deep learning model needs to be retrained to ensure its inference performance. If the deep learning model is retrained, it will consume a lot of time and manpower costs.

[0026] Based on the above findings, unlike the above-mentioned solution of retraining the deep learning model to ensure its inference performance in actual deployment, the inventors have proposed a method of adjusting the parameters on the image signal processor side to effectively improve the inference performance of the deep learning model in actual deployment. Because the solution described in this disclosure does not require retraining the deep learning model, it helps save the time and manpower costs consumed by repeatedly training the deep learning model. The specific implementation methods of this disclosure are described in detail below with reference to the accompanying drawings.

[0027] FIG1 shows a flow chart of a method for adjusting image signal processor parameters according to an embodiment of the present disclosure. As shown in FIG1 , method 100 may include: in step 101, a sample data set formed by raw image data collected by a target lens and a target image sensor may be obtained. The raw image data is in RAW (Raw Image Format) data format. The sample data set may include one or more raw image data. The target lens and the target sensor may be used in conjunction with each other to collect raw image data. Due to the differences in the models, configurations, etc. of lenses and image sensors used by different manufacturers or in different scenarios, the lenses and image sensors at different deployment ends may have different image styles. The target lens and / or target image sensor in the embodiment of the present disclosure may select one or more lenses and / or one or more image sensors used by the target network model in actual deployment as needed.

[0028] In some preferred embodiments, the raw image data in the sample data set can cover a variety of deployment scenarios of the target lens and target image sensor. For example, a sample data set formed by multiple raw image data collected by the target lens and target image sensor in deployment scenarios such as different times of the day, different brightness, different weather, and different road sections is conducive to improving the generalization of the image signal processor after adjusting the parameters.

[0029] Then, in step 102, a training domain data set for training the target network model can be obtained. In some embodiments, the target network model may include a deep learning model based on computer vision tasks. Computer vision ("CV") refers to the use of cameras and electronic devices to replace the human eye to identify, track, and measure targets, and further perform graphics processing so that the electronic devices process the images into images that are more suitable for human eye observation or transmission to instruments for detection. In some embodiments, the deep learning model based on computer vision tasks (referred to as CV model) may include one or more target detection models, target positioning models, target segmentation models, classification models, etc. for application scenarios such as autonomous driving, road monitoring, and face recognition. The training domain data set is the training data used to train the target network model. The style of the training domain data set determines the data style that the target network model can learn, and also determines the data style that the trained target network model is good at reasoning.

[0030] The process can then proceed to step 103, where the learnable image signal processing model can be trained using the sample dataset and the training domain dataset. In some embodiments, the training domain dataset can be used as a target domain to train the learnable image signal processing model (referred to as the learnable ISP), which can serve as a reference target during the training process of the learnable image signal processing model. In other embodiments, the learnable image signal processing model can be trained in an unsupervised manner.

[0031] In some embodiments, before training the learnable image signal processing model, method 100 may further include: generating a learnable image signal processing model based on an image signal processor. The image signal processor may include an intelligent image signal processing unit ("AI ISP"). A complete imaging system may include a lens, an image sensor, and a back-end image signal processor. Therefore, the image signal processor performing parameter adjustment in this embodiment may be an image signal processor used in conjunction with the aforementioned target lens and target image sensor, or may be an image signal processor used in actual deployment of the target network model.

[0032] In other embodiments, a learnable image signal processing model can be generated by acting as a proxy for an image signal processor. Specifically, a neural network (e.g., a convolutional neural network) can be used to model the image signal processor. This neural network is trained to fit the processing flow of the image signal processor. The fitted neural network can then act as a proxy for the functions of the image signal processor, thereby serving as a learnable image signal processing model.

[0033] In some preferred embodiments, generating a learnable image signal processing model based on an image signal processor may include: performing differentiable processing on the image signal processor to obtain a learnable image signal processing model. The differentiable processing may include using a deep learning framework such as Pytorch or Tensorflow to perform differentiable implementation of the image signal processor to obtain a learnable image signal processing model. Compared with the proxy image signal processor method, the learnable image signal processing model obtained by differentiable processing can maintain a high degree of consistency with the hardware ISP (i.e., image signal processor), will not be affected by changes in the style of the input data, and can also avoid the problems that may exist in the aforementioned proxy image signal processor method, such as the image signal processor having too many parameters and complex functions, which makes it difficult to implement the proxy and has poor generalization capabilities.

[0034] Furthermore, after the learnable image signal processing model is generated, the similarity of image data output by the image signal processor and the learnable image signal processing model under the same input and configuration can be verified to evaluate the compatibility between the learnable image signal processing model and the image signal processor. After the similarity between the learnable image signal processing model and the image signal processor is evaluated to meet a preset standard, the learnable image signal processing model can be trained in step 103.

[0035] As further shown in FIG1 , in step 104, the parameters of the trained learnable image signal processing model can be configured to the image signal processor. Specifically, the parameters of the trained learnable image signal processing model can be exported and then configured to the image signal processor hardware for use, thereby completing the parameter adjustment of the image signal processor.

[0036] It can be understood that the image signal processor after adjusting the parameters can normalize the style of the data collected by the target lens and target sensor to the style of the training domain dataset of the target network model, so that the target network model can still perform reasoning based on the data style that it is good at reasoning during actual deployment, thereby improving the reasoning performance of the target network model during actual deployment.

[0037] The method for adjusting image signal processor parameters according to an embodiment of the present disclosure is described above in conjunction with FIG1 . It will be understood that the above description is exemplary and non-restrictive. For example, in some embodiments, training a learnable image signal processing model using a sample dataset and a training domain dataset may include: inputting the sample dataset into the learnable image signal processing model to obtain an inference domain dataset; and performing a similarity evaluation on the inference domain dataset and the training domain dataset, thereby adjusting the parameters of the learnable image signal processing model based on the similarity evaluation results. This will be explained below in conjunction with FIG2 .

[0038] FIG2 is a schematic diagram illustrating a method flow diagram including similarity evaluation according to an embodiment of the present disclosure. As will be apparent from the following description, process 200 shown in FIG2 may be a specific embodiment of method 100 described above in conjunction with FIG1 . Therefore, the description of method 100 in conjunction with FIG1 may also be applicable to the following description of process 200.

[0039] As shown in Figure 2, process 200 may include: obtaining a sample dataset 201 captured via a target lens and a target image sensor; then, inputting sample dataset 201 into a learnable image signal processing model 202 to obtain an inference domain dataset 203. In some embodiments, inference domain dataset 203 may include inference image data in RGB (Red, Green, Blue) data format, where each inference image data piece may be generated from a corresponding piece of original image data in sample dataset 201. Process 200 may also include: obtaining a training domain dataset 204 for training the target network model.

[0040] After obtaining the reasoning domain dataset 203 and the training domain dataset 204, the process 200 may further include: performing a similarity evaluation 205 on the reasoning domain dataset 203 and the training domain dataset 204. The similarity evaluation 205 may be an evaluation of the reasoning domain dataset 203 and the training domain dataset 204 in terms of the dataset dimension. By performing the similarity evaluation 205, the learnable image signal processing model 202 may be trained based on the similarity evaluation results. For example, the parameters of the learnable ISP may be adjusted based on the similarity evaluation results. After one or more parameter optimizations, the learnable ISP may reduce or even eliminate the domain difference between the reasoning domain dataset 203 and the training domain dataset 204, so that the image style of the generated reasoning domain dataset 203 is highly similar to the image style of the training domain dataset 204, thereby enabling the trained learnable ISP to generate reasoning data that is more consistent with the style of the training domain dataset 204.

[0041] In some embodiments, performing similarity evaluation 205 on the inference domain dataset 203 and the training domain dataset 204 may include similarity evaluation based on a domain classification network. In other embodiments, the domain classification network may include a classification network such as a VGG network model or a ResNet network model, or may include other achievable network structures for image classification. In still other embodiments, the last fully connected layer of the classification network may be modified to a binary classification structure to serve as the domain classification network.

[0042] Specifically, a generative adversarial network (GAN) can be formed based on the learnable image signal processing model 202 and the domain classification network. In the GAN, the learnable image signal processing model 202 is used to generate the reasoning domain dataset 203, and the domain classification network is used to discriminate input data to determine whether the input data belongs to the reasoning domain dataset 203 or the training domain dataset 204.

[0043] Then, the learnable image signal processing model 202 and the domain classification network can be alternately trained, wherein the domain classification network is discriminatively trained using the inference domain dataset 203 and the training domain dataset 204. That is, the inference domain dataset 203 and the training domain dataset 204 can be input into the domain classification network to evaluate the similarity between the inference domain dataset 203 and the training domain dataset 204. During the alternating training, the network parameters of one of the learnable image signal processing model 202 and the domain classification network can be fixed, while the network parameters of the other can be updated. This alternating iteration continues until the learnable image signal processing model 202 can capture the feature distribution of the training domain dataset 204, that is, a inference domain dataset 203 with a style that is highly consistent with the training domain dataset 204 can be generated.

[0044] Furthermore, during alternating training, other optimization strategies, techniques, and loss functions of the generative adversarial network can be used to allow the learnable image signal processing model 202 and the domain classification network to compete with each other and be further optimized. For example, the inference domain dataset 203 and the training domain dataset 204 can be labeled separately to distinguish whether the data input to the domain classification network comes from the inference domain dataset 203 or the training domain dataset 204, so as to facilitate training of the domain classification network. For example, loss functions such as, but not limited to, binary cross-entropy (BCE) can be used to train and optimize the learnable image signal processing model 202. After training convergence, the parameters of the learnable image signal processing model 202 can be exported and configured to the image signal processor.

[0045] It can be understood that when discriminative training is performed on the domain classification network using the inference domain dataset 203 and the training domain dataset 204, the easier it is for the domain classification network to discriminate the source of input data, the lower the similarity between the inference domain dataset 203 and the training domain dataset 204; the more difficult it is for the domain classification network to discriminate the source of input data, the higher the similarity between the inference domain dataset 203 and the training domain dataset 204. Therefore, when alternately training the learnable image signal processing model 202 and the domain classification network, discriminative training of the domain classification network using the inference domain dataset 203 and the training domain dataset 204 can evaluate the similarity between the inference domain dataset 203 and the training domain dataset 204, and the parameters of the learnable image signal processing model 202 can be optimized based on this similarity evaluation result. The similarity evaluation result here can include the probability that the input data belongs to the inference domain dataset 203 and the training domain dataset 204, that is, it reflects the difficulty of the domain classification network in discriminating the input data.

[0046] The above description of the method including similarity evaluation according to the embodiment of the present disclosure is provided in conjunction with FIG2 . It is understood that the above description is exemplary and non-restrictive. For example, similarity evaluation of the inference domain dataset and the training domain dataset may not be limited to the aforementioned domain classification network-based similarity evaluation system. Other similarity evaluation methods for optimizing the learnable image signal processing model may also be used, such as similarity evaluation methods based on image feature distribution. An exemplary description will be provided below in conjunction with FIG3 .

[0047] FIG3 is a flowchart illustrating a method for similarity evaluation based on image feature distribution according to an embodiment of the present disclosure. As will be apparent from the following description, method 300 is a specific embodiment of similarity evaluation 205 described above in conjunction with FIG2 . Therefore, the description of similarity evaluation 205 in conjunction with FIG2 also applies to the following description.

[0048] As shown in FIG3 , method 300 may include: in step 301, image features of each inference image data in the inference domain dataset may be extracted to generate a first image feature distribution of the inference domain dataset. In some embodiments, the image features may include at least one of the following: image brightness; image contrast; image sharpness; and image noise intensity. Image brightness refers to the brightness of an image. Image contrast refers to the measurement of the different brightness levels between the brightest white and the darkest black in the light and dark areas of an image, that is, the magnitude of the grayscale contrast of an image. A larger difference range represents a greater image contrast, and a smaller difference range represents a smaller image contrast. Image sharpness, sometimes also referred to as clarity, is an indicator of the clarity of the image plane and the sharpness of the image edges. The higher the image sharpness, the higher the contrast of the details in the image, making it appear clearer. Image noise intensity refers to the intensity of unnecessary or redundant interference information present in the image data.

[0049] In other embodiments, generating the first image feature distribution of the inference domain dataset can be achieved by statistically analyzing the distribution of image features of each inference image data. The first image feature distribution can include the distribution of one or more image features of each inference image data in the inference domain dataset. The first image feature distribution can include, but is not limited to, being presented in the form of distribution graphs such as histograms, curve graphs, and pie charts. In still other embodiments, analyzing the distribution of image features of each inference image data can include analyzing the distribution of at least one of the following: the average value, the maximum value, the minimum value, and the distance between the maximum and minimum values ​​of the image features of each inference image data.

[0050] For example, taking the average value of statistical image features and the image feature being image brightness as an example, in step 301, the average value of the image brightness of each inference image data in the inference domain dataset can be extracted, and then the average value of the image brightness of each inference image data can be used as a distribution point to statistically calculate the distribution of the average value of the image brightness of all inference image data in the inference domain dataset, thereby forming a first image feature distribution.

[0051] As further shown in FIG3 , method 300 may further include extracting image features of each training image data in the training domain dataset in step 302 to generate a second image feature distribution for the training domain dataset. The image features of the training image data here may be the same as or similar to the image features of the inference image data described in step 301 and are not further described here.

[0052] In other embodiments, generating the second image feature distribution of the training domain dataset can be achieved by statistically analyzing the distribution of image features of each training image data. The second image feature distribution may include the distribution of one or more image features of each training image data in the training domain dataset. The second image feature distribution may include, but is not limited to, being presented in the form of distribution graphs such as histograms, curve graphs, and pie charts. In still other embodiments, analyzing the distribution of image features of each training image data may include analyzing the distribution of at least one of the following: the average value, the maximum value, the minimum value, and the distance between the maximum and minimum values ​​of the image features of each training image data.

[0053] Then, the process can proceed to step 303, and the similarity evaluation can be performed on the first image feature distribution and the second image feature distribution. In some embodiments, the image feature types and statistical methods selected by the first image feature distribution and the second image feature distribution can be the same to facilitate similarity evaluation. In other embodiments, the evaluation indicators of the similarity evaluation may include, for example, chi-square similarity and / or Euclidean distance. That is, the similarity evaluation result between the reasoning domain dataset and the training domain dataset can be determined by calculating the chi-square similarity and / or Euclidean distance between the first image feature distribution and the second image feature distribution. The higher the chi-square similarity between the first image feature distribution and the second image feature distribution, and the smaller the Euclidean distance, the more similar the first image feature distribution and the second image feature distribution are, which also means that the reasoning domain dataset and the training domain dataset are more similar.

[0054] Based on this, a corresponding loss function can be designed to optimize the parameters of the learnable image signal processing model. For example, when using Euclidean distance to evaluate the similarity between the first image feature distribution and the second image feature distribution, the Euclidean distance loss function can be directly used to optimize the learnable image signal processing model. For another example, when using chi-square similarity to evaluate the similarity between the first image feature distribution and the second image feature distribution, a loss function can be designed based on the negative or inverse of the chi-square similarity.

[0055] The above description of the similarity evaluation method according to the embodiment of the present disclosure is illustrative and non-restrictive, in conjunction with FIG3 . For example, based on the technical concept disclosed in the present disclosure, other evaluation indicators other than chi-square similarity and Euclidean distance can also be used for similarity evaluation as needed, and this is not limited herein.

[0056] This disclosure also provides a data processing method, which will be described below in conjunction with FIG4 . FIG4 illustrates a flow chart of a data processing method according to an embodiment of this disclosure. As shown in FIG4 , method 400 may include: In step 401, raw data to be processed may be acquired via a target lens and a target image sensor. The raw data to be processed is image data in RAW format. The acquired raw data to be processed may be a single image data or multiple image data.

[0057] Then, in step 402, the image signal processor whose parameters have been adjusted according to any of the methods described above in combination with Figures 1 to 3 can be used to process the raw data to be processed to obtain target image data. The target image data can be in RGB data format. The target image data obtained according to the embodiment of the present disclosure can have the same or similar feature distribution as the training domain data set of the target network model deployed downstream of the image signal processor. Based on an input piece of raw data to be processed, the image signal processor can process and generate a corresponding piece of target image data; based on multiple input pieces of raw data to be processed, the image signal processor can process and generate corresponding multiple pieces of target image data.

[0058] In other embodiments, method 400 may further include: performing visual task processing on target image data using a target network model. Visual tasks may include, for example, one or more tasks such as target detection, target recognition, segmentation, and classification. It is understandable that the learnable image signal processing model trained by any of the methods described in Figures 1 to 3 can learn ISP parameters close to the training data style of the target network model, so that the image signal processor after parameter adjustment can normalize the original image data collected by different lenses and image sensors used in actual deployment scenarios to the data style of the training domain dataset, so that the target network model can obtain better reasoning performance in actual deployment scenarios without retraining.

[0059] Through the above description, those skilled in the art can understand that the disclosed embodiment uses a sample data set and a training domain data set to train a learnable image signal processing model, which can enable the learnable image signal processing model to learn the image style of the training domain data set, and then enable the image signal processor after adjusting the parameters to normalize the RAW images collected by different target lenses and / or different target image sensors, which is conducive to improving the reasoning performance of the target network model in different deployments. It can be understood that the solution of the disclosed embodiment starts from the data itself, rather than from the perspective of training the CV model to improve its reasoning performance, so there is no need to repeatedly train the CV model, saving a lot of time and labor costs.

[0060] Furthermore, in some embodiments, by performing similarity evaluation on the inference domain dataset and the training domain dataset, the parameters of the learnable image signal processing model can be optimized based on the similarity evaluation results to optimize the parameters of the image signal processor, so that the target image data generated by the image signal processor after parameter optimization has a high degree of consistency in style with the training data of the target network model.

[0061] Based on the above description, those skilled in the art can also understand that the present disclosure also discloses an electronic device, which will be exemplarily described below with reference to FIG5 .

[0062] FIG5 shows a schematic block diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG5 , the electronic device 500 may include a processor 510 that can be used to execute program instructions. The processor 510 may be a general-purpose processor, a dedicated processor, or a combination of the two. Furthermore, the electronic device 500 disclosed in the present disclosure may also include a memory 520 that can store program instructions. When the program instructions are loaded and executed by the processor 510, the processor 510 executes the method for adjusting image signal processor parameters described in any of FIG1-FIG3 or the data processing method described in FIG4 .

[0063] In some implementation scenarios, electronic device 500 may include a device with image processing capabilities, such as a computer, a personal computer (PC), an image analyzer, or other terminal devices. In practical applications, electronic device 500 may consist of a single device with the aforementioned multiple functions, or a combination of multiple devices with partial functions. The disclosed solution does not limit the structural design of electronic device 500.

[0064] In response to the aforementioned program instructions, the present disclosure also discloses a computer-readable storage medium on which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the method for adjusting image signal processor parameters as described in any one of Figures 1-3 or the data processing method as described in the above text in combination with Figure 4 is implemented.

[0065] FIG6 shows a structural diagram of a combined processing device according to an embodiment of the present disclosure. The combined processing device 600 may implement or include the electronic device disclosed above. As shown in FIG6 , the combined processing device 600 may include a computing processing device 602, an interface device 604, other processing devices 606, and a storage device 608. Depending on different application scenarios, the computing processing device 602 may include one or more computing devices 610, which may be configured to perform various computing operations, including but not limited to various operations involved in machine learning in the field of artificial intelligence, such as various operations in the learnable image signal processing model disclosed herein.

[0066] In different embodiments, the computing processing device 602 of the present disclosure can be configured to perform user-specified operations. In an exemplary application, the computing processing device 602 can be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Similarly, one or more computing devices 610 included in the computing processing device can also be implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core. When multiple computing devices 610 are implemented as artificial intelligence processor cores or partial hardware structures of artificial intelligence processor cores, the computing processing device 602 of the present disclosure can be regarded as having a single-core structure or a homogeneous multi-core structure.

[0067] In exemplary operation, the computing processing device 602 of the present disclosure can interact with other processing devices 606 through the interface device 604 to jointly complete the operation specified by the user. Depending on the implementation, the other processing devices 606 of the present disclosure may include one or more types of processors in general and / or special processors such as a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence processor, etc. These processors may include but are not limited to a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As previously mentioned, only with respect to the computing processing device 602 of the present disclosure, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing processing device 602 and other processing devices 606 are considered together, the two can be regarded as forming a heterogeneous multi-core structure.

[0068] In one or more embodiments, the other processing device 606 can serve as an interface between the computing device 602 (which can be embodied as an artificial intelligence computing device such as a neural network computing device) and external data and control, performing basic control including but not limited to data transfer, starting and / or stopping the computing device, etc. In other embodiments, the other processing device 606 can also collaborate with the computing device 602 to jointly complete computing tasks.

[0069] In one or more embodiments, the interface device 604 can be used to transmit data and control instructions between the computing and processing device 602 and other processing devices 606. For example, the computing and processing device 602 can obtain input data from other processing devices 606 via the interface device 604 and write the data to the storage device (or memory) on the computing and processing device chip. Furthermore, the computing and processing device 602 can obtain control instructions from other processing devices 606 via the interface device 604 and write the data to the control cache on the computing and processing device chip. Alternatively or optionally, the interface device 604 can also read data from the storage device of the computing and processing device and transmit it to the other processing device.

[0070] Additionally or alternatively, the combined processing device 600 of the present disclosure may further include a storage device 608. As shown in the figure, the storage device 608 is connected to the computing processing device 602 and the other processing device 606. In one or more embodiments, the storage device 608 may be used to store data of the computing processing device 602 and / or the other processing device 606. For example, the data may be data that cannot be fully stored in the internal or on-chip storage devices of the computing processing device or the other processing device.

[0071] In some embodiments, the present disclosure further discloses a chip (e.g., chip 702 shown in FIG7 ). In one implementation, the chip is a system-on-chip (SoC) and is integrated with one or more combined processing devices 600 as shown in FIG6 . The chip can be connected to other related components through an external interface device (e.g., external interface device 706 shown in FIG7 ). The related components can be, for example, a camera, a display, a mouse, a keyboard, a network card, or a Wifi (Wireless Fidelity) interface. In some application scenarios, other processing units (e.g., video codecs) and / or interface modules, such as a DRAM (Dynamic Random Access Memory) interface, etc., can be integrated on the chip. In some embodiments, the present disclosure further discloses a chip packaging structure, which includes the above-mentioned chip. In some embodiments, the present disclosure further discloses a board card, which includes the above-mentioned chip packaging structure. The board card will be described in detail below in conjunction with FIG7 .

[0072] FIG7 shows a schematic diagram of the structure of a board according to an embodiment of the present disclosure, which may be arranged with or include an electronic device as disclosed in the present disclosure. As shown in FIG7 , the board 700 may include a storage device 704 for storing data, which includes one or more storage units 710. The storage device 704 may be connected to and transmit data with a control device 708 and the chip 702 described above by, for example, a bus. Further, the board 700 may also include an external interface device 706, which is configured to relay or transfer data between the chip (or the chip in the chip package structure) and an external device 712. For example, the data to be processed may be transmitted to the chip 702 by the external device via the external interface device 706. For another example, the calculation result of the chip 702 may be transmitted back to the external device 712 via the external interface device 706. Depending on different application scenarios, the external interface device 706 may have different interface forms, for example, it may adopt a standard PCIE (Peripheral Component Interconnect Express) interface.

[0073] In one or more embodiments, the control device 708 in the disclosed board 700 may be configured to regulate the state of the chip 702. To this end, in one application scenario, the control device 708 may include a microcontroller unit (MCU) for regulating the working state of the chip 702.

[0074] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will understand that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teachings of the present disclosure, those skilled in the art will understand that some of the steps therein can be performed in other orders or simultaneously. Further, those skilled in the art will understand that the embodiments described in the present disclosure can be regarded as optional embodiments, that is, the actions or modules involved therein are not necessarily necessary for the implementation of one or more schemes of the present disclosure. In addition, depending on the different schemes, the description of some embodiments of the present disclosure also has different emphases. In view of this, those skilled in the art will understand that the parts that are not described in detail in a certain embodiment of the present disclosure may also refer to the relevant descriptions of other embodiments.

[0075] In terms of specific implementation, based on the disclosure and teachings of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented in other ways not disclosed herein. For example, with respect to the various devices or units in the electronic device or apparatus embodiments described above, this document divides them based on the consideration of logical functions, and there may be other ways of division in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. With respect to the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0076] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network elements. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the solution described in the embodiments of this disclosure. In addition, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically separately.

[0077] In some implementation scenarios, the above-mentioned integrated unit can be implemented in the form of a software program module. If implemented in the form of a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable memory. Based on this, when the scheme of the present disclosure is embodied in the form of a software product (such as a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to enable a computer device (such as a personal computer, a server or a network device, etc.) to perform some or all of the steps of the method described in the embodiment of the present disclosure. The aforementioned memory may include, but is not limited to, various media that can store program code, such as a USB flash drive, a flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0078] In other implementation scenarios, the above-mentioned integrated units can also be implemented in the form of hardware, that is, as specific hardware circuits, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which can be, for example, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM and RAM, etc.

[0079] The foregoing content can be better understood in accordance with the following terms:

[0080] Clause A1. A method of adjusting parameters of an image signal processor, comprising:

[0081] Acquire a sample data set formed by raw image data collected by a target lens and a target image sensor;

[0082] Obtain a training domain dataset for training the target network model;

[0083] The learnable image signal processing model is trained using the sample data set and the training domain data set; and the parameters of the trained learnable image signal processing model are configured to an image signal processor.

[0084] Clause A2. The method of clause A1, wherein training the learnable image signal processing model using the sample dataset and the training domain dataset comprises:

[0085] The sample data set is input into the learnable image signal processing model to obtain an inference domain data set; and a similarity evaluation is performed on the inference domain data set and the training domain data set to adjust the parameters of the learnable image signal processing model according to the similarity evaluation result.

[0086] Clause A3. The method according to clause A2, further comprising:

[0087] forming a generative adversarial network based on the learnable image signal processing model and the domain classification network; and

[0088] The learnable image signal processing model and the domain classification network are alternately trained, wherein the domain classification network is discriminatively trained using the inference domain dataset and the training domain dataset to evaluate the similarity between the inference domain dataset and the training domain dataset.

[0089] Clause A4. The method of clause A3, wherein the domain classification network comprises a VGG network model or a ResNet network model.

[0090] Clause A5. The method according to clause A2, wherein performing similarity evaluation on the inference domain dataset and the training domain dataset comprises:

[0091] Extracting image features of each inference image data in the inference domain dataset to generate a first image feature distribution of the inference domain dataset;

[0092] extracting image features of each training image data in the training domain dataset to generate a second image feature distribution of the training domain dataset; and

[0093] A similarity evaluation is performed on the first image feature distribution and the second image feature distribution.

[0094] Clause A6. The method of clause A5, wherein the image characteristics include at least one of:

[0095] Image brightness;

[0096] Image contrast;

[0097] Image sharpness; and

[0098] Image noise intensity.

[0099] Clause A7. The method according to clause A5 or clause A6, wherein the evaluation index of the similarity evaluation includes chi-square similarity and / or Euclidean distance.

[0100] Clause A8. The method according to any one of clauses A1-A7, wherein, before training the learnable image signal processing model, the method further comprises:

[0101] Based on the image signal processor, the learnable image signal processing model is generated.

[0102] Clause A9. The method of clause A8, wherein generating the learnable image signal processing model based on the image signal processor comprises:

[0103] The image signal processor is subjected to differentiable processing to obtain the learnable image signal processing model.

[0104] Clause A10. A data processing method comprising:

[0105] Acquiring raw data to be processed collected by a target lens and a target image sensor; and

[0106] The image signal processor whose parameters have been adjusted according to any one of the methods described in clauses A1-A9 is used to process the raw data to obtain target image data.

[0107] Clause A11. The data processing method according to Clause A10, further comprising:

[0108] Perform visual task processing on the target image data using a target network model.

[0109] Clause A12. An electronic device comprising:

[0110] a processor for executing program instructions; and

[0111] A memory storing the program instructions, which, when loaded and executed by the processor, enables the processor to perform the method for adjusting image signal processor parameters according to any one of clauses A1-A9, or the data processing method according to clause A10 or A11.

[0112] Item A13. A computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by one or more processors, implements the method for adjusting image signal processor parameters as described in any of Items A1-A9, or the data processing method as described in Item A10 or A11.

[0113] Although a plurality of embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may conceive of many modifications, changes, and alternatives without departing from the ideas and spirit of the present disclosure. It should be understood that in practicing the present disclosure, various alternatives to the embodiments of the present disclosure described herein may be adopted. The appended claims are intended to define the scope of protection of the present disclosure and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for adjusting parameters of an image signal processor, comprising: Acquire a sample data set formed by raw image data collected by a target lens and a target image sensor; Obtain a training domain dataset for training a target network model; Training a learnable image signal processing model using the sample data set and the training domain data set; as well as The parameters of the trained learnable image signal processing model are configured to the image signal processor.

2. The method according to claim 1, wherein training the learnable image signal processing model using the sample dataset and the training domain dataset comprises: Inputting the sample data set into the learnable image signal processing model to obtain an inference domain data set; as well as A similarity evaluation is performed on the reasoning domain data set and the training domain data set to adjust parameters of the learnable image signal processing model according to the similarity evaluation result.

3. The method according to claim 2, further comprising: Based on the learnable image signal processing model and the domain classification network, forming a generative adversarial network; as well as The learnable image signal processing model and the domain classification network are alternately trained, wherein the domain classification network is discriminatively trained using the reasoning domain dataset and the training domain dataset to evaluate the similarity between the reasoning domain dataset and the training domain dataset.

4. The method according to claim 3, wherein the domain classification network comprises a VGG network model or a ResNet network model.

5. The method according to claim 2, wherein performing similarity evaluation on the reasoning domain dataset and the training domain dataset comprises: Extracting image features of each inference image data in the inference domain data set to generate a first image feature distribution of the inference domain data set; Extracting image features of each training image data in the training domain data set to generate a second image feature distribution of the training domain data set; as well as A similarity evaluation is performed on the first image feature distribution and the second image feature distribution.

6. The method according to claim 5, wherein the image features include at least one of the following: Image brightness; Image contrast; Image sharpness; and Image noise intensity.

7. The method according to claim 5 or 6, wherein the evaluation index of the similarity evaluation comprises chi-square similarity and / or Euclidean distance.

8. The method according to any one of claims 1 to 7, before training the learnable image signal processing model, the method further comprises: Based on the image signal processor, the learnable image signal processing model is generated.

9. The method according to claim 8, wherein generating the learnable image signal processing model based on the image signal processor comprises: The image signal processor is subjected to differentiable processing to obtain the learnable image signal processing model.

10. A data processing method, comprising: Acquire raw data to be processed collected by a target lens and a target image sensor; as well as The image signal processor whose parameters are adjusted according to any one of the methods described in claims 1 to 9 is used to process the raw data to be processed to obtain target image data.

11. The data processing method according to claim 10, further comprising: The target image data is processed using a target network model for a visual task.

12. An electronic device comprising: a processor for executing program instructions; as well as A memory storing the program instructions, which, when loaded and executed by the processor, enables the processor to execute the method for adjusting image signal processor parameters according to any one of claims 1 to 9, or the data processing method according to claim 10 or 11.

13. A computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by one or more processors, implement the method for adjusting image signal processor parameters as described in any one of claims 1 to 9, or the data processing method as described in claim 10 or 11.

Citation Information

Patent Citations

  • Short video classification method

    CN110457525A

  • Unsupervised pedestrian re-identification method based on camera distribution difference alignment constraint

    CN113065409A

  • Method and system for reliable classification using a neural network

    EP3779799A1

  • System and method for an adjustable neural network

    WO2022072796A1

Cited By

  • Graphic contour detection method and system of industrial product, storage medium and device

    CN122066968A