Image processing chip parameter optimization method and device, electronic equipment and storage medium
By using the identification detection results of the image detection model and the adaptive evolution strategy of covariance, the parameters of the image processing chip are automatically debugged, and the problem of inefficient debugging in the existing technology is solved, and more efficient image processing and better image effects are achieved.
Patent Information
- Application Number
- CN202510048839.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-10
- Publication Date
- 2025-05-09
AI Technical Summary
In the prior art, the debugging efficiency of the image processing chip is low and it is difficult to achieve the best image effect. This is mainly due to the high coupling and complexity between the modules, resulting in low manual debugging efficiency.
By inputting the original image and sample processing images into the trained image detection model for identification and detection, the technical parameter type of the image processing chip is determined based on the differences in the detection results, and the parameters are debugged using the covariance adaptive evolution strategy to obtain optimization parameters to control the operation of the image processing chip.
It improves the debugging efficiency of the image processing chip, shortens the debugging cycle, improves the generalization ability of the image detection model, and can achieve the best image effect more effectively.
Smart Images
Figure CN119963519A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image processing chip parameter optimization method, device, electronic device and storage medium. Background Art
[0002] Image signal processing unit (ISP) is a type of integrated circuit specially used for processing image data. It is widely used in electronic devices such as mobile phones, computers, and smart cameras. It can process the collected images through traditional image processing technology combined with AI (Artificial Intelligence) technology, such as denoising, tone mapping, color correction, sharpening, automatic exposure, automatic white balance, etc. However, due to the coupling between the modules in the image processing chip, the complexity is high, and the current debugging of image processing chips mainly relies on manual debugging of image processing parameters in the algorithm module, which is inefficient and difficult to achieve the best image effect. Summary of the invention
[0003] The embodiments of the present application provide a method, device, electronic device and storage medium for optimizing image processing chip parameters, which can improve the debugging efficiency of image processing chips and the generalization capability of image detection models.
[0004] In a first aspect, an embodiment of the present application provides a method for optimizing image processing chip parameters, comprising:
[0005] Inputting the original image into an image detection model for recognition detection to obtain a first image detection result, wherein the image detection model is a trained model;
[0006] Inputting a sample processed image into the image detection model for the recognition detection to obtain a second image detection result, wherein the sample processed image is an image generated after the image processing chip processes the original image;
[0007] Determining the type of the technical parameter of the image processing chip based on the difference between the first image detection result and the second image detection result;
[0008] Using a covariance adaptive evolution strategy, the technical parameters are debugged using an image detection scoring standard as a training target to obtain optimized parameters, wherein the image detection scoring standard is used to evaluate the model performance of the image detection model;
[0009] Based on the optimization parameters, the image processing chip is controlled to perform image processing.
[0010] In some embodiments, the image detection model includes an encoder and a decoder;
[0011] The identification detection includes:
[0012] Extracting image features of an input image using the encoder, wherein the input image includes the original image and the sample processed image, and the image features include a plurality of bottom-level features and a plurality of high-level features;
[0013] fusing the image feature and the text feature of the input image using the decoder to form a fused feature, wherein the text feature is extracted based on the original image;
[0014] A convolution operation is performed on the fused features to output an image detection result, where the image detection result includes the first image detection result and the second image detection result.
[0015] In some implementations, the using the decoder to fuse the image feature with the text feature of the input image to form a fused feature includes:
[0016] Adjusting the weights between the bottom-level features and the high-level features in the image features, and fusing the bottom-level features and the high-level features to form a plurality of secondary fused features;
[0017] The plurality of secondary fusion features are fused with the text features respectively to form a plurality of fusion features.
[0018] In some implementations, performing a convolution operation on the fused features to output an image detection result includes:
[0019] Performing fusion operations on the plurality of fusion features respectively to output a plurality of the image detection results;
[0020] Based on the image detection scoring standard, the first image detection result and the second image detection result are screened out from the plurality of image detection results.
[0021] In some embodiments, the image detection scoring criteria include edge ODS;
[0022] The determining the type of the technical parameter of the image processing chip based on the difference between the first image detection result and the second image detection result includes:
[0023] Based on the difference between the edge ODS corresponding to the original image and the edge ODS corresponding to the sample processed image, determine the image quality index difference between the original image and the sample processed image, wherein the image quality index difference is used to characterize the factors affecting the edge ODS in the image features;
[0024] Based on the image quality indicator difference, the type of technical parameter of the image processing chip is determined.
[0025] In some implementations, the covariance adaptive evolution strategy is used to debug the technical parameters using the image detection scoring standard as a training target to obtain the optimized parameters, including:
[0026] Selecting multiple sample technical parameters within the value range corresponding to the technical parameters;
[0027] Based on the plurality of sample technical parameters, controlling the image processing chip to process the original image to obtain a plurality of sample processed images;
[0028] Inputting the plurality of sample processed images into the image detection model respectively for performing the recognition detection to obtain a plurality of second image detection results;
[0029] Based on the fit between the image detection scoring standard and the second image detection result, the sample technical parameters are iteratively trained to obtain the optimized parameters.
[0030] In some implementations, the iterative training of the sample technical parameters based on the fit between the image detection scoring standard and the second image detection result to obtain the optimized parameters includes:
[0031] Constructing an optimization function according to the fit between the image detection scoring standard and the second image detection result;
[0032] Using the covariance adaptive evolution strategy, the optimization function is processed to obtain a minimum value of the optimization function;
[0033] When the minimum value is less than or equal to a preset value, and / or the iterative training reaches a preset number of times, the corresponding sample technical parameters are selected from the plurality of sample processing images based on the minimum value as the optimization parameters.
[0034] In some implementations, the image parameter optimization method further includes:
[0035] In each iterative training, a plurality of sample technical parameters are generated by adjusting the weight coefficients corresponding to different technical parameters, and the sum of the weight coefficients corresponding to the different technical parameters is 1.
[0036] In a second aspect, an embodiment of the present application provides an image processing chip parameter optimization device, comprising:
[0037] A first detection unit, used for inputting the original image into an image detection model for recognition detection to obtain a first image detection result, wherein the image detection model is a trained model;
[0038] A second detection unit is used to input a sample processed image into the image detection model to perform the recognition detection to obtain a second image detection result, wherein the sample processed image is an image generated after the image processing chip processes the original image;
[0039] a determining unit, configured to determine a type of the technical parameter of the image processing chip based on a difference between the first image detection result and the second image detection result;
[0040] A debugging unit, used to debug the technical parameters using a covariance adaptive evolution strategy and taking an image detection scoring standard as a training target to obtain optimized parameters, wherein the image detection scoring standard is used to evaluate the model performance of the image detection model;
[0041] A control unit is used to control the image processing chip to perform image processing based on the optimization parameters.
[0042] In a third aspect, an embodiment of the present application provides an electronic device, including:
[0043] Memory for storing computer programs;
[0044] A processor is used to execute the computer program to implement the steps of any one of the methods in the first aspect.
[0045] In a fourth aspect, an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the steps of any one of the methods described in the first aspect.
[0046] The image processing chip parameter optimization method proposed in the embodiment of the present application uses the processing results of the image detection model to reflect the image processing quality improvement of the image processing chip, determines the parameter type that needs to be debugged, and uses the image detection scoring standard that reflects the image detection model as the training target, and uses the covariance adaptive evolution strategy to debug the parameters that need to be debugged of the image processing chip to obtain the optimized parameters, and based on the optimized parameters, controls the image processing chip to perform image processing. Compared with the traditional manual debugging of the image processing chip, the use of a gradient-free optimization algorithm to debug the parameters of the image processing chip can fully shorten the debugging cycle, improve the debugging efficiency, and also improve the generalization ability of the image detection model.
[0047] The technical effects obtained in the above-mentioned second to fourth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0049] Figure 1 A schematic diagram of a flow chart of an image processing chip parameter optimization method proposed in an embodiment of the present application;
[0050] Figure 2 This is a schematic diagram of an application of an image processing chip parameter optimization method proposed in an embodiment of the present application;
[0051] Figure 3 A schematic diagram of the structure of an image processing chip parameter optimization device proposed in an embodiment of the present application;
[0052] Figure 4 A schematic diagram of the structure of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION
[0053] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0054] It should be understood that the "multiple" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.
[0055] In addition, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product, or apparatus.
[0056] The image processing chip can process the original Bayer image into an RGB image, which involves the coordination of multiple internal modules, such as denoising, tone mapping, color correction, sharpening, automatic exposure, automatic white balance, etc.
[0057] However, currently, the debugging of image processing chips mainly relies on manual debugging of image processing parameters in algorithm modules. Since the modules in the image processing chip are coupled and highly complex, the debugging of image processing chips in related technologies has the technical problems of low efficiency and difficulty in achieving the best image effect.
[0058] In view of this, an embodiment of the present application proposes a method for optimizing image processing chip parameters to at least solve the above technical problems. Figure 1 A flow chart of an image processing chip parameter optimization method proposed in an embodiment of the present application is shown in FIG. Figure 1 As shown, the above-mentioned image processing chip parameter optimization method includes steps S110 to S150:
[0059] S110, inputting the original image into an image detection model for recognition detection to obtain a first image detection result, where the image detection model is a trained model.
[0060] S120, inputting the sample processed image into the image detection model for recognition detection to obtain a second image detection result, wherein the sample processed image is an image generated after the image processing chip processes the original image.
[0061] For example, Figure 2 This is a schematic diagram of an application of an image processing chip parameter optimization method proposed in an embodiment of the present application, such as Figure 2 As shown, the original image may be the aforementioned Bayer image, which may be understood as an image that has not undergone some image processing operations or simple image processing operations. One of the two identical original images is directly sent to the above image detection model for recognition detection to obtain a first image detection result, and the other is first processed by ISP to generate a sample processed image, which is then sent to the above image detection model for the same recognition detection to obtain a second image detection result.
[0062] The above image detection model can be used for OCR (Optical Character Recognitio) or a neural network model for contour detection. The above image detection results are indicators based on the usefulness of the image detection model, for example, for image semantic information recognition, how much is the recognition accuracy, if it is used for contour detection, how many images have marked the corresponding object contour, how much is the accuracy, etc.
[0063] It should be noted that, in order to control variables, the above image detection model processes the original image and the sample processed image in the same way. In some examples, the above image detection model can be a neural network model based on the ResNet architecture, including an encoder and a decoder, the encoder is used to extract image features, and the decoder is used for feature fusion. Recognition detection based on the above image detection model can be performed in the following ways:
[0064] The encoder is used to extract image features of an input image, where the input image includes an original image and a sample processed image, and the image features include multiple low-level features and multiple high-level features.
[0065] Exemplarily, after receiving the input image, the image detection model uses an encoder to extract the underlying features and high-level features in the image, where the underlying features represent features that reflect the visual information of the image, such as color, texture, shape, etc., and the high-level features represent features with a higher degree of abstraction, which reflect the semantic information of the image.
[0066] The decoder is used to fuse the image features with the text features of the input image to form fused features, and the text features are extracted based on the original image.
[0067] Exemplarily, the above text features can be extracted by a text encoder, such as a CLIP (Contrastive Language-Image Pretraining) model.
[0068] It should be noted that after the original image and the sample processed image are input into the image detection model, they are fused with the same text features, and the text features are extracted from the original image. That is, when the feature fusion operation is performed on the original image and the sample processed image, the image features (bottom-level features, high-level features) of the two are different, but the text features are the same.
[0069] A convolution operation is performed on the fused features to output an image detection result, where the image detection result includes a first image detection result and a second image detection result.
[0070] Exemplarily, convolution operations are performed on the fusion features formed by the original image and the fusion features formed by the sample processed image, respectively, so as to obtain a first image detection result corresponding to the original image and a second image detection result corresponding to the sample processed image, respectively.
[0071] In this embodiment, the same text features are used to fuse the original image and the sample processed image, so that the difference in the fusion features formed by the two depends on the image features, and the obtained image detection results can better reflect the difference in image visual quality between the two. Therefore, the role of the image processing chip in improving image quality can be more clearly concluded.
[0072] In some examples, the above-mentioned use of the decoder to fuse the image features and the text features of the input image to form the fused features includes:
[0073] The weights between the underlying features and the high-level features in the image features are adjusted, and the underlying features and the high-level features are fused to form multiple secondary fusion features.
[0074] Multiple secondary fusion features are fused with text features respectively to form multiple fusion features.
[0075] Exemplarily, image features are divided into multiple levels, with the bottom-level features at a lower level and the high-level features at a higher level. The multiple bottom-level features and multiple high-level features extracted in the embodiment of the present application may have different numbers of layers. During the fusion process, the ratio of the two is adjusted to form multiple secondary fusion features. Assume that the image features are divided into five levels from 1 to 5, with 1 being the lowest and 5 being the highest. Layers 1 to 5 reflect the change from bottom-level features to high-level features. Layers 3 and below represent bottom-level features, and layers 4 and 5 represent high-level features. When fusion is performed, the weights between bottom-level features and high-level features are adjusted. For example, layers 1, 2 and 5 may be fused, layers 1, 2, 3 and 4 may be fused, or layers 2, 3 and 4, 5 may be fused, so that multiple secondary fusion features can be obtained.
[0076] The formed multiple secondary fusion features are then fused with the above text features respectively to obtain corresponding multiple fusion features.
[0077] Then, the multiple fusion features are fused separately, and the above image detection model can output multiple image detection results.
[0078] And based on the image detection scoring standard, the first image detection result and the second image detection result are screened out from the multiple image detection results.
[0079] Exemplarily, for the original image and the sample processed image, during the recognition detection process, by adjusting the fusion ratio of the underlying features and the high-level features, an image detection result can be generated at each ratio, that is, the original image and the sample processed image respectively obtain a plurality of first image detection results and a second image detection result. Suitable detection results are screened out from the plurality of first image detection results and the plurality of second image detection results for subsequent comparison operations. The screening criteria are determined based on the image detection scoring criteria, and the image detection scoring criteria may include accuracy, for example, the IoU (Intersection over Union) of contour detection. Based on the scoring criteria, the detection result with the highest score can be selected as the above-mentioned first image detection result and the second image detection result for subsequent comparison operations.
[0080] In this example, different fusion ratios of multiple underlying image features and high-level image features are selected to form different fusion features, and multiple detection results are formed based on different fusion features. Then specific results are screened out for subsequent operations. This can overcome the errors caused by the image detection model itself, thereby obtaining detection results that better reflect the image quality itself.
[0081] S130: Determine the type of the technical parameter of the image processing chip based on the difference between the first image detection result and the second image detection result.
[0082] Exemplarily, the differences between the image detection results are determined based on the usefulness or function of the above-mentioned image detection model. Figure 2 As shown, taking the image detection model used for image contour detection as an example, the image detection result can be to mark the edge pixels on the image, and based on the differences in the grayscale values, colors, tones, etc. of the edge pixels, generate the difference between the first image detection result corresponding to the original image and the second image detection result corresponding to the sample processed image. From this difference, it can be concluded that for the above-mentioned image detection model, which image features (grayscale value, color, tones, etc.) have a greater impact on the recognition accuracy of the image detection model. For example, if the grayscale values of the edge pixels in the first image detection result and the second image detection result are very different, and the other differences are not large, it indicates that for the image detection model, the image brightness feature has a relatively large impact, and the parameters affecting the image brightness feature in the ISP can be debugged, that is, the type of technical parameters to be debugged for the above-mentioned image processing chip is determined.
[0083] In some examples, the ISP may process the original image to generate the sample processed image and determine the type of technical parameter to be debugged in the following manner:
[0084] The original image is input into the image processing chip for ablation experiment, and the image processing submodules are turned off in sequence to generate different sample processed images.
[0085] Exemplarily, the steps of the ablation experiment may include:
[0086] S1. List all internal image processing submodules involved in ISP, such as those responsible for denoising, tone mapping, color correction, etc.
[0087] S2. Determine an experimental group for the order in which the above-mentioned image processing submodules are closed in sequence.
[0088] S3. Close the processing submodules in sequence, and generate a corresponding sample processed image based on the original image.
[0089] Then, all the generated sample processed images are input into the above-mentioned image detection model for recognition and detection to generate multiple second image detection results. The multiple second image detection results are respectively compared with the above-mentioned first image detection results to determine the experimental group with obvious differences. Based on the image processing sub-module corresponding to the experimental group, the type of corresponding technical parameters to be debugged is determined.
[0090] In some examples, the image detection scoring criteria include edge ODS (Optimal Dataset Scale, global best), and the step S130 may include:
[0091] Based on the difference between the edge ODS corresponding to the original image and the edge ODS corresponding to the sample processed image, the image quality index difference between the original image and the sample processed image is determined, and the image quality index difference is used to characterize the factors affecting the edge ODS in the image features.
[0092] Based on the differences in image quality indicators, the type of technical parameters of the image processing chip is determined.
[0093] It should be noted that edge ODS uses the overall detection of the entire image set as the score. For each image, different thresholds are used to convert the edge probability Figure 2 The image detection model is quantified, and the precision and recall of each image are calculated based on the value. The F-score (i.e., F score, an indicator used to measure the accuracy of a binary classification model in statistics and machine learning) of the entire image set is calculated based on the average of the precision and recall of each image. The maximum threshold is determined as the score of the above-mentioned edge ODS. In the embodiment of the present application, using the entire image set as the scoring standard can more accurately reflect the performance of the image detection model, thereby being able to better judge the difference in image quality between the original image and the sample processed image.
[0094] S140, using the covariance adaptive evolution strategy, the image detection scoring standard is used as a training target to debug the technical parameters to obtain optimized parameters, and the image detection scoring standard is used to evaluate the model performance of the image detection model.
[0095] For example, Figure 2 As shown, an optimizer can be used to debug the parameters in the ISP using the above image detection scoring criteria as the training target. The strategy adopted by the optimizer is the covariance adaptive evolution strategy (CMA-ES). In the optimizer, the initial parameters of the ISP are debugged based on the variance set by CMA-ES until the preset effect is achieved. By using the CMA-ES strategy and a gradient-free optimization algorithm to debug the parameters of the image processing chip, the debugging cycle can be fully shortened and the debugging efficiency can be improved.
[0096] It is understandable that the above image detection scoring criteria have been described in detail above and will not be repeated here.
[0097] In some examples, the above step S140 may be performed in the following manner:
[0098] Select multiple sample technical parameters within the value range corresponding to the technical parameters.
[0099] It should be noted that after determining the type of technical parameters that need to be debugged based on step S130, the initial technical parameters of this type are used as the initial seed node, multiple child nodes are selected within the value range of the initial seed node, and a new seed node is determined in the next step of processing. The value range can be determined based on the initial variance of the covariance adaptive evolution strategy.
[0100] Based on multiple sample technical parameters, the image processing chip is controlled to process the original image to obtain multiple sample processed images.
[0101] Exemplarily, the ISP is controlled to process the same original image based on the multiple sample technical parameters selected above, thereby obtaining multiple corresponding sample processed images.
[0102] The plurality of sample processed images are respectively input into the image detection model for recognition detection to obtain a plurality of second image detection results.
[0103] Based on the fit between the image detection scoring criteria and the second image detection result, the sample technical parameters are iteratively trained to obtain optimized parameters.
[0104] Exemplarily, the second image detection result that has the highest degree of fit with the above-mentioned image detection scoring criteria is selected from multiple second image detection results, and the second image detection result is set as a new seed node. Based on the seed node, the value range is determined to continue the optimization process until the iteration requirements are met, and the sample processing parameters corresponding to the final second image detection result are determined as the above-mentioned optimization parameters.
[0105] In the above example, based on the fit between the image detection scoring standard and the second image detection result, iteratively training the sample technical parameters to obtain the optimized parameters may include:
[0106] An optimization function is constructed according to the fit between the image detection scoring standard and the second image detection result.
[0107] For example, the optimization function may be as follows:
[0108]
[0109] Wherein, T represents the value of the optimization function; K represents the number of iterations, and K is greater than 1; p represents the image processing evaluation index; and p′ represents the image quality evaluation index.
[0110] The optimization function is processed using the covariance adaptive evolution strategy to obtain the minimum value of the optimization function.
[0111] Exemplarily, in each iteration, the sample processing parameter corresponding to the minimum value of the optimization function among the multiple sample processing parameters is used as a new seed node.
[0112] When the minimum value is less than or equal to a preset value, and / or the iterative training reaches a preset number of times, corresponding sample technical parameters are selected from multiple sample processing images based on the minimum value as optimization parameters.
[0113] Exemplarily, the criterion for stopping the optimization is set when the minimum value of the optimization function reaches a preset value, or the number of iterations has been reached, or both are used as the criterion for stopping the iteration. When the iteration is stopped, the sample technical parameter corresponding to the minimum value of the optimization function in the last iteration is selected as the optimization parameter.
[0114] According to some embodiments, the image parameter optimization method provided in the embodiments of the present application also includes: in each iterative training, by adjusting the weight coefficients corresponding to different technical parameters, a plurality of sample technical parameters are generated, and the sum of the weight coefficients corresponding to the different technical parameters is 1.
[0115] Exemplarily, ISP affects the quality of image processing through the combined effect of multiple technical parameters. For example, there are three parameters a, b, and c. Initially, the ratio of a, b, and c is 1:2:3. When determining the sample processing parameters corresponding to the three parameters a, b, and c, they can be adjusted to multiple groups of proportional coefficients such as 2:1:3, 3:2:1, or 2.5:1.5:2 to form multiple sample technical parameters. By adjusting the weights between the various parameters to select multiple sample processing parameters, it is possible to better reflect the proportion of the impact of different parameters on image quality, thereby better determining the debugging direction of the parameters and improving the debugging efficiency.
[0116] S150, controlling the image processing chip to perform image processing based on the optimized parameters.
[0117] Exemplarily, the ISP performs image processing based on the optimization parameters obtained in step S140 to obtain an image that best matches the image quality evaluation index.
[0118] In summary, the image processing chip parameter optimization method proposed in the embodiment of the present application uses the processing results of the image detection model to reflect the image processing quality improvement of the image processing chip, determines the parameter type that needs to be debugged, and uses the image detection scoring standard that reflects the image detection model as the training target, and uses the covariance adaptive evolution strategy to debug the parameters that need to be debugged of the image processing chip to obtain the optimized parameters, and based on the optimized parameters, controls the image processing chip to perform image processing. Compared with the traditional manual debugging of the image processing chip, the use of a gradient-free optimization algorithm to debug the parameters of the image processing chip can fully shorten the debugging cycle and improve the debugging efficiency.
[0119] The method embodiments according to the present application are described above by way of example. Based on the image processing chip parameter optimization method provided in the embodiments of the present application, the embodiments of the present application provide an image processing chip parameter optimization device. Figure 3 This is a schematic diagram of the structure of an image processing chip parameter optimization device provided in an embodiment of the present application. Figure 3 , the device includes the following units.
[0120] A first detection unit 310 is used to input the original image into an image detection model for recognition detection to obtain a first image detection result, wherein the image detection model is a trained model;
[0121] A second detection unit 320 is used to input a sample processed image into the image detection model for the recognition detection to obtain a second image detection result, wherein the sample processed image is an image generated after the image processing chip processes the original image;
[0122] A determination unit 330, configured to determine a type of the technical parameter of the image processing chip based on a difference between the first image detection result and the second image detection result;
[0123] A debugging unit 340, configured to use a covariance adaptive evolution strategy to debug the technical parameters using an image detection scoring standard as a training target to obtain optimized parameters, wherein the image detection scoring standard is used to evaluate the model performance of the image detection model;
[0124] The control unit 350 is used to control the image processing chip to perform image processing based on the optimization parameters.
[0125] The above describes the device embodiments of the present application. For detailed descriptions of the specific execution processes of data, terms, nouns, steps, technical issues and effects, alternative methods and combinations, please refer to the descriptions in the method embodiments, which will not be repeated here.
[0126] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the steps of any one of the methods described in the first aspect.
[0127] The present application also provides an electronic device, including:
[0128] Memory for storing computer programs;
[0129] A processor is used to execute the computer program to implement the steps of any one of the methods in the first aspect.
[0130] For example, Figure 4 As shown, the electronic device includes a processor 410, at least one communication bus 420, a user interface 430, at least one external communication interface 440 and a memory 450. The communication bus 420 is configured to achieve connection and communication between these components. The user interface 430 may include a display screen, and the external communication interface 440 may include a standard wired interface and a wireless interface. The memory 500 stores a computer program. The processor 410 is used to execute the computer program stored in the memory 450.
[0131] The description of the above computer program product, computer-readable storage medium, and electronic device is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the computer program product, computer-readable storage medium, and electronic device of this application, please refer to the description of the method embodiment of this application for understanding.
[0132] The sequence of serial numbers or introduction of the embodiments of the present application is for description only and does not represent the superiority or inferiority of the embodiments.
[0133] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0135] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0136] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiment of the present application may be a non-volatile storage medium, in other words, a non-transient storage medium.
[0137] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the scene data of the current frame in the three-dimensional virtual scene, the device information of the client, and the scene interaction information involved in the embodiments of this application are all obtained with full authorization.
[0138] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.
Claims
1. A method for optimizing image processing chip parameters, characterized in that: include: Inputting the original image into an image detection model for recognition detection to obtain a first image detection result, wherein the image detection model is a trained model; Inputting a sample processed image into the image detection model for the recognition detection to obtain a second image detection result, wherein the sample processed image is an image generated after the image processing chip processes the original image; Determining the type of the technical parameter of the image processing chip based on the difference between the first image detection result and the second image detection result; Using a covariance adaptive evolution strategy, the technical parameters are debugged using an image detection scoring standard as a training target to obtain optimized parameters, wherein the image detection scoring standard is used to evaluate the model performance of the image detection model; Based on the optimization parameters, the image processing chip is controlled to perform image processing.
2. The method according to claim 1, characterized in that: The image detection model includes an encoder and a decoder; The identification detection includes: Extracting image features of an input image using the encoder, wherein the input image includes the original image and the sample processed image, and the image features include a plurality of bottom-level features and a plurality of high-level features; fusing the image feature and the text feature of the input image using the decoder to form a fused feature, wherein the text feature is extracted based on the original image; A convolution operation is performed on the fused features to output an image detection result, where the image detection result includes the first image detection result and the second image detection result.
3. The method according to claim 2, characterized in that The step of fusing the image feature and the text feature of the input image using the decoder to form a fused feature includes: Adjusting the weights between the bottom-level features and the high-level features in the image features, and fusing the bottom-level features and the high-level features to form a plurality of secondary fused features; The plurality of secondary fusion features are fused with the text features respectively to form a plurality of fusion features.
4. The method according to claim 3, characterized in that The performing a convolution operation on the fused features to output an image detection result includes: Performing fusion operations on the plurality of fusion features respectively to output a plurality of the image detection results; Based on the image detection scoring standard, the first image detection result and the second image detection result are screened out from the plurality of image detection results.
5. The method according to claim 4, characterized in that The image detection scoring criteria include edge ODS; The determining the type of the technical parameter of the image processing chip based on the difference between the first image detection result and the second image detection result includes: Based on the difference between the edge ODS corresponding to the original image and the edge ODS corresponding to the sample processed image, determine the image quality index difference between the original image and the sample processed image, wherein the image quality index difference is used to characterize the factors affecting the edge ODS in the image features; Based on the image quality indicator difference, the type of technical parameter of the image processing chip is determined.
6. The method according to any one of claims 1 to 5, characterized in that The covariance adaptive evolution strategy is used to debug the technical parameters by taking the image detection scoring standard as the training target to obtain the optimized parameters, including: Selecting multiple sample technical parameters within the value range corresponding to the technical parameters; Based on the plurality of sample technical parameters, controlling the image processing chip to process the original image to obtain a plurality of sample processed images; Inputting the plurality of sample processed images into the image detection model respectively for performing the recognition detection to obtain a plurality of second image detection results; Based on the fit between the image detection scoring standard and the second image detection result, the sample technical parameters are iteratively trained to obtain the optimized parameters.
7. The method according to claim 6, characterized in that The iterative training of the sample technical parameters based on the fit between the image detection scoring standard and the second image detection result to obtain the optimized parameters includes: Constructing an optimization function according to the fit between the image detection scoring standard and the second image detection result; Using the covariance adaptive evolution strategy, the optimization function is processed to obtain a minimum value of the optimization function; When the minimum value is less than or equal to a preset value, and / or the iterative training reaches a preset number of times, the corresponding sample technical parameters are selected from the plurality of sample processing images based on the minimum value as the optimization parameters.
8. The method according to claim 7, characterized in that Also includes: In each iterative training, a plurality of sample technical parameters are generated by adjusting the weight coefficients corresponding to different technical parameters, and the sum of the weight coefficients corresponding to the different technical parameters is 1.
9. An image processing chip parameter optimization device, characterized in that: The device comprises: A first detection unit, used for inputting the original image into an image detection model for recognition detection to obtain a first image detection result, wherein the image detection model is a trained model; A second detection unit is used to input a sample processed image into the image detection model to perform the recognition detection to obtain a second image detection result, wherein the sample processed image is an image generated after the image processing chip processes the original image; a determining unit, configured to determine a type of the technical parameter of the image processing chip based on a difference between the first image detection result and the second image detection result; A debugging unit, used to debug the technical parameters using a covariance adaptive evolution strategy and taking an image detection scoring standard as a training target to obtain optimized parameters, wherein the image detection scoring standard is used to evaluate the model performance of the image detection model; A control unit is used to control the image processing chip to perform image processing based on the optimization parameters.
10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the method according to any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the steps of the method according to any one of claims 1 to 8.